Test Automation Framework Design for DevOps

A framework that works well locally often breaks in CI for reasons that have nothing to do with the tests themselves: hardcoded paths, environment assumptions, sequential test dependencies. Designing for Azure DevOps from the start avoids those pitfalls.

---

Framework layers

┌─────────────────────────────────────────┐
│         Test Cases (specs)              │  Business scenarios
├─────────────────────────────────────────┤
│         Page Object Model               │  UI abstractions
├─────────────────────────────────────────┤
│         Test Helpers & Fixtures         │  Data, auth, utilities
├─────────────────────────────────────────┤
│         Configuration Layer            │  Env vars, settings
├─────────────────────────────────────────┤
│         Driver / Client Layer           │  Browser, HTTP client
└─────────────────────────────────────────┘

Each layer has a single responsibility. Tests don't directly manipulate the browser. Pages don't know about test data. Config doesn't contain business logic.

---

Configuration management for multiple environments

TYPESCRIPT
1// config/config.ts
2type Environment = 'local' | 'staging' | 'uat' | 'production'
3
4interface EnvConfig {
5  baseUrl: string
6  apiBaseUrl: string
7  adminEmail: string
8  adminPassword: string
9  defaultTimeout: number
10}
11
12const configs: Record<Environment, EnvConfig> = {
13  local: {
14    baseUrl: 'http://localhost:3000',
15    apiBaseUrl: 'http://localhost:4000',
16    adminEmail: 'admin@local.test',
17    adminPassword: 'localpass',
18    defaultTimeout: 10_000,
19  },
20  staging: {
21    baseUrl: process.env.STAGING_URL || '',
22    apiBaseUrl: process.env.STAGING_API_URL || '',
23    adminEmail: process.env.STAGING_ADMIN_EMAIL || '',
24    adminPassword: process.env.STAGING_ADMIN_PASSWORD || '',
25    defaultTimeout: 20_000,
26  },
27  uat: {
28    baseUrl: process.env.UAT_URL || '',
29    apiBaseUrl: process.env.UAT_API_URL || '',
30    adminEmail: process.env.UAT_ADMIN_EMAIL || '',
31    adminPassword: process.env.UAT_ADMIN_PASSWORD || '',
32    defaultTimeout: 25_000,
33  },
34  production: {
35    baseUrl: 'https://app.com',
36    apiBaseUrl: 'https://api.app.com',
37    adminEmail: '',
38    adminPassword: '',
39    defaultTimeout: 30_000,
40  },
41}
42
43const env = (process.env.TEST_ENV as Environment) || 'local'
44export const config = configs[env]

---

Page Object Model

TYPESCRIPT
1// pages/LoginPage.ts
2import { Page, Locator, expect } from '@playwright/test'
3
4export class LoginPage {
5  readonly page: Page
6  readonly emailInput: Locator
7  readonly passwordInput: Locator
8  readonly signInButton: Locator
9  readonly errorMessage: Locator
10
11  constructor(page: Page) {
12    this.page = page
13    this.emailInput    = page.getByTestId('email-input')
14    this.passwordInput = page.getByTestId('password-input')
15    this.signInButton  = page.getByRole('button', { name: 'Sign in' })
16    this.errorMessage  = page.getByTestId('error-message')
17  }
18
19  async navigate() {
20    await this.page.goto('/login')
21    await expect(this.emailInput).toBeVisible()
22  }
23
24  async login(email: string, password: string) {
25    await this.emailInput.fill(email)
26    await this.passwordInput.fill(password)
27    await this.signInButton.click()
28  }
29
30  async expectError(message: string) {
31    await expect(this.errorMessage).toContainText(message)
32  }
33}

---

Test data factory

TYPESCRIPT
1// factories/user-factory.ts
2import { randomUUID } from 'crypto'
3import { config } from '../config/config'
4
5let adminToken: string | null = null
6
7async function getAdminToken(): Promise<string> {
8  if (adminToken) return adminToken
9  const res = await fetch(`${config.apiBaseUrl}/auth/login`, {
10    method: 'POST',
11    headers: { 'Content-Type': 'application/json' },
12    body: JSON.stringify({ email: config.adminEmail, password: config.adminPassword }),
13  })
14  const { token } = await res.json()
15  adminToken = token
16  return token
17}
18
19export async function createTestUser(overrides: Partial<User> = {}): Promise<User> {
20  const token = await getAdminToken()
21  const userData = {
22    id: randomUUID(),
23    email: `test-${randomUUID()}@example.com`,
24    name: 'Test User',
25    role: 'member',
26    ...overrides,
27  }
28  const res = await fetch(`${config.apiBaseUrl}/users`, {
29    method: 'POST',
30    headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
31    body: JSON.stringify(userData),
32  })
33  return res.json()
34}
35
36export async function deleteTestUser(userId: string): Promise<void> {
37  const token = await getAdminToken()
38  await fetch(`${config.apiBaseUrl}/users/${userId}`, {
39    method: 'DELETE',
40    headers: { Authorization: `Bearer ${token}` },
41  })
42}

---

Pipeline integration patterns

Environment-specific variable groups

Library
  ├── staging-env        (staging URLs, non-secret config)
  ├── staging-secrets    (credentials — secret variables)
  ├── uat-env
  └── uat-secrets

YAML
1variables:
2  - group: staging-env
3  - group: staging-secrets
4  - name: TEST_ENV
5    value: staging

Fail fast, then report

YAML
1steps:
2  - script: npx jest tests/smoke  # Fast health check (30 sec)
3    displayName: Smoke (fail fast)
4
5  - script: npx jest tests/       # Full suite
6    displayName: Full regression
7    condition: succeeded()
8
9  - task: PublishTestResults@2    # Always publish
10    condition: always()

---

Common errors and fixes

Error: Config values are undefined in CI but work locally Fix: Check the variable group is linked to the pipeline AND the pipeline is authorised to use it. In Library → Variable Group → Pipeline permissions.

Error: Factory creates data but cleanup fails silently Fix: Wrap cleanup in try/catch and log failures. If cleanup consistently fails, add a pre-run cleanup step that deletes all test-prefixed records older than 24 hours.

Error: POM locators break after UI update Fix: Prefer data-testid attributes over CSS selectors or XPath. Ask the development team to add data-testid attributes to key elements. These are stable across UI redesigns.