Azure DevOps Pipeline for Automated Testing

Creating a pipeline for automated testing in Azure DevOps involves three decisions: what triggers the pipeline, what steps it runs, and how it reports results. This guide walks through each decision with complete, working YAML.

---

Creating the pipeline

Option A: From the UI (YAML)

1. Go to Pipelines → + New pipeline
2. Select your source: Azure Repos Git (or GitHub)
3. Select your repository
4. Choose Starter pipeline (edits in the browser) or Existing Azure Pipelines YAML file (if you already have a YAML file)

Option B: Commit azure-pipelines.yml

Create the file at the root of your repository:

BASH
1touch azure-pipelines.yml
2git add azure-pipelines.yml
3git commit -m "Add CI pipeline"
4git push

Azure DevOps detects this file automatically when you create the pipeline.

---

Pipeline structure for testing

A well-structured testing pipeline has three stages:

Stage 1: Validate       (fast checks — linting, compilation)
Stage 2: Unit Tests     (fast, isolated tests — run first)
Stage 3: Integration/E2E (slower, environment-dependent tests)

Failing fast in Stage 1 saves agent minutes and developer waiting time.

---

Complete pipeline YAML

YAML
1# azure-pipelines.yml
2trigger:
3  branches:
4    include:
5      - main
6      - release/*
7  paths:
8    exclude:
9      - docs/**
10      - '*.md'
11
12pr:
13  branches:
14    include:
15      - main
16
17pool:
18  vmImage: ubuntu-latest
19
20variables:
21  - group: test-environment-vars   # Variable group from Library
22
23stages:
24  # ── Stage 1: Validate ─────────────────────────────────────────────────────
25  - stage: Validate
26    displayName: Validate
27    jobs:
28      - job: Lint
29        displayName: Lint and type check
30        steps:
31          - task: NodeTool@0
32            inputs:
33              versionSpec: '20.x'
34
35          - script: npm ci
36            displayName: Install dependencies
37
38          - script: npm run lint
39            displayName: ESLint
40
41          - script: npm run typecheck
42            displayName: TypeScript check
43
44  # ── Stage 2: Unit Tests ───────────────────────────────────────────────────
45  - stage: UnitTests
46    displayName: Unit Tests
47    dependsOn: Validate
48    jobs:
49      - job: Unit
50        displayName: Run unit tests
51        steps:
52          - task: NodeTool@0
53            inputs:
54              versionSpec: '20.x'
55
56          - script: npm ci
57            displayName: Install
58
59          - script: |
60              npm run test:unit -- \
61                --reporter=junit \
62                --outputFile=results/unit-results.xml \
63                --coverage
64            displayName: Run unit tests
65
66          - task: PublishTestResults@2
67            displayName: Publish unit results
68            inputs:
69              testResultsFormat: JUnit
70              testResultsFiles: results/unit-results.xml
71              testRunTitle: Unit Tests — $(Build.BuildNumber)
72              mergeTestResults: true
73            condition: always()
74
75          - task: PublishCodeCoverageResults@1
76            displayName: Publish coverage
77            inputs:
78              codeCoverageTool: Cobertura
79              summaryFileLocation: coverage/cobertura-coverage.xml
80            condition: always()
81
82  # ── Stage 3: E2E Tests ────────────────────────────────────────────────────
83  - stage: E2ETests
84    displayName: E2E Tests
85    dependsOn: UnitTests
86    condition: succeeded()
87    jobs:
88      - job: Playwright
89        displayName: Playwright E2E
90        timeoutInMinutes: 30
91        steps:
92          - task: NodeTool@0
93            inputs:
94              versionSpec: '20.x'
95
96          - script: npm ci
97            displayName: Install
98
99          - script: npx playwright install --with-deps chromium firefox
100            displayName: Install browsers
101
102          - script: npx playwright test --reporter=junit,html
103            displayName: Run Playwright tests
104            env:
105              BASE_URL: $(STAGING_URL)
106              TEST_EMAIL: $(TEST_USER_EMAIL)
107              TEST_PASSWORD: $(TEST_USER_PASSWORD)
108            continueOnError: false
109
110          - task: PublishTestResults@2
111            displayName: Publish E2E results
112            inputs:
113              testResultsFormat: JUnit
114              testResultsFiles: playwright-results/results.xml
115              testRunTitle: E2E Tests — $(Build.BuildNumber)
116            condition: always()
117
118          - task: PublishPipelineArtifact@1
119            displayName: Upload HTML report
120            inputs:
121              targetPath: playwright-report
122              artifact: playwright-html-report
123              publishLocation: pipeline
124            condition: always()
125
126          - task: PublishPipelineArtifact@1
127            displayName: Upload screenshots on failure
128            inputs:
129              targetPath: test-results
130              artifact: test-screenshots
131            condition: failed()

---

Setting up variables and secrets

Never hardcode credentials in YAML. Use variable groups:

1. Go to Pipelines → Library → + Variable group
2. Name: test-environment-vars
3. Add variables:

   STAGING_URL          = https://staging.yourapp.com
   TEST_USER_EMAIL      = testuser@example.com
   TEST_USER_PASSWORD   = [mark as secret] ****
   

4. Reference the group in pipeline YAML: - group: test-environment-vars

Secret variables are masked in logs automatically.

---

Pull request validation

To block merges when tests fail:

1. Go to Repos → Branches → [main branch] → Branch Policies
2. Click + Add build validation
3. Select your pipeline
4. Set: Required (not Optional)
5. Trigger: Automatic on PR updates

Now PRs to main cannot be merged until the pipeline passes.

---

Common errors and fixes

Error: Pipeline runs but no test results appear in the Tests tab Fix: The PublishTestResults task must find actual XML files. Add a debug step: - script: find $(System.DefaultWorkingDirectory) -name "*.xml" -type f to see what's generated.

Error: "The pipeline is not valid. Job 'E2E' has a dependency on 'UnitTests' which doesn't exist" Fix: dependsOn references the stage name (not displayName). Match it exactly: dependsOn: UnitTests.

Error: E2E tests time out after 10 minutes Fix: Add timeoutInMinutes: 30 (or appropriate value) to the job definition. Default timeout is 60 minutes for Microsoft-hosted agents, but PR validation pipelines have a 10-minute default.

Error: Tests fail with "ECONNREFUSED" — can't reach the application Fix: The staging environment URL is unreachable from the pipeline agent. Check if staging is behind a VPN or firewall that blocks Azure DevOps agents. Use self-hosted agents inside the network if needed.

Error: Variable group not found in pipeline Fix: Go to Library → [Variable group] → Pipeline permissions and authorize the pipeline to use the group.