CI/CD for QA Engineers Using Azure DevOps

CI/CD is no longer just a developer concern. QA engineers who understand how pipelines work — and how to integrate tests into them — deliver significantly more value than those who only test manually. This guide explains CI/CD from the ground up for QA engineers who want to get practical quickly.

---

What CI/CD means for QA

Continuous Integration (CI): Every time a developer pushes code, an automated process runs — compiling the code, running tests, and reporting results. If tests fail, the pipeline fails and the developer is notified immediately.

Continuous Delivery (CD): After CI succeeds, the code is automatically deployed to staging (or production, in continuous deployment). QA engineers test in staging, knowing it always contains the latest passing code.

For QA, CI/CD means:

• Tests run automatically on every code change — no waiting for a dev to "push to staging"
• Every deployment to staging is known to have passed automated tests
• Failures are caught immediately, not days later
• You can run regression tests on-demand without setting anything up manually

---

How Azure Pipelines works

An Azure Pipeline is defined in a YAML file stored in the repository (azure-pipelines.yml). When a trigger event occurs (push, PR, schedule), Azure DevOps reads the YAML and executes the defined jobs on an agent (a virtual machine).

YAML
1# azure-pipelines.yml — minimal QA-focused pipeline
2
3trigger:
4  branches:
5    include:
6      - main
7
8pool:
9  vmImage: ubuntu-latest    # Microsoft-hosted Linux VM
10
11stages:
12  - stage: Test
13    jobs:
14      - job: RunTests
15        steps:
16          - script: echo "Pipeline started"
17          - script: npm ci
18          - script: npm test

---

Your first test-integrated pipeline

YAML
1trigger:
2  branches:
3    include:
4      - main
5      - feature/*
6
7pool:
8  vmImage: ubuntu-latest
9
10variables:
11  NODE_VERSION: '20.x'
12
13stages:
14  - stage: Build
15    displayName: Build Application
16    jobs:
17      - job: Build
18        steps:
19          - task: NodeTool@0
20            inputs:
21              versionSpec: $(NODE_VERSION)
22            displayName: Setup Node.js
23
24          - script: npm ci
25            displayName: Install dependencies
26
27          - script: npm run build
28            displayName: Build application
29
30  - stage: UnitTest
31    displayName: Unit Tests
32    dependsOn: Build
33    jobs:
34      - job: Unit
35        steps:
36          - script: npm ci
37          - script: npm run test:unit -- --reporter=junit --output=test-results/unit.xml
38            displayName: Run unit tests
39
40          - task: PublishTestResults@2
41            displayName: Publish unit test results
42            inputs:
43              testResultsFormat: JUnit
44              testResultsFiles: test-results/unit.xml
45              testRunTitle: Unit Tests
46            condition: always()
47
48  - stage: E2ETest
49    displayName: End-to-End Tests
50    dependsOn: Build
51    jobs:
52      - job: E2E
53        steps:
54          - script: npm ci
55          - script: npx playwright install --with-deps chromium
56            displayName: Install Playwright browsers
57
58          - script: npx playwright test --reporter=junit,html
59            displayName: Run E2E tests
60            env:
61              BASE_URL: $(STAGING_URL)
62
63          - task: PublishTestResults@2
64            displayName: Publish E2E results
65            inputs:
66              testResultsFormat: JUnit
67              testResultsFiles: test-results/e2e.xml
68              testRunTitle: E2E Tests
69            condition: always()
70
71          - task: PublishPipelineArtifact@1
72            displayName: Upload test report
73            inputs:
74              targetPath: playwright-report
75              artifact: playwright-report
76            condition: always()

---

Reading pipeline results

After a pipeline run, navigate to Pipelines → [Pipeline] → [Run].

The summary shows:

• Stage results (pass/fail for each stage)
• Test tab: total tests, passed, failed, skipped
• Artifacts: downloadable reports

Click the Tests tab to see:

• All test cases sorted by status
• Failed tests with error messages
• Duration per test
• Test history (was this test failing before?)

For each failed test, click it to see:

• The exact assertion that failed
• Stack trace
• Any attached screenshots (if your test framework captures them)

---

Investigating pipeline failures

When a pipeline fails, QA engineers need to distinguish between:

1. New product bug — a test that was passing now fails due to a code change
2. Environment issue — the staging environment has a problem unrelated to the code
3. Test flakiness — the test sometimes passes and sometimes fails for non-deterministic reasons
4. Test code bug — the test itself is wrong or outdated

Diagnosis steps:

1. Check: was this test passing before this commit?
   → Yes: Likely a regression introduced by this commit
   → No: Was it failing for multiple commits? Likely flaky or environment

2. Click the failed test → read the error message and stack trace
   → Assertion error: test expects X but got Y → product bug or test bug
   → Timeout error: test waited for element that never appeared → flaky or slow environment
   → Connection error: can't reach the application → environment issue

3. Re-run the failed job (click Re-run failed jobs)
   → Passes on re-run: flaky test
   → Still fails: product bug or test bug

---

Scheduled nightly regression

Run the full regression suite overnight:

YAML
1schedules:
2  - cron: "0 1 * * 1-5"   # 1 AM Monday-Friday UTC
3    displayName: Nightly regression
4    branches:
5      include:
6        - main
7    always: true            # Run even if no code change
8
9trigger: none               # Don't run on push (handled by PR pipeline)

Set up email notifications: Pipeline → Notifications → Add subscription for "Build fails" and "Build succeeds after failure".

---

Common errors and fixes

Error: Pipeline fails with "npm: command not found" Fix: Add the NodeTool@0 task before any npm commands to install Node.js on the agent.

Error: PublishTestResults task finds no files Fix: Check the glob pattern in testResultsFiles. Run find . -name "*.xml" in a preceding script step to debug the actual file path.

Error: E2E tests fail with "Browser not found" Fix: Add npx playwright install --with-deps chromium before the test script. Microsoft-hosted agents don't have Playwright browsers pre-installed.

Error: STAGING_URL is undefined in the test script Fix: Declare the variable in Pipeline → Variables or Library → Variable Groups. Reference it in YAML as $(STAGING_URL) and pass it to scripts as env: BASE_URL: $(STAGING_URL).

Error: Pipeline runs but tests never execute Fix: Check whether npm test or npx playwright test exits successfully even with 0 tests. Verify the test file discovery pattern in your test config.