Parallel Test Execution in Azure DevOps

A 90-minute test suite run sequentially is a 15-minute test suite run across 6 parallel agents. Parallel execution is the single highest-impact optimisation you can make to a slow CI pipeline.

---

Three parallelism strategies

---

Strategy 1: Job parallelism

Run different test types simultaneously:

YAML
1stages:
2  - stage: Test
3    jobs:
4      # All three jobs start at the same time
5      - job: Unit
6        displayName: Unit tests (2 min)
7        steps:
8          - script: npm test:unit
9
10      - job: API
11        displayName: API tests (5 min)
12        steps:
13          - script: npm test:api
14
15      - job: E2E
16        displayName: E2E tests (15 min)
17        steps:
18          - script: npx playwright test

Total time: 15 minutes (the slowest job) instead of 22 minutes (sequential).

---

Strategy 2: Matrix strategy

Run the same tests on multiple configurations simultaneously:

YAML
1jobs:
2  - job: CrossBrowser
3    displayName: Cross-browser E2E
4    strategy:
5      matrix:
6        Chrome:
7          BROWSER: chromium
8        Firefox:
9          BROWSER: firefox
10        Safari:
11          BROWSER: webkit
12      maxParallel: 3
13
14    steps:
15      - script: npm ci
16      - script: npx playwright install --with-deps $(BROWSER)
17      - script: npx playwright test --project=$(BROWSER)
18        displayName: Run $(BROWSER) tests
19      - task: PublishTestResults@2
20        inputs:
21          testResultsFiles: results/$(BROWSER)-results.xml
22          testRunTitle: E2E — $(BROWSER) — $(Build.BuildNumber)
23        condition: always()

All three browsers run simultaneously. Total time: time of slowest browser.

Matrix for multiple environments

YAML
1strategy:
2  matrix:
3    Staging:
4      ENV_NAME: staging
5      BASE_URL: https://staging.app.com
6    UAT:
7      ENV_NAME: uat
8      BASE_URL: https://uat.app.com
9  maxParallel: 2

---

Strategy 3: Test sharding (Playwright)

Split a single test suite across multiple agents:

YAML
1jobs:
2  - job: Shard
3    strategy:
4      matrix:
5        Shard1: { SHARD_INDEX: 1, SHARD_TOTAL: 4 }
6        Shard2: { SHARD_INDEX: 2, SHARD_TOTAL: 4 }
7        Shard3: { SHARD_INDEX: 3, SHARD_TOTAL: 4 }
8        Shard4: { SHARD_INDEX: 4, SHARD_TOTAL: 4 }
9      maxParallel: 4
10
11    steps:
12      - script: npm ci
13      - script: npx playwright install --with-deps chromium
14      - script: |
15          npx playwright test \
16            --shard=$(SHARD_INDEX)/$(SHARD_TOTAL) \
17            --reporter=blob
18        displayName: Run shard $(SHARD_INDEX)/$(SHARD_TOTAL)
19      - task: PublishPipelineArtifact@1
20        inputs:
21          targetPath: blob-report
22          artifact: blob-report-$(SHARD_INDEX)
23        condition: always()
24
25  - job: MergeResults
26    dependsOn: Shard
27    condition: always()
28    steps:
29      - script: npm ci
30      - task: DownloadPipelineArtifact@2
31        inputs:
32          targetPath: all-blob-reports
33          patterns: 'blob-report-*/**'
34      - script: npx playwright merge-reports --reporter=html all-blob-reports
35      - task: PublishPipelineArtifact@1
36        inputs:
37          targetPath: playwright-report
38          artifact: playwright-merged-report

A 60-minute suite sharded across 4 agents completes in ~15 minutes.

---

Strategy 3b: pytest-xdist sharding

For Python test suites:

YAML
1jobs:
2  - job: PytestShard
3    strategy:
4      matrix:
5        Shard1: { SHARD_NUM: 0, SHARD_TOTAL: 4 }
6        Shard2: { SHARD_NUM: 1, SHARD_TOTAL: 4 }
7        Shard3: { SHARD_NUM: 2, SHARD_TOTAL: 4 }
8        Shard4: { SHARD_NUM: 3, SHARD_TOTAL: 4 }
9      maxParallel: 4
10
11    steps:
12      - script: pip install pytest pytest-xdist
13      - script: |
14          pytest tests/ \
15            --splits=$(SHARD_TOTAL) \
16            --group=$(SHARD_NUM) \
17            --junitxml=results/shard-$(SHARD_NUM).xml
18      - task: PublishTestResults@2
19        inputs:
20          testResultsFiles: results/shard-$(SHARD_NUM).xml
21          testRunTitle: Shard $(SHARD_NUM)
22        condition: always()

---

Calculating optimal shard count

Optimal shards = ceiling(Suite duration / Target duration)

Example:
  Suite takes 60 minutes
  Target: < 15 minutes
  Shards needed: ceiling(60/15) = 4 shards

Consideration: Each shard has setup overhead (~2-3 minutes for
browser install, npm install). Factor this in:
  Effective test time per shard = 60/4 = 15 min
  Plus setup: 15 + 2.5 = 17.5 min total

Beyond 8–10 shards, diminishing returns from setup overhead often outweigh gains.

---

Common errors and fixes

Error: Jobs run sequentially despite matrix strategy Fix: Check maxParallel is set. Also, you need enough agents in your agent pool. Free tier Microsoft-hosted agents allow up to 10 parallel jobs. Check Organisation Settings → Parallel jobs.

Error: Test results from shards not appearing in the pipeline Tests tab Fix: Each shard must publish its own results with PublishTestResults. Ensure the task runs with condition: always() and the file paths are correct per shard.

Error: Playwright shards have uneven distribution (one shard takes much longer) Fix: Playwright distributes tests by file. If one test file is very slow, it dominates a shard. Split large test files into smaller ones for better distribution.

Error: maxParallel is set to 4 but only 2 jobs run at once Fix: You may have reached your parallel job limit. Free tier allows 1 parallel job; paid tiers allow more. Check Organisation Settings → Billing.