Risk-Based Testing in Azure DevOps

Risk-based testing is about focus: testing the parts of your system that matter most first. In Azure DevOps, this means configuring pipelines, tagging test cases by risk level, and using coverage data to make confident release decisions.

---

Risk assessment in Azure Boards

Start by assessing risk at the user story level. Add a custom field Risk Level to the User Story work item type:

1. Go to Organisation Settings → Process → [Your process] → Work item types → User Story
2. Add field: Risk Level (Picklist: Critical, High, Medium, Low)

Now every user story has a risk designation. Test cases linked to Critical and High stories run first.

---

Risk classification criteria

---

Tagging test cases by risk

Tag test cases in Azure Test Plans:

risk:critical  — payment flows, auth, data integrity
risk:high      — core user journeys, key integrations
risk:medium    — secondary features
risk:low       — cosmetic, rarely used

Create a query-based suite for each risk level:

Risk Critical Suite:
  Type = Test Case AND Tags CONTAINS "risk:critical"

Risk High Suite:
  Type = Test Case AND Tags CONTAINS "risk:high"

---

Risk-ordered pipeline execution

Run critical and high-risk tests first, fail fast:

YAML
1stages:
2  # Critical risk tests — run on every PR
3  - stage: CriticalTests
4    displayName: Critical Risk Tests
5    jobs:
6      - job: Critical
7        steps:
8          - script: npx playwright test --grep "@critical"
9            displayName: Critical path tests
10
11  # High risk — run on merge to main
12  - stage: HighRiskTests
13    displayName: High Risk Tests
14    dependsOn: CriticalTests
15    condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
16    jobs:
17      - job: HighRisk
18        steps:
19          - script: npx playwright test --grep "@high"
20            displayName: High risk tests
21
22  # Full regression — nightly
23  - stage: FullRegression
24    displayName: Full Regression
25    dependsOn: HighRiskTests
26    condition: eq(variables['Build.Reason'], 'Schedule')
27    jobs:
28      - job: All
29        steps:
30          - script: npx playwright test
31            displayName: All tests

---

Risk coverage report for release decisions

Before releasing, produce a risk coverage summary:

Sprint 9 Release — Risk Coverage Report

Critical Risk Areas (Payment, Auth):
  Test cases:  24
  Executed:    24  (100%)
  Passed:      24  (100%)
  Open bugs:    0
  Decision:    ✓ CLEAR

High Risk Areas (Checkout, Profile):
  Test cases:  42
  Executed:    42  (100%)
  Passed:      41  (97.6%)
  Open bugs:    1  (P2 — discount rounding edge case)
  Decision:    ✓ RELEASE WITH KNOWN DEFECT (documented)

Medium Risk Areas (Admin, Reports):
  Test cases:  28
  Executed:    28  (100%)
  Passed:      28  (100%)
  Decision:    ✓ CLEAR

Low Risk Areas (UI polish):
  Test cases:  14
  Executed:    10  (71%)  ← Not fully executed
  Passed:      10  (100% of executed)
  Decision:    ✓ ACCEPTABLE (low risk, not fully tested)

Overall:
  Critical/High risk: 100% coverage ✓
  Release recommendation: APPROVED

---

Common errors and fixes

Error: Risk tags aren't consistent across the team Fix: Enforce tag standardisation in the Definition of Done. Before a test case moves to "Ready" state, it must have a risk: tag. Add this to your team's work item workflow validation.

Error: Critical tests run but high-risk tests are skipped even on main branch merges Fix: Check the condition on the HighRiskTests stage. ne(variables['Build.Reason'], 'PullRequest') should allow it to run on push to main. Verify with Build.Reason variable values: IndividualCI (push), PullRequest, Schedule.

Error: Release decision is made without checking all critical test results Fix: Add a required approval gate that references the risk coverage report. The approver must confirm they've reviewed the critical coverage before approving the production deployment.