Track Test Coverage & Defect Metrics: DevOps

Measuring quality without metrics is guesswork. Azure DevOps provides the data — test results, bug work items, pipeline runs — but it takes deliberate configuration to turn that data into actionable metrics.

---

Code coverage in Azure Pipelines

Publishing coverage from Jest

YAML
1- script: |
2    npm test -- \
3      --coverage \
4      --coverageReporters=cobertura \
5      --coverageDirectory=coverage
6  displayName: Run tests with coverage
7
8- task: PublishCodeCoverageResults@1
9  displayName: Publish code coverage
10  inputs:
11    codeCoverageTool: Cobertura
12    summaryFileLocation: coverage/cobertura-coverage.xml
13    reportDirectory: coverage
14  condition: always()

Publishing coverage from pytest

YAML
1- script: |
2    pip install pytest-cov
3    pytest tests/ \
4      --cov=src \
5      --cov-report=xml:coverage.xml \
6      --cov-fail-under=80
7  displayName: Run tests with coverage
8
9- task: PublishCodeCoverageResults@1
10  inputs:
11    codeCoverageTool: Cobertura
12    summaryFileLocation: coverage.xml
13  condition: always()

The pipeline summary now shows a Code Coverage tab with line, branch, and function coverage percentages.

Coverage gate (fail if below threshold)

YAML
1# In Jest config:
2coverageThreshold:
3  global:
4    lines: 80
5    branches: 75
6    functions: 80
7
8# In pytest:
9--cov-fail-under=80

If coverage drops below the threshold, the pipeline fails — preventing coverage regression from being merged.

---

Requirement coverage metrics

Query: uncovered user stories

Find user stories with no linked test cases:

Work Item Type = User Story
AND Iteration = @CurrentIteration
AND State = Active
AND [Not linked to: Test Case with Tests link]

Save as a chart widget on your dashboard. Any bar in this chart represents a testing gap.

Requirement coverage formula

Coverage % = (Stories with all test cases passed / Total stories) × 100

Azure DevOps calculates this automatically in the Requirements tab of a test plan.

---

Defect metrics

Defect density query

Total bugs found:
  Type = Bug
  AND Created >= @StartOf("sprint")

Bugs by component:
  Type = Bug AND Area Path UNDER "App/Checkout"
  → count gives defect density for checkout module

Bug escape rate dashboard

Bugs found in testing:
  Type = Bug AND Tags CONTAINS "found-in-testing"
  AND Created >= @StartOf("month")

Bugs found in production:
  Type = Bug AND Tags CONTAINS "found-in-production"
  AND Created >= @StartOf("month")

Escape rate = production bugs / (testing bugs + production bugs) × 100

Tag every bug with found-in-testing or found-in-production when it's created. This enables escape rate tracking over time.

---

Building a metrics scorecard

Create a monthly metrics export:

Q3 2025 QA Metrics Scorecard

Test Coverage:
  Requirement coverage: 94%  (target: >90%) ✓
  Code coverage:        82%  (target: >80%) ✓
  Automated %:          68%  (target: >65%) ✓

Defect Metrics:
  Bugs found (QA):      47
  Bugs found (prod):     3
  Escape rate:           6%  (target: <10%) ✓
  Defect density:       1.4  (target: <2.0) ✓

Pipeline Health:
  Avg pass rate:        96.8% (target: >95%) ✓
  Flaky test count:        4  (target: <5)  ✓
  Mean pipeline time:   11m   (target: <15m) ✓

---

Common errors and fixes

Error: Coverage report shows 0% despite tests running Fix: The test runner must be configured to instrument source files. Check that the --cov=src or --coverage flag points to your source directory, not the test directory.

Error: Cobertura XML not found by PublishCodeCoverageResults Fix: The path in summaryFileLocation must be relative to $(System.DefaultWorkingDirectory). Use $(System.DefaultWorkingDirectory)/coverage/cobertura-coverage.xml for explicit paths.

Error: Bug escape rate query includes testing environment bugs Fix: Standardise the tagging: use environment:staging and environment:production tags. Filter queries by environment tag to separate test environment bugs from production escapes.