Continuous Testing Strategy in Azure DevOps

Continuous testing means quality is checked at every stage of the delivery pipeline — not just at the end. In Azure DevOps, this is implemented through layered pipeline stages, each running an appropriate set of tests before allowing progress to the next stage.

---

The continuous testing pyramid in Azure DevOps

              ▲
             /E2E\          Stage 4: Staging validation
            /─────\         (Playwright, manual smoke)
           / Integ \        Stage 3: Integration tests
          /─────────\       (API, service, contract)
         /   Unit    \      Stage 2: Unit tests
        /─────────────\     (Jest, JUnit, pytest)
       / Static Checks \    Stage 1: Lint, types, SAST
      ───────────────────

Each layer is a stage gate. A failure at any stage stops the pipeline and provides fast feedback to the developer.

---

Stage 1: Static checks (< 2 minutes)

YAML
1- stage: StaticChecks
2  displayName: Static Analysis
3  jobs:
4    - job: Checks
5      steps:
6        - script: npm run lint
7        - script: npm run typecheck
8        - script: npm audit --audit-level=high   # Security check
9        - script: npx license-checker --onlyAllow 'MIT;Apache-2.0;BSD-3-Clause'

---

Stage 2: Unit tests (< 5 minutes)

YAML
1- stage: UnitTests
2  dependsOn: StaticChecks
3  jobs:
4    - job: Unit
5      steps:
6        - script: npm run test:unit -- --coverage --reporter=junit
7        - task: PublishTestResults@2
8          inputs:
9            testResultsFormat: JUnit
10            testResultsFiles: results/unit.xml
11          condition: always()
12        - task: PublishCodeCoverageResults@1
13          inputs:
14            codeCoverageTool: Cobertura
15            summaryFileLocation: coverage/cobertura-coverage.xml

Fail the build if coverage drops below threshold (configured in jest.config.js).

---

Stage 3: Integration tests (< 10 minutes)

YAML
1- stage: IntegrationTests
2  dependsOn: UnitTests
3  jobs:
4    - job: Integration
5      services:
6        postgres: postgres    # Service container
7        redis: redis
8      steps:
9        - script: npm run db:migrate
10        - script: npm run test:integration
11        - task: PublishTestResults@2
12          condition: always()

---

Stage 4: Deploy to staging + smoke tests (< 15 minutes)

YAML
1- stage: Staging
2  dependsOn: IntegrationTests
3  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
4  jobs:
5    - deployment: DeployStaging
6      environment: staging
7      strategy:
8        runOnce:
9          deploy:
10            steps:
11              - script: ./scripts/deploy-staging.sh
12
13    - job: SmokeTests
14      dependsOn: DeployStaging
15      steps:
16        - script: npx playwright test --grep @smoke
17          env:
18            BASE_URL: $(STAGING_URL)
19        - task: PublishTestResults@2
20          condition: always()

---

Stage 5: Full regression + approval gate

YAML
1- stage: RegressionAndApproval
2  dependsOn: Staging
3  jobs:
4    - job: FullRegression
5      steps:
6        - script: npx playwright test --workers=6
7          env:
8            BASE_URL: $(STAGING_URL)
9        - task: PublishTestResults@2
10          condition: always()
11
12    - job: QAApproval
13      dependsOn: FullRegression
14      pool: server    # Human approval — no agent needed
15      steps:
16        - task: ManualValidation@0
17          inputs:
18            instructions: |
19              Full regression: $(RegressionPassed) tests passed.
20              Review the test report before approving production deployment.
21              Link: $(System.CollectionUri)$(System.TeamProject)/_build/results?buildId=$(Build.BuildId)
22            onTimeout: reject
23            timeout: 1d   # 24 hours to approve

---

Test selection strategy

Not all tests should run on every trigger:

YAML
1# Conditional test selection
2- script: npx playwright test --grep $(TEST_GREP)
3  env:
4    TEST_GREP: ${{ if eq(variables['Build.Reason'], 'PullRequest') }}'@smoke'${{ else }}''${{ end }}

---

Quality gates for release

Before production deployment, verify:

YAML
1- script: |
2    PASS_RATE=$(cat test-results/metrics.json | jq '.passRate')
3    echo "Pass rate: $PASS_RATE%"
4    if (( $(echo "$PASS_RATE < 95" | bc -l) )); then
5      echo "##vso[task.logissue type=error]Pass rate $PASS_RATE% is below 95% threshold"
6      exit 1
7    fi
8  displayName: Quality gate check

---

Common errors and fixes

Error: Integration tests fail because database isn't ready Fix: Use service container health checks. The job won't start until the health check passes. Add --health-cmd, --health-interval, --health-retries to the container options.

Error: Manual approval gate expires during long weekends Fix: Set timeout to at least 72 hours for weekends. Also configure notifications so approvers are alerted immediately when the gate opens.

Error: Regression stage runs even when staging deployment failed Fix: Use dependsOn: DeployStaging with condition: succeeded() on the regression job. Without the condition, the job may still run when the deployment job fails.