End-to-End Automated QA Pipeline: DevOps

This article presents a complete, production-ready automated QA pipeline in Azure DevOps — the kind used by mature engineering teams. Every stage is justified, every gate is actionable, and the whole thing completes in under 25 minutes for a medium-sized application.

---

Pipeline overview

Stage 1:  Static (2 min)     → lint, types, secrets, SAST
Stage 2:  Unit (3 min)       → unit tests + coverage gate
Stage 3:  Integration (5 min)→ API, service tests + DB containers
Stage 4:  Build (2 min)      → Docker image + push to registry
Stage 5:  Deploy Staging (1 min) → deploy to staging environment
Stage 6:  Smoke (3 min)      → critical path tests on staging
Stage 7:  E2E (8 min)        → full Playwright regression (4 shards)
Stage 8:  Security (3 min)   → dependency scan + DAST
Stage 9:  QA Gate (manual)   → QA lead approves for production
Stage 10: Deploy Production  → blue/green production deployment

Total automated time: ~27 minutes. Manual gate: asynchronous (QA approves when ready).

---

Complete pipeline YAML

YAML
1# azure-pipelines.yml
2trigger:
3  branches:
4    include: [main]
5pr:
6  branches:
7    include: [main]
8schedules:
9  - cron: "0 1 * * 1-5"
10    displayName: Nightly regression
11    branches:
12      include: [main]
13    always: true
14
15pool:
16  vmImage: ubuntu-latest
17
18variables:
19  - group: qa-environment-secrets
20  - group: release-config
21  - name: IMAGE_NAME
22    value: $(Build.Repository.Name)
23  - name: IMAGE_TAG
24    value: $(Build.BuildId)
25  - name: isPR
26    value: ${{ eq(variables['Build.Reason'], 'PullRequest') }}
27  - name: isMain
28    value: ${{ eq(variables['Build.SourceBranch'], 'refs/heads/main') }}
29
30stages:
31  # ── 1. Static checks ─────────────────────────────────────────────────────
32  - stage: Static
33    displayName: Static Analysis
34    jobs:
35      - job: Checks
36        steps:
37          - task: NodeTool@0
38            inputs: { versionSpec: '20.x' }
39          - script: npm ci
40          - script: npm run lint -- --format=junit --output-file=results/lint.xml
41          - script: npm run typecheck
42          - script: npm audit --audit-level=high
43          - task: PublishTestResults@2
44            inputs:
45              testResultsFormat: JUnit
46              testResultsFiles: results/lint.xml
47              testRunTitle: Lint — $(Build.BuildNumber)
48            condition: always()
49
50  # ── 2. Unit tests ─────────────────────────────────────────────────────────
51  - stage: Unit
52    dependsOn: Static
53    displayName: Unit Tests
54    jobs:
55      - job: Jest
56        steps:
57          - task: NodeTool@0
58            inputs: { versionSpec: '20.x' }
59          - script: npm ci
60          - script: |
61              npm test -- \
62                --coverage \
63                --coverageReporters=cobertura \
64                --reporter=junit \
65                --outputFile=results/unit.xml
66          - task: PublishTestResults@2
67            inputs:
68              testResultsFormat: JUnit
69              testResultsFiles: results/unit.xml
70              testRunTitle: Unit — $(Build.BuildNumber)
71            condition: always()
72          - task: PublishCodeCoverageResults@1
73            inputs:
74              codeCoverageTool: Cobertura
75              summaryFileLocation: coverage/cobertura-coverage.xml
76            condition: always()
77
78  # ── 3. Integration tests ──────────────────────────────────────────────────
79  - stage: Integration
80    dependsOn: Unit
81    displayName: Integration Tests
82    jobs:
83      - job: API
84        services:
85          postgres:
86            image: postgres:16
87            env:
88              POSTGRES_PASSWORD: testpass
89              POSTGRES_DB: testdb
90            ports: ['5432:5432']
91            options: --health-cmd "pg_isready" --health-retries 5
92          redis:
93            image: redis:7-alpine
94            ports: ['6379:6379']
95        steps:
96          - task: NodeTool@0
97            inputs: { versionSpec: '20.x' }
98          - script: npm ci
99          - script: npm run db:migrate
100            env:
101              DATABASE_URL: postgresql://postgres:testpass@localhost:5432/testdb
102          - script: npm run test:integration -- --reporter=junit --outputFile=results/integration.xml
103            env:
104              DATABASE_URL: postgresql://postgres:testpass@localhost:5432/testdb
105              REDIS_URL: redis://localhost:6379
106          - task: PublishTestResults@2
107            inputs:
108              testResultsFormat: JUnit
109              testResultsFiles: results/integration.xml
110              testRunTitle: Integration — $(Build.BuildNumber)
111            condition: always()
112
113  # ── 4. Build Docker image ─────────────────────────────────────────────────
114  - stage: Build
115    dependsOn: Integration
116    condition: and(succeeded(), eq(variables.isMain, 'true'))
117    displayName: Build Image
118    jobs:
119      - job: Docker
120        steps:
121          - task: Docker@2
122            inputs:
123              command: buildAndPush
124              containerRegistry: $(ACR_SERVICE_CONNECTION)
125              repository: $(IMAGE_NAME)
126              dockerfile: Dockerfile
127              tags: |
128                $(IMAGE_TAG)
129                latest
130
131  # ── 5. Deploy to staging ──────────────────────────────────────────────────
132  - stage: DeployStaging
133    dependsOn: Build
134    displayName: Deploy Staging
135    jobs:
136      - deployment: Staging
137        environment: staging
138        strategy:
139          runOnce:
140            deploy:
141              steps:
142                - script: |
143                    az webapp config container set \
144                      --name $(STAGING_APP_NAME) \
145                      --resource-group $(RESOURCE_GROUP) \
146                      --docker-custom-image-name $(ACR_NAME).azurecr.io/$(IMAGE_NAME):$(IMAGE_TAG)
147                  displayName: Update staging container
148                  env:
149                    AZURE_SUBSCRIPTION: $(AZURE_SUBSCRIPTION)
150
151  # ── 6. Smoke tests ────────────────────────────────────────────────────────
152  - stage: Smoke
153    dependsOn: DeployStaging
154    displayName: Smoke Tests
155    jobs:
156      - job: PlaywrightSmoke
157        steps:
158          - task: NodeTool@0
159            inputs: { versionSpec: '20.x' }
160          - script: npm ci
161          - script: npx playwright install --with-deps chromium
162          - script: npx playwright test --grep @smoke
163            env:
164              BASE_URL: $(STAGING_URL)
165          - task: PublishTestResults@2
166            inputs:
167              testResultsFormat: JUnit
168              testResultsFiles: results/smoke.xml
169              testRunTitle: Smoke — $(Build.BuildNumber)
170            condition: always()
171
172  # ── 7. Full E2E (sharded) ─────────────────────────────────────────────────
173  - stage: E2E
174    dependsOn: Smoke
175    displayName: E2E Tests
176    condition: and(succeeded(), ne(variables.isPR, 'true'))
177    jobs:
178      - job: Shard
179        strategy:
180          matrix:
181            S1: { SHARD: '1/4' }
182            S2: { SHARD: '2/4' }
183            S3: { SHARD: '3/4' }
184            S4: { SHARD: '4/4' }
185          maxParallel: 4
186        steps:
187          - task: NodeTool@0
188            inputs: { versionSpec: '20.x' }
189          - script: npm ci
190          - script: npx playwright install --with-deps chromium
191          - script: npx playwright test --shard=$(SHARD) --reporter=blob
192            env:
193              BASE_URL: $(STAGING_URL)
194          - task: PublishPipelineArtifact@1
195            inputs:
196              targetPath: blob-report
197              artifact: blob-$(System.JobName)
198            condition: always()
199
200      - job: MergeAndPublish
201        dependsOn: Shard
202        condition: always()
203        steps:
204          - task: NodeTool@0
205            inputs: { versionSpec: '20.x' }
206          - script: npm ci
207          - task: DownloadPipelineArtifact@2
208            inputs:
209              targetPath: all-blobs
210              patterns: 'blob-*/**'
211          - script: npx playwright merge-reports --reporter=html,junit all-blobs
212          - task: PublishTestResults@2
213            inputs:
214              testResultsFormat: JUnit
215              testResultsFiles: results/merged.xml
216              testRunTitle: E2E Regression — $(Build.BuildNumber)
217            condition: always()
218          - task: PublishPipelineArtifact@1
219            inputs:
220              targetPath: playwright-report
221              artifact: e2e-report
222            condition: always()
223
224  # ── 8. Security ───────────────────────────────────────────────────────────
225  - stage: Security
226    dependsOn: Smoke
227    displayName: Security Scan
228    condition: and(succeeded(), ne(variables.isPR, 'true'))
229    jobs:
230      - job: OWASP
231        steps:
232          - script: |
233              docker run --rm \
234                -v $(Build.ArtifactStagingDirectory):/zap/wrk \
235                ghcr.io/zaproxy/zaproxy:stable \
236                zap-baseline.py -t $(STAGING_URL) -r zap-report.html -I
237          - task: PublishPipelineArtifact@1
238            inputs:
239              targetPath: $(Build.ArtifactStagingDirectory)/zap-report.html
240              artifact: security-report
241            condition: always()
242
243  # ── 9. QA approval gate ───────────────────────────────────────────────────
244  - stage: QAApproval
245    dependsOn: [E2E, Security]
246    displayName: QA Sign-Off
247    condition: and(succeeded(), ne(variables.isPR, 'true'))
248    jobs:
249      - job: Approval
250        pool: server
251        timeoutInMinutes: 2880   # 48 hours
252        steps:
253          - task: ManualValidation@0
254            inputs:
255              notifyUsers: $(QA_LEAD_EMAIL)
256              instructions: |
257                QA sign-off required for $(Build.BuildNumber).
258                Review: E2E results and security report.
259                Approve only if pass rate >= 95% and no critical security issues.
260
261  # ── 10. Production deployment ─────────────────────────────────────────────
262  - stage: Production
263    dependsOn: QAApproval
264    displayName: Deploy Production
265    jobs:
266      - deployment: Prod
267        environment: production
268        strategy:
269          runOnce:
270            deploy:
271              steps:
272                - script: ./scripts/deploy-production.sh $(IMAGE_TAG)
273                  displayName: Blue/green production deploy

---

Common errors and fixes

Error: Stage 6 (Smoke) fails because staging isn't ready yet after deploy Fix: Add a readiness check before smoke tests: curl --retry 10 --retry-delay 5 $(STAGING_URL)/health. This waits up to 50 seconds for the app to start.

Error: E2E shards don't merge because artifact names conflict Fix: Each shard uploads to a unique artifact name (blob-$(System.JobName)). The merge job downloads all artifacts matching blob-*/**. Ensure System.JobName is unique per shard (it is when using matrix strategy).

Error: Security stage causes pipeline to fail even when only informational findings exist Fix: Use -I flag with ZAP baseline to ignore informational alerts. Only fail on WARN or FAIL level findings.

Error: Production deployment runs even when E2E stage partially failed Fix: The QA approval gate requires human review, which catches this. But also add condition: and(succeeded('E2E'), succeeded('Security')) to the QAApproval stage for automatic enforcement.