CI/CD Pipeline Design: From Baseline to Expert¶

Estimated time to read: 13 minutes

Continuous Integration and Continuous Delivery (CI/CD) is the engine that drives modern software delivery. It is not a single tool or a single workflow it is an evolving set of practices that mature alongside your organisation's engineering capability.

The OpsAtScale Maturity Assessment maps the progression of CI/CD practices across multiple dimensions: Continuous Integration, Deployment Automation, Continuous Testing, Security, Risk Mitigation, and Tooling. This guide uses that framework as the backbone, expanding each level into practical implementation guidance.

For broader context on how CI/CD sits within DevOps, SRE, and Platform Engineering, see DevOps, SRE, and Platform Engineering: A Comparative Guide.

What Is a CI/CD Pipeline?¶

A CI/CD pipeline is an automated sequence of stages that carries a code change from a developer's machine to production. It enforces quality, security, and reliability gates at every step, replacing manual gate-keeping with deterministic, repeatable automation.

The pipeline is the primary delivery mechanism for the DORA metrics that your organisation tracks: Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Failed Deployment Recovery Time. A poorly designed pipeline degrades every one of those metrics.

graph LR
    A[Code Commit] --> B[Build & Unit Test]
    B --> C[SAST & Code Quality]
    C --> D[Integration & Contract Tests]
    D --> E[Artefact Build]
    E --> F[Deploy to Test Env]
    F --> G[DAST & Security Scan]
    G --> H[Performance Tests]
    H --> I[Deploy to Production]
    I --> J[Monitoring & Feedback]

The Five Stages of a Production-Grade Pipeline¶

Stage 1 Source & Trigger¶

The pipeline starts when a developer pushes code. How and when the pipeline fires matters.

The Git Trunk-Based Development model is the strongest foundation for fast CI: short-lived branches, frequent integration into main, and feature flags to decouple deployment from release.

Stage 2 Build & Unit Test¶

The build stage compiles the application and runs the fastest, most reliable tests you have.

Rules for this stage: - Build only once. Promote the same artefact through environments never rebuild per environment. - Unit tests must complete in under 5 minutes or they will be skipped under pressure. - A failing unit test must break the pipeline immediately no optional tests at this stage. - Build status notification must reach the committing developer automatically.

name: CI Build

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up JDK 21
        uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'

      - name: Build and Unit Test
        run: ./mvnw clean verify

      - name: Upload Build Artefact
        uses: actions/upload-artifact@v4
        with:
          name: app-jar
          path: target/*.jar

See GitHub Actions for Starters for the full workflow syntax reference.

Stage 3 Code Quality & SAST¶

Code quality gates and Static Application Security Testing (SAST) run after the build succeeds. They give developers fast feedback on structural issues and known vulnerability patterns before any deployment happens. This is the shift-left security principle in practice.

GitHub Dependabot automates the dependency scanning step and raises pull requests for vulnerable packages automatically.

Stage 4 Test Automation Pyramid¶

Modern CI pipelines implement a test automation pyramid: many fast unit tests at the base, fewer integration tests in the middle, and a small number of slow end-to-end tests at the top.

         /\
        /  \  ← E2E / UI Tests (slow, brittle  few)
       /----\
      /      \  ← Integration & Contract Tests (medium speed)
     /--------\
    /          \  ← Unit Tests (fast, stable  many)
   /____________\

Mocked vs Real Environments¶

At Intermediate maturity, integration tests run against mocked dependencies using tools like WireMock. At Advanced maturity, services can be tested in isolation and in parallel each microservice runs only the tests relevant to its own commit, dramatically reducing pipeline time.

Stage 5 Deployment & Release¶

Deployment is the act of putting code into an environment. Release is the act of exposing that functionality to users. Separating these two concerns unlocks the most powerful delivery patterns.

Deployment Strategies¶

graph TD
    A[Code in Main] --> B{CI Gates Pass?}
    B -- No --> C[Notify Developer - Fail Fast]
    B -- Yes --> D[Deploy to Dev/Test]
    D --> E{Integration Tests Pass?}
    E -- No --> C
    E -- Yes --> F[Deploy to Staging - Canary 10%]
    F --> G{Canary Metrics OK?}
    G -- No --> H[Automated Rollback]
    G -- Yes --> I[Full Production Rollout]
    I --> J[Monitor SLOs & Alert]

Security Integration DevSecOps in the Pipeline¶

Integrating security into the pipeline is the foundation of DevSecOps. The Software Supply Chain guide explains the threat landscape; this section covers how to mitigate those threats inside the pipeline.

The Security Lane What Runs Where¶

Commit → Build → SAST → Dependencies → Artefact Sign → Deploy → DAST → Monitor
   │        │       │          │               │              │        │
   │        │       │          │               │              │        └─ Runtime protection (Falco, WAF)
   │        │       │          │               │              └─ Automated pen-test (OWASP ZAP)
   │        │       │          │               └─ Container image signing (Cosign, Sigstore)
   │        │       │          └─ CVE scan (Snyk, Dependabot, OWASP)
   │        │       └─ Secret detection (GitLeaks), code pattern scanning (Semgrep)
   │        └─ Container image scan (Trivy)
   └─ Pre-commit hooks (prevent secrets at source)

Shift-Left Security Checklist¶

• Pre-commit hooks: block secret commits before they reach the remote
• SAST gate: fails build on high/critical findings
• Dependency CVE gate: fails build on CVSS ≥ 7.0 unpatched vulnerabilities
• Container image scan: scan base image and application layers (Trivy)
• Artefact signing: sign images with Cosign; verify at deployment time
• DAST in staging: automated scan of running application before production promotion
• Audit trail: every pipeline run produces a signed, immutable log of all gates and their results

The Security & Compliance maturity levels in the OpsAtScale framework map directly to the sophistication of these controls: from manual penetration testing at Baseline to automated remediation at Cutting-Edge.

Secrets Management in Pipelines¶

Secrets API keys, database passwords, certificates are the most abused vector in CI/CD. The wrong way is to store them in environment variables, .env files, or in source control. The right approach depends on your maturity level see the Secrets Management: Protecting the Keys to your Kingdom guide for a full deep dive.

# GitHub Actions  OIDC to AWS, no stored keys
permissions:
  id-token: write
  contents: read

- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789:role/GitHubActionsRole
    aws-region: eu-west-1

Artefact Management¶

Every pipeline should produce a single artefact, built once, promoted through environments unchanged. Rebuilding per environment introduces drift.

Key practice: tag every artefact with the Git commit SHA that produced it. This creates end-to-end traceability from a running container back to the exact line of code and the CI run that built it which is the full audit trail requirement at Expert maturity in the Risk Mitigation dimension.

Test Data Management¶

Test data is the silent blocker of pipeline maturity. A pipeline that cannot provision its own test data is permanently dependent on a shared, flaky environment.

Tools: Faker (synthetic data), Flyway/Liquibase (schema migrations), Testcontainers (ephemeral databases per test run).

Pipeline Design Principles¶

These principles are constant regardless of the tooling you choose.

1. Fail Fast¶

The pipeline must fail as early as possible. A build that spends 30 minutes running integration tests before revealing a compilation error has wasted developer time. Sequence your stages from fastest-to-slowest always.

2. Build Once, Promote Many¶

Build one artefact. Use environment-specific configuration injected at deployment time. Never re-build for production.

3. Immutable Infrastructure¶

Every environment is created fresh from code. No manual changes to running environments. If a server needs to be patched, the AMI or container image is rebuilt and redeployed not SSH'd into.

4. Everything in Source Control¶

Pipeline definitions, infrastructure code, test data seeds, configuration templates, and secret references (not the secrets themselves) all live in source control. The pipeline is the documentation.

5. Trunk Over Branches¶

Long-lived feature branches cause integration debt. Use trunk-based development with feature flags to keep integration continuous. The Git Trunk-Based Development guide covers this in depth.

6. Observability from Day One¶

Every pipeline run emits structured events. Deployment frequency, lead time, failure rate, and recovery time are measured automatically by the pipeline not via manual spreadsheets. This is how DORA metrics become operational, not theoretical.

Maturity Self-Assessment CI/CD Pipeline¶

Use this checklist to identify your current stage and what to target next:

✅ Baseline¶

• Builds run on a CI server (not developer machines)
• Unit tests run automatically on commit
• Build artefacts are stored and managed
• Committer is notified of build status

✅ Novice → Intermediate¶

• Source control triggers build via webhook (no polling)
• Code coverage is measured and breaks CI below threshold
• SAST integrated and results published
• Mocked integration tests run in CI
• Deployments to test environments are push-button

✅ Intermediate → Advanced¶

• SAST and dependency CVE gates break the pipeline
• Container is the main build artefact
• DAST runs against deployed application pre-release
• Code quality gates enforced (SonarQube, CodeClimate)
• Zero-downtime deployments to production
• Database schema changes are version controlled and automated

✅ Advanced → Expert¶

• Gated commits to main based on CI result (no broken main)
• Dynamic secrets via OIDC no stored cloud credentials
• Canary or blue/green deployments via traffic shifting
• Performance and security tests automated pre-release
• Full audit trail from commit to production artefact
• DORA metrics collected automatically from pipeline events

✅ Expert → Cutting-Edge¶

• Continuous deployment every passing commit ships to production
• Production testing via synthetic monitors and chaos experiments
• Security engine monitors and auto-remediates production state
• Cross-team pipeline data mining and business impact correlation

Tool Reference Pipeline Toolchain¶

The OpsAtScale Tools maturity dimension maps tooling choices to capability levels. Here is a consolidated reference:

CI Orchestration¶

Code Quality & SAST¶

Container & Dependency Scanning¶

Secret Management¶

Deployment Automation¶

Related Reading¶

• OpsAtScale Maturity Assessment the full delivery maturity framework this guide is based on
• DevOps, SRE & Platform Engineering how CI/CD fits within the three disciplines
• DORA & SPACE Metrics measuring the output of your pipeline
• Software Supply Chain Security securing what flows through the pipeline
• GitHub Actions for Starters hands-on pipeline automation
• GitHub Dependabot automating dependency security
• Git Trunk-Based Development the branching model that CI/CD depends on
• Technical Debt what accumulates when pipelines are skipped