Chapter 12 of 32

DevOps & CI/CD

Agile methods (see Agile Methodology, Scrum and Kanban & Extreme Programming) made development teams fast: working software every one or two weeks. But for years, that software then sat waiting. A separate operations team had to install it on servers, often by hand, once a month or once a quarter, and every release night was stressful. Developers were rewarded for changing the system; operations was rewarded for keeping it stable. The two goals pulled in opposite directions.

DevOps is the movement that removed this wall. CI/CD (continuous integration and continuous delivery/deployment) is the automated pipeline that makes it work in practice. Together they explain how companies like Amazon, Netflix, Flipkart or Zerodha can release changes many times a day without breaking things every time.

DevOps is a culture and set of practices that brings software development (Dev) and IT operations (Ops) together, with shared responsibility, heavy automation and fast feedback, so that changes can be delivered to users quickly, frequently and reliably.

DevOps is not a tool, a job title or a separate team by itself. It is a way of working. Tools such as Git, Jenkins, GitHub Actions, Docker and Kubernetes help, but buying them does not make an organization "do DevOps".


Where DevOps Came From

  • 2009: At the O'Reilly Velocity conference, John Allspaw and Paul Hammond of Flickr gave a famous talk, "10+ Deploys per Day: Dev and Ops Cooperation at Flickr".

  • 2009: Patrick Debois organized the first DevOpsDays conference in Ghent, Belgium. The hashtag #devops gave the movement its name.

  • 2010: Jez Humble and David Farley published Continuous Delivery, which described deployment pipelines in detail.

  • 2013 onwards: Books such as The Phoenix Project and The DevOps Handbook, and the yearly DORA research reports, spread the ideas widely.

The problem it solves

Traditional ("wall of confusion")

  Developers  --- "here's the release, good luck" --->  Operations
  goal: ship features                                   goal: stability
  measured by: features done                            measured by: uptime
  result: big, rare, risky releases; blame when things fail

DevOps

  One team (or closely working teams) owns the service end to end:
  build it, test it, deploy it, run it, watch it, fix it.
  result: small, frequent, low-risk releases; shared learning

Principles of DevOps

CALMS

A popular way to remember the pillars of DevOps is CALMS (originally CAMS from John Willis and Damon Edwards; Jez Humble added the L):

LetterPillarMeaning
CCultureShared ownership, trust, blameless post-incident reviews.
AAutomationBuilds, tests, deployments and infrastructure are automated, not manual.
LLeanSmall batches, limit WIP, remove waste and waiting.
MMeasurementMeasure delivery performance and system health; decide with data.
SSharingShare knowledge, tools and responsibilities across Dev and Ops.

The Three Ways

The DevOps Handbook describes three underlying principles:

  1. Flow: make work move fast from development to production (small batches, automation, visible work).

  2. Feedback: get fast information back from production and from tests, so problems are found and fixed early.

  3. Continual learning and experimentation: a culture where people can try things, learn from failures and improve.

The DevOps lifecycle

DevOps is usually drawn as an infinity loop, showing that delivery never ends:

        Plan → Code → Build → Test
         ↑                       ↓
     Monitor                  Release
         ↑                       ↓
      Operate  ←  ←  ←  ←  ←  Deploy

   Dev side: Plan, Code, Build, Test
   Ops side: Release, Deploy, Operate, Monitor
   Feedback from Monitor goes back into Plan.

Continuous Integration (CI)

Before CI, developers worked on separate branches for weeks and then tried to merge everything at the end. This "integration hell" produced huge conflicts and bugs nobody could trace. The term "continuous integration" was used by Grady Booch in 1991 and became a core practice of XP (see Kanban & Extreme Programming).

Continuous Integration is the practice of developers merging their code changes into a shared mainline frequently (at least daily), where each merge automatically triggers a build and a suite of automated tests, so that integration problems are detected within minutes.

The CI cycle

  1. Developer pulls the latest mainline and makes a small change.

  2. Runs tests locally.

  3. Pushes the change (or opens a pull request).

  4. The CI server checks out the code, builds it and runs all automated tests (and often linters and security scans).

  5. If anything fails, the team is notified and fixing the build becomes top priority.

Key CI practices

  • Keep everything in version control (code, tests, build scripts, configuration), so any version of the system can be rebuilt exactly.

  • Automate the build: one command must build the whole system.

  • Make the build self-testing: an automated test suite runs on every build.

  • Commit to the mainline often, so changes stay small (short-lived branches or trunk-based development).

  • Keep the build fast: XP's guideline is a ten-minute build.

  • Fix a broken build immediately. A red build blocks everyone.

Worked example: a CI pipeline that catches a bug

Suppose a library app has a late-fee function: ₹2 per day for the first 7 days, ₹5 per day after that. So 10 days late costs 2 × 7 + 5 × 3 = ₹29.

# fees.py
def late_fee(days_late):
    """Library late fee: Rs 2 per day for the first 7 days, Rs 5 per day after that."""
    if days_late < 0:
        raise ValueError("days_late cannot be negative")
    if days_late <= 7:
        return 2 * days_late
    return 2 * 7 + 5 * (days_late - 7)
# test_fees.py
import unittest
from fees import late_fee

class TestLateFee(unittest.TestCase):
    def test_on_time(self):
        self.assertEqual(late_fee(0), 0)

    def test_first_week(self):
        self.assertEqual(late_fee(7), 14)

    def test_after_first_week(self):
        self.assertEqual(late_fee(10), 29)

    def test_negative(self):
        with self.assertRaises(ValueError):
            late_fee(-1)

if __name__ == "__main__":
    unittest.main()

A CI workflow for GitHub Actions, stored as .github/workflows/ci.yml, runs these tests on every push and pull request:

name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.14"
      - name: Run unit tests
        run: python -m unittest -v

On a correct commit, the test step prints:

Output:

test_after_first_week (test_fees.TestLateFee.test_after_first_week) ... ok
test_first_week (test_fees.TestLateFee.test_first_week) ... ok
test_negative (test_fees.TestLateFee.test_negative) ... ok
test_on_time (test_fees.TestLateFee.test_on_time) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Now a developer "simplifies" the last line to return 2 * 7 + 5 * days_late and pushes. Within a minute the pipeline turns red:

Output:

F...
======================================================================
FAIL: test_after_first_week (test_fees.TestLateFee.test_after_first_week)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../test_fees.py", line 13, in test_after_first_week
    self.assertEqual(late_fee(10), 29)
AssertionError: 64 != 29

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=1)

python -m unittest exits with status 1, the CI job fails, and the pull request shows a red cross, so the bug never reaches users who would have been overcharged ₹64 instead of ₹29. That is the whole value of CI: fast, automatic feedback on every change.


Continuous Delivery vs Continuous Deployment

The "CD" in CI/CD has two meanings, and exams love to ask the difference.

Continuous Delivery: every change that passes the automated pipeline is kept in a releasable state and can be deployed to production at any time, but the actual production release is triggered by a human decision (often one click).

Continuous Deployment: every change that passes the automated pipeline is released to production automatically, with no manual approval step.

                  Continuous Integration
              +-----------------------------+
 commit → build → unit tests → integration tests → staging → [approve?] → production
              +--------------------------------------------+
                         Continuous Delivery  (human clicks "deploy")
              +-------------------------------------------------------------+
                         Continuous Deployment (no manual gate)
AspectContinuous IntegrationContinuous DeliveryContinuous Deployment
What is automatedBuild + tests on every mergeEverything up to a production-ready releaseEverything, including release to production
Release to productionNot coveredManual decisionAutomatic
RequiresAutomated tests, shared mainlineCI + automated deployment to test/staging environmentsContinuous Delivery + very high confidence in tests and monitoring
Typical usersAlmost every teamBanks, regulated industries, mobile apps (app store review)Web companies with strong automation

Note the nesting: continuous deployment ⊃ continuous delivery ⊃ continuous integration.

The deployment pipeline

A deployment pipeline is the automated sequence of stages that each change goes through. Cheap, fast checks run first; slower ones later. A change that fails any stage stops there.

StageTypical checksTypical time
Commit stageCompile, unit tests, linting, static analysisMinutes
Automated acceptanceIntegration and end-to-end tests in a test environmentTens of minutes
Non-functional checksPerformance, security scans, dependency vulnerability scansVaries
StagingProduction-like environment, smoke tests, optional manual exploratory testingVaries
ProductionRelease using a safe strategy, then monitoringMinutes

A golden rule: build the artifact once (a binary, package or container image) and promote the same artifact through all stages. Rebuilding for each environment risks testing one thing and shipping another.


Safe Release Strategies

StrategyHow it worksAdvantageCost
RollingReplace servers running the old version a few at a timeNo extra infrastructureTwo versions run together for a while; rollback is slower
Blue-greenTwo identical environments; deploy to the idle one (green), then switch traffic from blue to greenInstant switch and instant rollbackDouble the environment during release
CanarySend a small percentage of users (say 5%) to the new version, watch metrics, then increaseProblems affect few usersNeeds good monitoring and traffic routing
Feature flagsNew code is deployed but hidden behind a switch, turned on for chosen users laterSeparates deploy from release; easy to turn offOld flags must be cleaned up, or the code gets messy

Other Core DevOps Practices

  • Infrastructure as Code (IaC): servers, networks and databases are described in version-controlled files (tools such as Terraform, OpenTofu, Ansible, AWS CloudFormation) instead of being configured by hand. Environments become reproducible and reviewable like code.

  • Containers: packaging an application with its dependencies (for example as a Docker image) so it runs the same on a laptop, test server and production. Orchestrators such as Kubernetes run and scale them.

  • Monitoring and observability: logs, metrics and traces from production, with alerts. Without monitoring, frequent releases are blind releases.

  • Blameless postmortems: after an incident, the team analyses what in the system allowed the failure, instead of blaming a person.

  • DevSecOps / shift-left security: security checks (dependency scanning, secret scanning, static analysis) run inside the pipeline from the start, not as a final gate.


Measuring DevOps: The DORA Metrics

The DORA research programme (DevOps Research and Assessment, now part of Google Cloud) has studied thousands of teams since 2014. It started with four "key" metrics. In 2023 the recovery metric was renamed (from mean time to restore, MTTR, to failed deployment recovery time), and in 2024 a fifth metric, rework rate, was added. The current model has five metrics in two groups:

GroupMetricQuestion it answers
ThroughputChange lead timeHow long from a commit to that change running in production?
ThroughputDeployment frequencyHow often do we deploy to production?
ThroughputFailed deployment recovery timeWhen a deployment causes a failure, how quickly do we recover?
InstabilityChange fail rateWhat percentage of deployments cause a failure needing immediate fixing?
InstabilityDeployment rework rateWhat share of deployments are unplanned, done to fix a problem in production?

The most important finding of this research is that speed and stability are not a trade-off. The best-performing teams deploy more often and have lower failure rates, because small, frequent, automated changes are easier to test, understand and roll back than big, rare ones.

Small worked example

In one month a team made 40 production deployments. 4 of them caused incidents that needed a hotfix or rollback.

Deployment frequency ≈ 40 per month (about 2 per working day)
Change fail rate     = 4 ÷ 40 = 10%

DevOps vs Agile

AspectAgileDevOps
Main concernHow the team builds the right product iterativelyHow changes flow safely from code to production and back as feedback
Bridges the gap betweenCustomer and development teamDevelopment and operations
Feedback fromCustomer, Sprint ReviewsPipeline results, production monitoring
Key practicesIterations, backlog, stand-ups, retrospectivesCI/CD, IaC, monitoring, automated release
RelationshipComplementary: agile delivers working increments oftenComplementary: DevOps extends agile thinking from "done in development" to "running well in production"

Common Mistakes

  • Thinking DevOps is a tool or a team. Creating a "DevOps team" that sits between Dev and Ops can simply add a third silo.

  • Calling a scheduled nightly build "CI". CI means integrating to the mainline frequently, with automated tests on every integration.

  • Mixing up continuous delivery and continuous deployment. The difference is only the manual approval before production.

  • Automating a slow, flaky test suite as-is. Flaky tests teach people to ignore red builds.

  • Rebuilding the artifact per environment. Build once, promote the same artifact.

  • Deploying often without monitoring. Fast releases need fast detection and rollback.


Interview and Exam Questions

Q1. What is the difference between continuous delivery and continuous deployment?

In continuous delivery every passing change is releasable, but a person decides when to release to production. In continuous deployment every passing change goes to production automatically.

Q2. What does CALMS stand for?

Culture, Automation, Lean, Measurement, Sharing.

Q3. Name the DORA software delivery metrics.

Throughput: change lead time, deployment frequency, failed deployment recovery time. Instability: change fail rate, deployment rework rate. (Older sources list the four keys: deployment frequency, lead time for changes, change failure rate, mean time to restore.)

Q4. What is blue-green deployment?

Two identical production environments. The new version is deployed to the idle one and tested; then a router switches all traffic to it. Rollback is just switching back.

Q5. Why should the build be fixed immediately when CI fails?

A broken mainline blocks everyone: new changes cannot be verified, and errors pile up on top of each other, making each harder to find. CI only gives fast feedback if the mainline is normally green.

Q6. What is Infrastructure as Code?

Defining infrastructure (servers, networks, databases, permissions) in machine-readable files kept in version control and applied by tools, so environments are reproducible, reviewable and consistent.

Q7. A team deployed 25 times last month; 5 deployments caused failures. Change fail rate?

5 ÷ 25 = 20%.


Summary

  • DevOps unites development and operations through culture, automation, lean flow, measurement and sharing (CALMS).

  • CI: integrate to a shared mainline at least daily; every integration is built and tested automatically.

  • Continuous delivery: always releasable, human triggers the release. Continuous deployment: release is automatic.

  • A deployment pipeline runs fast checks first and promotes one artifact through stages.

  • Safe releases: rolling, blue-green, canary, feature flags.

  • DORA metrics measure throughput and instability; high performers are both fast and stable.

Small changes, automatically checked and frequently released, are safer than big changes released rarely.