CI/CD Is a Production Dependency: Break-Glass Deploy Paths for the Next CI/CD Outage

2026-11-20来源 dev.to GitHubEnglish

On November 11, 2026, at 20:48 UTC, GitHub opened an incident titled "Impact across multiple GitHub products including Actions, Pages, and Copilot products". The status page marked Actions, Pages and Copilot as a major outage and Webhooks as degraded. GitHub reported mitigation at 21:07 UTC and resolved the incident at 21:21 UTC, after the backlog of queued Actions jobs was processed.

That is a short window. But if your only way to ship was a GitHub Actions workflow, you could not deploy during it. Not a hotfix, not a rollback, not a config change. That is the real lesson of any CI/CD outage: your pipeline is a production dependency, and it deserves the same failover thinking as your database. The answer is a break-glass deploy path.

I posted a shorter take on X. This is the longer version.

Why your pipeline is part of production

We spend a lot of effort on multi-AZ databases and redundant load balancers. Then we route every change to production through one hosted service, triggered by one webhook system, authenticated by tokens that service issues.

Look at what a typical hosted pipeline depends on:

  • the source host being reachable
  • webhooks firing to trigger the workflow
  • hosted runners picking up jobs
  • CI-issued identity (for example, OIDC tokens) to reach your cloud
  • artifacts and images pulled from registries during the job

If any one of these fails, deploys stop. Most of the time that is fine, because you can wait. The problem is the day you cannot wait: an active incident, a bad release that needs a rollback, or an expiring certificate.

What a break-glass deploy path looks like

A break-glass deploy is a documented, tested, audited way to ship a known-good artifact to production when the normal pipeline is unavailable. It is not a back door. It should be slower and more controlled than the normal path, and every use should be reviewed afterwards.

In my view it needs four parts.

A runner you control. A self-hosted runner, or a secondary CI system, that can run the same deploy step. Self-hosted runners registered to GitHub still need GitHub to dispatch jobs, so they protect you from hosted runner capacity problems, not from a full Actions outage. For that, you need a deploy you can start from a hardened bastion or a second CI system with no dependency on the first.

Artifacts that already exist. Break-glass is for deploying something already built and scanned, not for building from a laptop. Keep release artifacts in storage you own, and mirror the container images you deploy into your own registry, so a deploy never pulls from a public registry or the CI provider's cache.

Credentials that do not depend on the CI provider. This is the part teams miss. OIDC federation between GitHub Actions and your cloud is a good default because it removes long-lived keys. But that trust only works when Actions is issuing tokens. If Actions is down, that path is gone. Keep a separate, tightly scoped break-glass role: it can only deploy, it needs phishing-resistant MFA and approval from a second person, sessions are short, and every assumption triggers an alert.

A runbook people have actually used. A runbook nobody has run is a guess.

A small break-glass runbook example

Here is the shape I like. Every value is a placeholder.

# 0. Declare it. Open an incident and record who approved break-glass. # 1. Assume the break-glass role (MFA required, short session, alerts on use) aws sts assume-role \ --role-arn "arn:aws:iam::<ACCOUNT_ID>:role/<BREAK_GLASS_DEPLOY_ROLE>" \ --role-session-name "breakglass-<INCIDENT_ID>-<YOUR_NAME>" \ --serial-number "<MFA_DEVICE_ARN>" --token-code "<MFA_CODE>" \ --duration-seconds 900 # 2. Deploy a known-good, already-scanned image by digest, never by tag kubectl --context <PROD_CONTEXT> -n <NAMESPACE> set image \ deployment/<APP> <CONTAINER>=<REGISTRY_MIRROR>/<APP>@sha256:<DIGEST> # 3. Verify, then record what changed in the incident ticket kubectl --context <PROD_CONTEXT> -n <NAMESPACE> rollout status deployment/<APP> 

Pinning by digest matters. Under pressure, nobody should be guessing which latest they just shipped.

Break-glass deploy checklist

  • Manual deploy runbook: step by step, stored somewhere that does not depend on your source host, with named owners and an approval rule.
  • Self-hosted or secondary runner: able to run the deploy step without hosted runners, and one option that does not need your CI provider at all.
  • Cached artifacts and images: release artifacts in storage you own, plus a registry mirror for every image you deploy, referenced by digest.
  • Independent credentials: a scoped, audited break-glass role that does not rely on CI-issued OIDC, with MFA, two-person approval, short sessions and alerts on use.
  • Drills: run the break-glass path on a schedule, for example once a quarter, in staging and then production with a no-op change. Time it, fix what broke, and update the runbook.
  • Post-use review: every break-glass use gets reviewed, and the credentials are rotated afterwards if your policy requires it.

Test it before you need it

The GitHub incident was resolved in well under an hour. Next time it might be longer, or it might be a different provider. The teams that stay calm are the ones that already know their fallback works because they ran it last month.

Write the runbook, stand up the runner, mirror the images, scope the role, and drill it. Then a CI/CD outage becomes an inconvenience instead of a blocker. I write more about platform reliability and security on my site.

I lead platform engineering for fintech. See more of my work or how we can work together at Nazmul Ahasan.

原文:https://www.a8q9.cn/en/article/ci-cd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-ci-cd-out-8c7308.html

相关文章