Skip to content

Rolling out to one branch before all of them

  • Home
  • Blog
  • Rolling out to one branch before all of them
Rolling out to one branch before all of them

A phased rollout branch is a Git or environment branch that receives a release before the rest of your infrastructure. You deploy there, watch traffic and errors, then promote forward. It limits the blast radius of a bad change to a small, reversible slice instead of every user at once.

Key Takeaways

  • A phased rollout means one branch or environment gets the change first, then a canary slice, then everything.
  • Blast radius is the number of users or systems a bad deploy can break; phasing shrinks it to a survivable size.
  • You don't need phased branches for a low-traffic brochure site — one environment plus good backups is the simpler, correct option.
  • Branch protection and required reviews stop a hotfix from skipping the queue.
  • Rollback is a promotion in reverse: redeploy the previous good version, don't patch forward under pressure.
  • The real cost is process and attention — extra environments, CI time, and someone watching metrics between phases.
How a phased rollout moves through branchesOrdered stages from a feature branch through staging, canary and wider rollout to production main, connected by arrows.How a phased rollout moves through branches1Featurebranch2Stagingbranch3Canarybranch4Widerrollout5Productionmain
The stages a change passes through on the way to production, from a feature branch through staging and canary to the full release on main.

What is a phased rollout branch?

A phased rollout branch is a Git branch or a named environment that receives a release before the rest of your fleet. You merge or deploy to staging, then to a canary branch, then to main. Each hop is a decision point where you can stop and roll back instead of shipping to everyone at once.

Branch means two things here, and both are common. In Git, it is an environment branch: main is production, canary is the next slice, staging is the test bed. In infrastructure, a branch can be a region, a node pool, or a percentage of traffic. The pattern is the same either way: forward movement only after a gate passes. Terms you will see for the same idea include progressive delivery, ring deployment, and canary release.

Why does phasing matter in production?

The variable that matters is blast radius — the number of users or systems a bad deploy can break. A phased rollout shrinks blast radius from everyone to a small cohort, so a broken migration costs a support ticket instead of a company-wide incident. Rollback is then a simple redeploy, not an emergency.

Staging catches syntax errors and missing config, but it never has production data, production traffic, or production's odd browser mix. The failures that hurt — a migration that locks a table, a library that breaks one payment method, a cache key collision — appear only under real load. When the whole fleet gets the change at once, you discover these by watching your error pages fill up. Phasing makes the first few percentage points of traffic your test. A bad outcome becomes a signal, not an incident, and mean time to recovery drops because the previous version is one command away.

When do you actually need one (and when don't you)?

You need a phased rollout when a change is high-risk, high-traffic, or hard to test in staging alone — schema migrations, payment rewrites, auth changes. You do not need one for a low-traffic brochure site or an internal tool with a handful of users, where one environment plus good backups is the simpler, correct option.

The decision comes down to three questions. How many users feel a bad deploy? What is the revenue or trust lost per minute of downtime? Can you measure the canary's behaviour separately from the rest? If the answers are thousands, real money, and yes, phasing earns its keep. If the site is a marketing page with a few hundred visits a week, the overhead of extra branches and soak windows is ceremony. A common mistake we see is applying a four-branch pipeline to a site that would be safer with a staging copy, a backup, and a careful pair of eyes on the deploy. Phased web development matters when the cost of being wrong is high.

How does the mechanism work?

The mechanism is a promotion queue: a change moves forward only after it passes a gate. In Git, a feature branch merges into staging, then a canary branch, then main. In infrastructure, a deployment targets a percentage of traffic, a region, or a node pool, and the controller shifts more traffic only while health checks pass.

On the code side, the gate is a pull request with required reviews and green CI. You cannot merge to canary or main without a human approving and the test suite passing — that is enforced by branch protection rules, not by discipline. On the infrastructure side, the gate is a health check: the canary reports its error rate and latency, and you compare those numbers against the stable version before widening traffic. Feature flags fit here too. They let you ship code dark and enable it per cohort, which pairs well with a phased branch: the branch controls which version runs, the flag controls who sees the new path. A pull request is the moment a human says this can move forward.

How do you set it up?

The setup sequence is: create the branch hierarchy, protect the branches, add CI gates, deploy to the first branch, watch signals, then promote. Each step is reversible, and the first promotion should be rehearsed on a low-traffic change before you trust the process for anything risky.

  1. Create three long-lived branches: main, canary, and staging.
  2. Protect main and canary with required pull requests and required status checks.
  3. Configure CI to run tests, lint, and a production build on every push to each branch.
  4. Push the change to a feature branch and open a pull request into staging; merge after review and green CI.
  5. Deploy staging and run smoke tests against a staging database that copies production schema.
  6. Merge into canary and deploy to a small traffic slice, or a canary environment; watch metrics for the soak window.
  7. Promote to main, widen traffic gradually, then delete or reset the canary branch.
git checkout -b feature/payment-flags
# commit the change
git push -u origin feature/payment-flags
# open a pull request into staging
# Rollback is a redeploy of the previous good version.
git revert <bad-commit>   # creates a new commit; safe on shared branches
# never force-push to main, canary, or staging

git revert is safe — it adds a commit that undoes the change, so shared history stays intact. Force-pushing rewrites history and will strand your teammates' work; treat it as destructive.

Configuration that matters

The configuration that changes outcomes is branch protection, required status checks, and the traffic-splitting rule. Without protection, a hotfix bypasses the queue; without status checks, a broken test still merges; without a traffic rule, you cannot shrink or widen a cohort quickly.

Branch protection is the non-negotiable part: require at least one human review, require CI to pass, and forbid direct pushes to main and canary. Status checks should include tests, lint, and the same build command production uses. The traffic rule — whether it is a load balancer weight, a service mesh route, or a flag percentage — must be changeable in one place without a redeploy. Each branch also needs its own environment variables, because staging pointed at production secrets is how a test turns into a data leak.

How do you verify it is safe?

Verification compares the canary branch against the previous version on the same traffic. You watch error rate, p95 latency, and a business signal such as successful checkouts, and you fail the rollout if any diverges beyond the threshold you set before you started.

A rollout timeline with checkpointsFive milestones along a timeline: staging deploy, canary at five percent, soak window, expansion to twenty-five percent, and full release.A rollout timeline with checkpointsStaging1smoke testsCanary25% trafficSoak window3error + latencyExpand425% trafficFull release5100% traffic
Checkpoints along a phased rollout: each milestone is a place to stop, read the signals, and decide to promote or roll back.

Set the thresholds before the deploy, not during it. A sensible baseline: error rate must stay under 1% on the canary, p95 latency must not grow more than 20%, and failed checkouts must not drop. If the canary runs at 0.3% errors for an hour, you promote. If it jumps to 4%, you roll back. The soak window — anywhere from an hour for a small change to a full day for a migration — is what turns a slow-burning problem into a caught problem.

Failure modes and how to debug them

The common failures are a release that passes staging but breaks on real data, a traffic-split misconfiguration that sends everything to the canary, and a branch protection gap that lets a hotfix skip the queue. The first check in each case is whether the change is actually isolated to the branch you think it is.

When staging and production disagree, diff the environment variables, secrets, and database versions first — a migration that ran against a stale staging copy will behave differently on real data. When the split misbehaves, check the routing rule, the load balancer weights, and the service mesh config; a typo there sends 100% of traffic to the canary in one step. When a canary branch lives too long, it drifts from main, so merge main back into it before promoting. And when a rollback fails, the usual cause is a destructive migration shipped in the same release as the code that depends on it. Failed software rollouts almost always trace back to one of these.

Cost and operational overhead

The cost of a phased rollout is process, not infrastructure: extra branches and environments, longer CI runs, and a person who watches metrics between phases. For a small team the simpler option often wins; for high-traffic revenue paths, the reduced incident risk justifies the operational tax.

You are paying for attention, not just machines. Each phase needs someone to look at dashboards and decide promote or roll back. That is cheap at first and tiring forever. If your team is one developer and a part-time reviewer, a canary plus a soak window may be more than you can honestly operate — in that case a staging environment with production-like data and a tested backup restore is the right amount of safety. For teams with an on-call rotation and a real metrics stack, the phased approach pays for itself the first time it catches a migration that would have taken the site down.

Security considerations

Phased rollouts touch security in two places: branch protection and secrets. Protected branches stop a compromised credential from pushing straight to production, and each environment needs its own secrets, never the same keys as production, because a canary is a smaller target but not a zero-trust zone.

Treat branch protection as a security control, not a workflow preference: require reviews, require CI, and rotate deploy credentials with least privilege. A canary branch runs production-adjacent code, so it needs the same dependency scanning and the same care with secrets. Audit who can promote a branch — that permission is effectively a production deploy permission, and it should be held by the fewest people who actually need it.

Common mistakes

The most common mistakes are skipping the soak window, letting a canary branch live long enough to drift, and treating rollback as a shameful event rather than a rehearsed procedure. Teams also over-engineer: four branches for a brochure site is ceremony, not safety.

  • Skipping the soak window because the change looks simple.
  • Long-lived branches that drift and produce a bad merge.
  • No pre-agreed rollback trigger, so decisions happen under pressure.
  • Reusing the same secrets across environments.
  • Phasing every change, including copy edits that don't need it.

If any of this sounds familiar, a review of your branch strategy is where we would start — custom software development should include a release path your team can actually operate, not one that exists only in a diagram.

A concrete rollout scenario

Consider a payment provider change on a booking site. You branch from main, merge to staging, run a test checkout, merge to canary, route 5% of traffic, watch failed-payment rate for six hours, then expand to 25%, then 100%, and you keep the rollback command ready the whole time.

At the 5% slice, everything looks normal for the first hour. Then a user in a different currency hits a rounding path that only exists on production data: the failed-payment rate on the canary climbs from 0.2% to 3%. The on-call engineer sees the alert, runs the rollback — a redeploy of the previous tag — and the canary returns to baseline inside minutes. The rest of the fleet never saw the change. Without the phased branch, that same bug would have hit 100% of users during the evening peak.

Alternatives compared

Canary, blue-green, feature flags, and all-at-once are the four strategies to choose between. The decision comes down to whether you need instant rollback, a dark launch, or measurement per cohort, and how much extra environment you are willing to run.

Which rollout strategy fits which workloadRows mapping each rollout strategy to the workload it suits.Which option appliesCanaryHigh-traffic services where you can measure error rate per sliceBlue-greenServices that need instant rollback with a full second environmentFeature flagsChanges you want to ship dark and enable per user cohortAll-at-onceLow-traffic brochure sites where extra environments outweigh the risk
How the common rollout strategies map to workload type, traffic level and the risk you are willing to carry.
StrategyBlast radiusRollback speedOperational overheadBest for
Canary / phased branchesSmall sliceMinutes (redeploy)Medium — metrics and soakHigh-traffic APIs, migrations
Blue-greenNone (old env kept)Seconds (switch)High — full duplicate environmentInstant rollback needs
Feature flagsPer-cohortInstant (kill flag)Medium — flag hygieneDark launches, gradual UI
All-at-onceEveryoneSlow if brokenLow — one deployLow-traffic brochure sites

In short

  • Phased rollout branches shrink blast radius by promoting a change through staging, canary, then main.
  • Protect the branches, require reviews, and set metric thresholds before you deploy.
  • Rollback is a redeploy of the previous version — rehearse it, don't improvise it.
  • If you can't measure the canary separately, or the traffic is tiny, the simpler one-environment setup wins.

People also search for

If you are planning a phased rollout and want a second pair of eyes on your branch strategy, traffic rules, or rollback path, our team can help you review what exists today and what to fix first. We work in your repositories and your accounts, and we hand over something your own team can operate. Contact us for a review, or see how we approach website maintenance once a release is live.

Frequently asked questions

  • A phased rollout branch is a Git branch that deploys only to a subset of environments, such as staging or a canary group, before the change is merged to the main branch that feeds all production servers. It isolates new code behind a branch boundary so you can observe behaviour without a full release.

  • Use it when a change touches shared infrastructure, a database schema, or high-traffic APIs where a bad deploy affects every user. Rolling to one branch first limits blast radius and gives you a rollback point that does not require reverting the main history. It suits changes that are risky but not easily behind a feature flag.

  • Create a release branch from a stable commit, deploy that branch to the first environment, then merge it to the main integration branch only after verification. Keep the branch name consistent with your pipeline’s deployment selector. Use protected branches so an accidental push does not promote the change early.

  • A branch controls which code is deployed to an environment; a feature flag controls which users see the code at runtime. Branch-based rollout requires a new deploy to change exposure, while a flag can be toggled without redeploying. Flags are better for per-user targeting, branches for environment-by-environment progression.

  • If you have not merged the change, delete or reset the rollout branch and redeploy the previous good commit to that environment. If the change is already merged, revert it on the release branch with `git revert` and deploy that revert before promoting. Back up the database or config state that the partial rollout may have changed.

  • The most common failure is drift between the rollout branch and the main branch, so later merges produce unexpected conflicts. Another is assuming the first environment matches production exactly; a different configuration, dataset, or cache layer can hide bugs. Finally, forgetting to promote or demote the branch leaves some environments permanently behind.

  • Compare the rollout branch against the previous release using `git diff` and review the changed files. Check application logs, error rates, and key metrics in the first environment for the same traffic profile. Run the existing test suite against that exact commit, not just the feature branch, then promote only when those checks pass.

  • Restrict who can push to rollout and main branches, because a compromised account could promote unvetted code to all environments. Use signed commits or branch protection rules. Audit the deployment credentials used by the first environment, since it now receives code earlier and may expose a smaller attack surface before the full release.

  • It adds one extra merge or promotion step per release and a second deploy target to monitor. For small teams this is manageable; for frequent releases it becomes a bottleneck. The cost is mainly engineer time and pipeline minutes, which grow with the number of environments. Use it only for changes that justify the delay.

  • Feature flags, canary deployments by traffic percentage, blue-green environments, and progressive delivery tools all achieve phased exposure without separate long-lived branches. A single main branch with short-lived feature branches and runtime flags is often simpler. Choose based on how you need to control exposure: per environment, per user, or per request.

0 comments

Be the first to share your thoughts.

Leave a comment

Chat on WhatsApp