Skip to main content
Change Risk Intel
Change Management

Deployment Rings: How Staged Rollouts Fit Into Change Control

Ben Ennis Founder, Ennis Studio · former Partner Technology Advisor, ServiceNow · 13 min read

No affiliate links appear in this article. Rollout schedules and controls are quoted from each vendor’s own documentation, linked in Sources.

Table of contents

Why this matters

Most change records describe what is changing and when. Far fewer describe how many systems will have the change at each point in time, and that is the number that decides how bad a defect gets. On July 19, 2024, CrowdStrike delivered two new IPC Template Instances to Windows sensors, and one of them produced the Channel File 291 crash. In its root cause analysis, CrowdStrike’s main process fix was that “each Template Instance should be deployed in a staged rollout,” promoted through “wider deployment rings” only after canary testing and bake-in time. It also gave customers control over “where and when Rapid Response Content updates are deployed.”

The pattern keeps repeating at smaller scale. Two weeks ago, Samsung said a SmartThings refrigerator update “was incorrectly distributed to some customers during testing,” per The Chosun Daily. That is a test-ring build that leaked into production. And on September 30, Azure’s own staged OS servicing rollout collided with a gateway-manager change. Because the servicing “proceeded gradually through multiple regions,” engineers could pause it once they saw the correlation, according to Azure status history (7Q30-010). Staging did not prevent that incident, but it gave operators something to stop.

For a change advisory board, the point is simple. Two changes with the same code and the same risk score can have very different worst cases depending on how they roll out. If the CAB only approves the change and not the rollout, it is approving half the risk. Our change risk score tool treats blast radius as a scoring input for exactly this reason.

What a deployment ring actually is

Microsoft’s deployment planning guide gives the cleanest definition: “Each ring comprises a group of users or devices that receive a particular update concurrently.” Administrators set criteria for each ring, based on “deferral time or adoption (completion),” that must be met before the update moves to the next, broader ring. Microsoft’s default names are Preview, Limited and Broad. It also lists common alternatives, such as “First > Fast > Broad” and “Canaries > Early Adopters > Users,” and says plainly that “there are no definite rules for exactly how many rings to have.”

Google’s SRE team uses the canary vocabulary instead. In the Site Reliability Workbook chapter on canarying, “the part of the service that receives the change is ‘the canary,’ and the remainder of the service is ‘the control.’” Its worked example shows why the first stage should be small. A canary sent to 5% of the population that throws 20% errors produces a 1% overall error rate. The defect is real and measurable, but most users never see it.

Three ingredients define a ring plan regardless of vocabulary:

  • Membership. Who or what is in each ring, and whether it represents the hardware, configurations and users of the full fleet.
  • Promotion criteria. The signals and bake time that must be clean before the next ring starts.
  • Stop control. Who can pause or halt, how fast, and what happens to systems that already have the change.

How the big platforms stage a release

The vendors that ship to the most devices publish their rollout mechanics, and they differ in useful ways. The table below is our side-by-side reading of their documentation.

PlatformFirst production stageHow exposure growsBake time statedStop control
Apple App Store phased release1% of automatic-update usersFixed: 1, 2, 5, 10, 20, 50, 100% over 7 daysOne day per stepPause up to 30 days total
Google Play staged rolloutDeveloper-chosen percentageManual; “won’t increase automatically”None statedHalt; users keep the version
AWS (Builders’ Library)One box, at most 10% of a Region’s or AZ’s requestsWaves: low-traffic Region, high-traffic Region, then more RegionsAt least 1 hour per one-box stage, 12 hours after wave 1Automatic rollback on alarms
Azure safe deploymentCanary region pairPilot, then incremental regions, one AZ per region”Extended bake times” (no figure)Halt “as long as no negative signals surface”
Windows rings (Microsoft)Preview ring of IT staffAdmin-set deferral or adoption criteria”Sufficient stabilization period”Admin pauses the ring
CrowdStrike post-2024Canary ringSuccessive wider rings after bake-inNot statedRollback “if problems are detected”

Apple’s phased release is the most prescriptive. Updates go to a random sample of users with automatic updates on, following a fixed schedule of 1, 2, 5, 10, 20, 50 and 100% over seven days. Google Play’s staged rollouts leave the percentage to the developer and pick users at random for each release. AWS goes further down the stack. Production starts with a one-box deployment that typically serves at most 10% of a Region’s or Availability Zone’s requests, then wave 1 is a single low-traffic Region, deployed one AZ or cell at a time. Azure starts with two canary regions, one with Availability Zones and one without, then a pilot phase with more hardware diversity, then regions in sequence where “a change only goes to one Availability Zone within a region.”

Bar chart showing Apple phased release exposure by day on a log scale: 1% on day 1, 2% on day 2, 5% on day 3, 10% on day 4, 20% on day 5, 50% on day 6 and 100% on day 7, against a dashed line at 100% for an all-at-once release. Days 1 to 3 are shaded to show that a defect paused by day 3 reaches at most 5% of devices.

The chart puts numbers on what the schedule buys. If a defect surfaces in the first three days and someone pauses the rollout, at most 5% of eligible automatic-update devices were ever offered the update. The same defect in an all-at-once release reaches everyone on day one. Detection speed and the person holding the pause button matter as much as the schedule itself.

Where each approach falls short

None of these models is complete, and a CAB should know the gaps before it treats “we use phased release” as a control.

  • Apple: Phased release only governs automatic updates. Apple notes the version can still be “manually downloaded from the App Store by anyone at any time.” Pausing is capped at 30 days in total, and removing the app from sale ends the phased release for that version.
  • Google Play: Exposure never grows on its own, so a forgotten rollout sits at 5% indefinitely. Once a rollout starts, countries cannot be removed from it.
  • AWS: The model assumes cells, Availability Zones and automated rollback alarms. Most enterprises don’t have that structure for internal apps, so copying the wave names without the alarms gives you the delays without the safety.
  • Azure: Microsoft says global services that “cannot progressively deploy to different clusters, regions, or service rings” update in phases by shifting traffic instead. Those services are where regional staging gives the least protection.
  • Microsoft’s Windows guidance: It sets no ring sizes and no bake durations. Each organization has to pick its own, and many never write them down.
  • CrowdStrike’s commitments: The RCA describes rings, canary testing and bake-in, but it does not state ring sizes or durations. Customers have to rely on the controls they were given over Rapid Response Content timing.
  • Google SRE: The workbook refuses to set a fixed canary length because it depends on release cadence. “If you release daily, you can’t let your canary last for a week.” That is good engineering advice, but it leaves the duration to the team.

Halting is not rolling back

A common mistake in change records is treating “halt” as “undo.” Google Play is explicit: after a halt, “no additional users will receive” the version, and users who already received it “will remain on that version.” To fix them, you ship a new release. Apple’s documentation describes pause and resume, but no rollback control. A paused rollout picks up “on the day it was left off” when it resumes.

That changes what a backout plan has to say. For a ringed release of client software, the backout plan is usually “halt, then roll forward with a fixed build,” not “revert.” The CAB should ask how long building, testing and staging that fixed build takes, because that is the real exposure window for the devices already hit. Samsung’s fridge case shows the extreme version. Once a bad build reaches a device that can no longer take updates, there is no rollout control left, and the fix becomes a repair visit.

Server-side changes are different. AWS says that during the entire bake time, “the deployment is automatically rolled back if the team’s high-severity aggregate alarm goes into the alarm state.” Azure deploys “only as long as no negative signals surface.” If your platform supports real rollback, say so in the record. If it only supports halt, say that instead.

What the change board should approve

A ring plan turns one big change into a series of smaller decisions. The CAB does not need to sit in on every promotion. It needs to approve the rules that make promotion safe without it. We suggest four fields on any normal or standard change that ships in stages:

  1. Ring map. Ring names, membership and size, including whether the first ring is representative. Microsoft’s guidance asks that the Limited ring include “a representative sample of devices and applications across the network.”
  2. Promotion gate. The metrics, thresholds and minimum bake time between rings, and whether promotion is automatic or needs a named approver.
  3. Halt authority. Who can stop the rollout without convening the CAB, and the time to halt from first alert.
  4. Recovery path. Rollback, or halt and roll forward, with the estimated time to a fixed build for each.

This fits the way we recommend running the CAB meeting. Approve the plan once as a standard change pattern, then audit promotions after the fact. It also helps with provider-side changes. If a vendor won’t tell you its ring structure, treat that as a third-party risk and add it to the standing cloud-incident agenda item. More change-process guides are on our change management hub.

What to do about it

  1. Inventory how your releases actually roll out. For each major release path, including endpoint agents, mobile apps, SaaS config pushes and server deploys, record whether it is all-at-once, ringed by percentage, or ringed by region.
  2. Add the four ring fields to your change template. Ring map, promotion gate, halt authority and recovery path. Make them mandatory for any change that will reach more than one environment.
  3. Write down bake times. Microsoft and Google both leave duration to you. Pick a number per ring, tied to your release cadence, and record why.
  4. Separate test and production channels physically. The Samsung case was a distribution error, not a code defect. Test builds should be signed or tagged so production devices refuse them.
  5. Rehearse the halt. Once a quarter, have the named halt owner pause a low-risk rollout and time it from alert to stop.
  6. Ask your critical vendors for their ring plan. Endpoint security, MDM and CDN vendors push content to your fleet. Ask for their first-ring size, bake time and your customer controls. Then use those controls, as CrowdStrike now lets customers do for Rapid Response Content.

FAQ

What is the difference between a deployment ring and a canary release?

They describe the same idea at different levels. A canary is the first, small group that receives a change while the rest of the service acts as the control. Deployment rings are the full ordered sequence, from canary or preview through to broad deployment, with criteria between each ring.

How many deployment rings should we have?

Microsoft says there are no fixed rules. Three is a common starting point: a preview ring of IT staff, a limited ring that represents your hardware and apps, and a broad ring. Add a separate ring for mission-critical systems if they need extra stabilization time.

How long should each ring bake?

There is no universal figure. Google’s SRE workbook ties canary length to release cadence, and AWS says a typical pipeline waits at least one hour after each one-box stage and at least 12 hours after the first regional wave. Choose a duration long enough to see your key error signals, write it into the change record, and review it after incidents.

Does halting a staged rollout undo the update?

Usually not. Google Play says users who already received a halted version stay on it, and Apple’s phased release offers pause, not rollback. Plan for halt and roll forward with a fixed build unless your platform supports real rollback.

Should a CAB approve every ring promotion?

Not normally. Approve the ring plan, promotion criteria and halt authority once, then let automated gates or a named release owner promote. Bring promotions back to the CAB only when a gate fails or a threshold is waived.

What did CrowdStrike change after the July 2024 outage?

Its root cause analysis says each Template Instance would be deployed in a staged rollout, promoted through wider rings after canary testing and bake-in, and rolled back if problems were detected. Customers also gained control over where and when Rapid Response Content is deployed.

Sources

#deployment rings #staged rollout #canary release #change advisory board #release management #blast radius

Published October 5, 2026.