The model-upgrade pipeline

A new model version ships. It's better on average — and quietly worse on one slice of your traffic, because "better on average" is a statement about someone else's benchmark, not your workload. This page measures the three ways to take that upgrade: let an alias auto-upgrade you, gate it behind your golden set, or stage it through a small traffic slice. The differences are counted in users who got a bad answer before anyone noticed.

pin → shadow → stage → rollbackgolden set slice regression

An upgrade with a hidden slice regression

The setup: 10,000 requests/day. The candidate model is fine everywhere except one slice — 12% of traffic (say, invoices in one regional format) where quality drops from 95% to 70%. The one dial you control in advance: how many cases from that slice your golden set happens to contain.

Three ways to take the upgrade

slice cases in your golden set out of ~200 total cases — coverage is the load-bearing variable

Walk the bars left to right. The alias path is the honest name for "we didn't decide": the model under your prompts changed the day the vendor said so, the regression ran at full traffic, and the detection signal was support complaints — a slow, angry channel. The staged path caps the blast radius by arithmetic: 5% of traffic for the day it takes the per-arm metric to show the gap. The shadow path can be perfect — zero users exposed, the regression caught on your golden set before any traffic — but only if the golden set contains the slice. Slide the coverage dial to 0 and shadow testing waves the regression straight through: a golden set can only defend the traffic it represents.

# the pipeline, as configuration rather than heroics
model  : claude-sonnet-4-5        # PINNED — upgrades are decisions, not surprises
shadow : run golden set (incl. every known slice) on candidate; gate on no-regression-per-slice
stage  : 5% traffic, per-arm quality metric, auto-halt on significant gap
rollback: one config change, rehearsed — the stage gate is only real if the exit is

⚠️ Traps & honesty: the regression size (95%→70%), slice share (12%), and detection lags (3 days via complaints, 1 day via per-arm metrics) are assumptions of the toy — the ordering they produce (alias ≫ staged > shadow-with-coverage) is the transferable part, the exact counts are not · shadow detection here requires the candidate to fail clearly more slice cases than the incumbent (≥2 above baseline), which is why tiny coverage numbers are coin-flips — real gates need enough cases per slice for the comparison to mean something · "better on average" and "worse on your slice" genuinely coexist; that is the entire reason per-slice gating exists · staging assumes you HAVE a per-arm quality metric — staging without one is just shipping to 5% of users with extra steps.

Takeaways: pin model versions — an alias means your system changes without a decision · shadow-test candidates on the golden set per slice, because a set without the slice cannot catch the slice · stage behind a small traffic arm with a quality metric and an automatic halt · rehearse the rollback; an exit you've never taken isn't an exit · the deep dependency: this whole pipeline is only as good as golden-set coverage — which is why eval-driven development and the Bench eval station sit underneath the lifecycle domain. Next: back to the pro track.

Second opinion (taught here — these corroborate): Anthropic docs · developing evals · Google SRE · release engineering.