A/B testing mechanics

An online experiment is not “show two prompts and pick the larger score.” It is a chain: choose a stable assignment unit, collect enough independent observations, quantify uncertainty, defend guardrails, and make a decision against a business-sized effect. Break any link and a precise-looking percentage can recommend the wrong release.

stable assignment95% interval false positive / false negativeguardrail

One experiment, all five decisions visible

The toy outcome is “ticket resolved without reopening in seven days.” Arm A is the incumbent; B is the candidate. The release rule is deliberately stricter than “statistically different”: B must have a positive 95% interval, beat a pre-agreed +2 percentage-point minimum useful effect, and keep incorrect actions within +1 point of A.

Experiment readout

assignment unit

true scenario

sample per arm more observations narrow the interval; they do not create business value

Assignment comes first. Hashing a stable user id gives every returning user one experience: the measured 24-user sample has 0 crossovers. Hashing each request puts 23 of 24 users into both arms across four visits. That contamination is not cured by a larger sample; it changes what “B” even means because people carry learning and state from one version into the other.

Uncertainty is the width, not a ceremonial p-value. In the measured small-sample run (200 per arm), the useful +3-point candidate happens to look +8.5 points better, yet its 95% interval runs from −0.2 to +17.2 points. The experiment is underpowered and produces a false negative: not enough evidence to ship even though the underlying effect is useful. At 1,000 per arm the same deterministic stream narrows to +0.04…+7.76 points and clears the gate.

Significance is not the product decision. With 10,000 observations per arm, the tiny-effect scenario measures +1.52 points with an interval of +0.28…+2.76: evidence that B differs from A, but still below the pre-agreed +2-point business threshold. Conversely, the “unlucky null” run is a real false positive: equal underlying rates happen to produce +4.5 observed points and a positive interval. Repetition, pre-registration and correction for multiple looks are how teams manage that error rate; the words “statistically significant” never guarantee truth or value.

# release decision written before looking at the result
assignment_unit = hash(authenticated_user_id)   # sticky, mutually exclusive arms
primary_gate    = lower_95_ci(candidate - incumbent) > 0
value_gate      = observed_uplift >= 0.02       # minimum effect worth operating
guardrail_gate  = incorrect_action_delta <= 0.01
decision        = stage_only_if(all_three_gates_pass)

⚠️ Traps & honesty: this is a seeded Bernoulli simulation, not evidence about any real prompt or model · the interval is the familiar unpooled normal approximation, useful for intuition but not a substitute for an experiment design appropriate to clustered users, rare events, sequential looks or multiple metrics · “95%” does not mean a 95% probability this one fixed interval contains the truth · the business and guardrail thresholds are product decisions, not facts statistics can choose for you.

Takeaways: assign on the unit that can carry treatment effects, usually the authenticated user or account · decide sample size from the smallest effect worth detecting, before launch · a false positive ships noise; a false negative misses a real improvement — small samples invite the second, repeated peeking invites the first · significance says “evidence of a difference,” not “valuable” · primary wins do not overrule safety, quality, latency or cost guardrails. Next: optimize cost only after the quality floor is defended.

Second opinion (taught here — these corroborate): NIST · two proportions · Trustworthy Online Controlled Experiments · companion material.