Cost optimization that keeps the product

The cheapest token is not the cheapest system. A weak model that fails the hard slice, a cache invalidated by a timestamp, a real-time call nobody waits for, or 3,000 irrelevant context tokens all turn “low price” into waste. The defensible unit is cost per successful task, among designs that clear a measured quality floor.

quality floor firstcost / successful task routingprompt caching batchingcontext budget

A 10,000-task workload, priced and graded

The fixed mix is 70% routine classification, 20% offline document work, and 10% hard reasoning. Every strategy sees the same deterministic tasks. “Capable everywhere” is the baseline; the levers below change model choice and billed tokens. The quality floor defaults to 92% successful tasks.

Fleet economics

model policy

quality floor a release constraint, not a knob to lower after seeing cost

cache the stable prefix keep volatile tokens after the cache breakpoint batch the offline document slice trim irrelevant volatile context by 35%

The measured baseline spends $132.30 for 10,000 tasks, succeeds on 96.36%, and costs $0.01373 per successful task. “Small everywhere” cuts model spend to $35.28 and looks heroic on a token dashboard — but quality falls to 91.78%, below the 92% floor, with the hard slice at 74.1%. It is cheap because it quietly stopped delivering the product.

Routing spends the small model on the 70% routine slice and keeps the capable model for documents and hard reasoning. In the measured run it clears quality at 94.63% and drops cost per success to $0.00910. Then independent token levers compound: an 85% cache-hit rate on stable prefixes, half-price batch treatment for the offline slice, and a 35% trim of volatile context bring the total to $43.62 and cost per success to $0.00461, while quality remains 94.58%.

Caching teaches an architectural lesson rather than a switch. Put a timestamp, request id or user turn before the breakpoint and the measured hit rate goes 85% → 0%: the same optimized workload rises from $43.62 to $57.25 with no quality gain. Batching is similarly conditional: it belongs on the offline document work, not the human-facing request. Context trimming must be evaled by slice — deleting irrelevant tokens saves money; deleting evidence creates a cheaper failure.

# optimize in this order — the order is a control
quality_floor = measured_outcome_success >= 0.92
route          = cheapest_model_that_clears_each_workload_floor
cache          = stable_system_tools_reference + BREAKPOINT + volatile_request
batch          = only_where_no_human_waits
context        = retrieve/trim to evidence_needed; reserve output + tool-turn budget
decision_unit  = total_model_spend / successful_tasks
ship           = quality_floor AND every_critical_slice_floor

⚠️ Traps & honesty: prices and quality rates are explicit toy assumptions ($0.80/$4 per million input/output tokens for “small,” $3/$15 for “capable,” 90% cached-input discount, 50% batch discount) — use your provider's current rates and your own usage fields · cost per successful task here is model spend divided by measured successes; a real business view should also include retries, human review, latency and incident cost · cache-hit and quality rates come from this fixed workload, not a vendor guarantee · never infer that a particular real model is “small” or “capable” for your task without an eval.

Takeaways: defend quality overall and by critical slice before comparing cost · route per workload; “one model everywhere” overpays easy cases or under-serves hard ones · cache exact stable prefixes and keep volatility after the breakpoint · batch work whose consumer is not waiting · context is billed input and an attention budget, so retrieve and trim deliberately · optimize cost per successful task, not price per token. Next: put policies in enforceable homes.

Second opinion (taught here — these corroborate): Anthropic docs · prompt caching · Anthropic docs · Message Batches.