How Smart Routing Can Cut Your AI Costs by 40%

The first version of our cost-optimization router was three lines of code: get the per-token price for every model that supports the requested capabilities, pick the cheapest, send the request. It saved roughly 60% on token spend and was unusable in production within a week.

The problem is obvious in hindsight. The cheapest model on paper is not the cheapest model on your workload. If GPT-4o-mini answers correctly the first time but Llama-3.1-8b needs two retries plus a fallback, the cheap option is more expensive once you account for retries, latency penalties, and the engineering time spent investigating user complaints. The interesting problem is not "find the cheapest token price" but "find the cheapest expected cost to produce a correct answer."

This post walks through how the AllRoutes router actually decides, what we measured, and where the 40% number comes from.

The cost function

We model the expected cost of routing a request r to a model m as:

expected_cost(r, m) = price(m) * tokens(r, m) / quality(r, m)
                      + retry_penalty(m, r)
                      + latency_penalty(m, r)

Each term breaks down further:

• price(m) is the headline per-token cost, fetched from the provider's pricing API and refreshed hourly. We track input, output, and cached-input rates separately.
• tokens(r, m) is the expected token count for this request on this model. Different tokenizers produce different token counts for the same text — Claude tokens are roughly 1.15x OpenAI tokens for English prose, and the ratio shifts for code and non-Latin scripts. We tokenize the request with the actual model tokenizer to get an accurate count.
• quality(r, m) is a probability between 0 and 1 representing "the model produces an output the caller will accept." We learn this per workload from Compass Radar evals.
• retry_penalty(m, r) captures the expected cost of a retry. If a model fails 8% of the time on this workload, the expected cost is 1.087x the base cost, plus the latency hit.
• latency_penalty(m, r) is a small term that converts excess p95 latency into dollars at a configurable rate, defaulting to zero so cost-only callers do not get surprised.

The router then picks argmin_m expected_cost(r, m) from the set of models that satisfy the request's capability constraints (supports tools, supports JSON mode, context window large enough, etc).

Where the data comes from

The interesting part is quality(r, m). The naive approach is "use the leaderboard", which fails for two reasons. First, leaderboards measure aggregate quality on benchmarks that probably do not match your traffic. Second, quality drifts: providers ship silent updates, your prompt distribution changes, your users start asking different questions.

We solve this with a continuous evaluation loop:

1. A configurable percentage of production requests (default 1%) get shadow-routed: the primary response goes to the user, but we also send the request to a basket of candidate models.
2. The shadow responses are scored by an LLM-as-judge running a rubric the customer wrote. The judge runs on a held-out model so its scores are independent of the candidates.
3. Scores get aggregated per model, per workload, per week into a quality matrix stored in ClickHouse.
4. The router reads from this matrix when computing quality(r, m).

The first time the router sees a new workload, the quality matrix is empty and we fall back to provider defaults from our public benchmarks. After 1000 requests, the workload-specific quality scores have meaningfully tighter confidence intervals than the public benchmarks.

What we measured

We ran cost-optimized routing across 18 production workloads from design partners. Workloads ranged from customer-support chatbots to code-review assistants to RAG-backed knowledge bases. We measured spend per accepted answer, where "accepted" means the customer's own success metric (resolved ticket, shipped PR comment, etc).

Median savings: 41%. p10: 18%. p90: 58%. The variance is real and depends on workload characteristics.

Three patterns predict big savings:

• Wide capability headroom. Workloads that use a frontier model "just to be safe" but where a mid-tier model is empirically just as good save the most. Customer-support routing in particular tends to be over-provisioned.
• High volume of short prompts. When prompts are short, the per-call quality variance is lower and the router can confidently send most requests to a cheap model.
• Diverse traffic mix. When 80% of requests are easy and 20% are hard, sending all 100% to a frontier model is wasteful. The router catches the easy ones and routes them down.

Conversely, three patterns predict small savings:

• Long-context reasoning. Frontier models genuinely outperform on long-context multi-step reasoning, and the router correctly sends those requests to expensive models.
• Strict format constraints. If your prompt requires an exact JSON shape with deeply nested fields, smaller models fail more often, and the retry penalty wipes out the price savings.
• Already-tuned single-model setups. If your team already picked the right model and tuned the prompt for it, there is less room to save.

A worked example

A design partner runs a documentation Q&A bot. Pre-AllRoutes, every request went to gpt-4o at $5/$15 per million input/output tokens. Their average request was 1200 input tokens, 300 output tokens. Cost per request: $0.0105.

After enabling cost-optimized routing with their accuracy rubric:

Model	Share of traffic	Quality score	Cost per request
gpt-4o-mini	71%	0.94	$0.00073
claude-haiku-4	18%	0.93	$0.00091
gpt-4o	9%	0.97	$0.0105
claude-sonnet-4	2%	0.98	$0.0098

Blended cost per request dropped to $0.0021. That is an 80% reduction on this workload, which is well above the 41% median because the workload was extremely well-suited to small models.

Configuring it

Cost-optimized routing is one line:

client.chat.completions.create(
    model="auto",
    messages=[...],
    extra_body={"routing": {"strategy": "cost", "min_quality": 0.9}},
)

The min_quality parameter is the safety rail. It says "do not pick a model whose learned quality on this workload is below 0.9." If no model in the candidate set meets the bar, the router falls back to your default model rather than picking a worse-but-cheaper option. This is the single most useful tunable.

The full reference is in the routing docs. If you are running multi-provider traffic today and have not measured what cost-optimized routing would do, you are probably leaving real money on the table.