Cost Optimization

Patterns and AllRoutes features that compound to cut your AI bill 40-90% without quality loss.

Overview

AI costs scale superlinearly with sloppy implementation. The same workload can cost $100/month or $5,000/month depending on caching, model selection, and prompt design. This guide covers the high-impact levers in rough order of payoff.

A typical production application that applies all of these sees 70-90% cost reduction versus a naive baseline.

1. Cache Aggressively (40-80% Savings)

Semantic caching is enabled by default. Make sure you're not accidentally defeating it:

Stable system prompts -- avoid embedding timestamps or user IDs in the system prompt; put dynamic data in user messages
Lower the threshold for high-repeat workloads -- FAQ bots can drop from 0.92 to 0.85 for 2-3x more hits
Anthropic prompt caching -- mark long stable contexts with cache_control to get a 90% discount on cache hits (stacks with the gateway cache)

{
  "model": "claude-sonnet-4-20250514",
  "messages": [
    {
      "role": "system",
      "content": [
        {"type": "text", "text": "<5KB system prompt>", "cache_control": {"type": "ephemeral"}}
      ]
    },
    {"role": "user", "content": "..."}
  ]
}

Workload Type	Typical Hit Rate	Typical Savings
FAQ / Support bot	60-80%	55-75%
Classification / Tagging	50-70%	45-65%
Summarization (templated)	30-50%	25-45%

2. Right-Size the Model (30-90% Savings)

The default reflex of "use GPT-4o for everything" is the most common cost sink. Match the model to the task:

Task	Cheap Tier	Expensive Tier	Savings
Classification, routing, intent detection	`gpt-4o-mini`, `claude-haiku-3-5`, `gemini-2.0-flash`	`gpt-4o`, `claude-sonnet-4`	~95%
Code completion (single-turn)	`deepseek-coder-v3`, `qwen-2.5-coder-32b`	`claude-sonnet-4`, `gpt-4o`	~90%
Summarization (under 8K tokens)	`gpt-4o-mini`, `gemini-2.0-flash`	`gpt-4o`, `claude-opus-4`	~95%
Multi-step agentic reasoning	`o1-mini`, `claude-sonnet-4`	`o1`, `claude-opus-4`	~80%
Vision (chart/document QA)	`gemini-2.0-flash`, `gpt-4o-mini`	`gpt-4o`, `claude-sonnet-4`	~90%

Run an A/B test via Model Groups before committing -- the cheap tier is sometimes good enough, sometimes not.

3. Use BYOK (5-50% Savings vs. Other Gateways)

BYOK on AllRoutes is 0% commission, always. If you currently route through a gateway that charges 5-10% on top of provider rates, switching saves the spread instantly. On $10K/month spend, that's $500-1000/month with zero workflow change.

4. Stream Responses (10-30% Effective Savings)

Streaming doesn't cut token cost, but it cuts the cost of unwanted tokens. With stream: true, you can abort a generation as soon as you have what you need -- early stopping in long-form generation often saves 30-50% of completion tokens.

const controller = new AbortController();
const stream = await client.chat.completions.create(
  { model: "gpt-4o", messages, stream: true },
  { signal: controller.signal }
);

for await (const chunk of stream) {
  const text = chunk.choices[0]?.delta?.content ?? "";
  // Stop as soon as we hit the answer marker
  if (text.includes("ANSWER_END")) {
    controller.abort();
    break;
  }
}

5. Keep Prompts Lean (5-20% Savings)

Drop redundant few-shot examples -- 2-3 well-chosen examples usually beat 10
Compress system prompts -- 1500 tokens is plenty for almost any task
Reuse context via prompt caching rather than restating it in every turn
Strip whitespace and commentary from JSON in tool definitions

Audit with the usage field on every response. If prompt_tokens is consistently 5x your completion_tokens, the prompt is the bottleneck.

6. Use Free Routes for Dev (100% Savings on Dev Workload)

In development and CI, point at the free tier explicitly:

client.chat.completions.create(
    models=["meta-llama/llama-3.1-70b-instruct:free", "gpt-4o-mini"],
    messages=msgs,
)

CI test runs that hit the free route cost $0; production traffic falls through to paid.

7. Rerank Instead of Long Context (50-70% Savings on RAG)

If your RAG pipeline stuffs 30 documents into the prompt to "be safe," you're paying input tokens 30x. Use Embeddings for recall (top-50), then Rerank to pick the top 3-5. The rerank API costs cents; the saved input tokens save dollars.

candidates = vector_store.search(query, k=50)        # ~$0
reranked = client.rerank(model="rerank-english-v3.0",
                         query=query,
                         documents=[c.text for c in candidates],
                         top_n=5)                     # ~$0.01
top_5 = [candidates[r["index"]] for r in reranked["results"]]
context = "\n\n".join(c.text for c in top_5)         # 5x smaller prompt

8. Set Budgets and Alerts

Cost runaway is the #1 reason teams overspend. Defense in depth:

Per-key daily/monthly budgets -- see API Keys
Per-model-group budgets -- cap aggregate spend across an alias
budget.threshold webhooks -- get notified at 50%, 80%, 100%
Per-request cost_cap -- reject requests projected to cost over a threshold

response = client.chat.completions.create(
    model="claude-opus-4",
    messages=msgs,
    metadata={"cost_cap_usd": 0.50},  # reject if estimated cost > $0.50
)

9. Bulk Where Possible

Batch operations are cheaper per item due to amortized fixed costs:

Embeddings batching -- one request with 100 inputs vs. 100 sequential requests; same token cost but ~10x lower per-request overhead
OpenAI Batch API -- 50% discount on 24-hour-turnaround completions (supported via Files API → batch jobs)
Async job patterns -- queue overnight workloads instead of inline on-request

10. Monitor and Iterate

Every other lever requires that you actually look at the numbers. The Analytics dashboard surfaces:

Top 10 most expensive requests in the last 7 days
Model-by-model cost vs. quality (refusal rate, healing rate)
Cache hit rate trend
p95 latency vs. cost trade-off

Set a recurring weekly 15-minute review.

A Concrete Combo

A typical SaaS support bot before optimization:

100% GPT-4o, no caching, no rerank, naive RAG with 20 docs in prompt
$4,200/month at 50K conversations

After applying the above:

gpt-4o-mini for intent classification (95% of routing)
claude-sonnet-4 only for the final response on hard cases (10%)
Semantic cache at threshold 0.88 (62% hit rate)
BYOK keys (0% commission)
Rerank top-5 instead of top-20
Weekly review

Result: $480/month -- an 89% reduction with no measurable quality drop.