Semantic Caching

3-tier caching system that saves 40-80% on repeated queries with exact match, semantic similarity, and provider passthrough.

Overview

AllRoutes includes a built-in 3-tier caching system that reduces costs and latency for repeated or similar queries. Caching can save 40-80% on workloads with repetitive prompts, such as FAQ bots, classification tasks, and templated generation.

How It Works

AllRoutes checks three cache tiers in order for every request:

Tier 1: Exact Match (SHA-256)

The fastest tier. A SHA-256 hash is computed from the full request body (model, messages, parameters). If an identical request has been made before, the cached response is returned instantly.

Hit rate: High for identical repeated queries
Latency: Sub-millisecond lookup
Use case: Identical API calls (retries, duplicate user messages)

Tier 2: Semantic Similarity (Vector KNN)

If no exact match is found, AllRoutes computes a vector embedding of the prompt and performs a K-nearest-neighbors search against cached prompts. If a cached prompt exceeds the similarity threshold (default: 0.92), its response is returned.

Hit rate: Medium-high for paraphrased or similar queries
Latency: ~5-15ms lookup
Similarity threshold: 0.92 (configurable)
Use case: FAQ bots, support agents, search queries with natural variation

Tier 3: Provider Passthrough

If neither cache tier matches, the request is forwarded to the upstream provider. The response is then stored in both cache tiers for future lookups.

Response Headers

Every response includes cache-related headers:

Header	Values	Description
`X-AllRoutes-Cache`	`hit`, `miss`	Whether the response came from cache
`X-AllRoutes-Cache-Tier`	`exact`, `semantic`	Which cache tier matched (only on hits)
`X-AllRoutes-Cache-Savings`	float	Estimated cost saved in USD (only on hits)
`X-AllRoutes-Cache-Similarity`	float	Semantic similarity score (only on semantic hits)

Configuration

Get Cache Config

GET https://api.allroutes.ai/v1/cache/config

{
  "enabled": true,
  "exact_match": true,
  "semantic_match": true,
  "similarity_threshold": 0.92,
  "ttl_seconds": 86400,
  "max_cache_size_mb": 500,
  "excluded_models": []
}

Update Cache Config

PUT https://api.allroutes.ai/v1/cache/config

curl -X PUT https://api.allroutes.ai/v1/cache/config \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "semantic_match": true,
    "similarity_threshold": 0.95,
    "ttl_seconds": 43200,
    "excluded_models": ["gpt-4o-realtime"]
  }'

Configuration Fields

Field	Type	Default	Description
`enabled`	boolean	`true`	Master switch for caching
`exact_match`	boolean	`true`	Enable Tier 1 exact match
`semantic_match`	boolean	`true`	Enable Tier 2 semantic similarity
`similarity_threshold`	float	`0.92`	Minimum cosine similarity for semantic cache hits (0.0-1.0)
`ttl_seconds`	integer	`86400`	Time-to-live for cached entries in seconds (default: 24 hours)
`max_cache_size_mb`	integer	`500`	Maximum cache storage per organization
`excluded_models`	array	`[]`	Models to exclude from caching

Purge Cache

POST https://api.allroutes.ai/v1/cache/purge

Clear cached entries. You can purge all entries or filter by model or time range.

Request Body

Field	Type	Description
`model`	string	Purge entries for a specific model (optional)
`before`	string	Purge entries created before this ISO 8601 timestamp (optional)
`all`	boolean	Purge all cache entries (default: `false`)

# Purge all cache
curl -X POST https://api.allroutes.ai/v1/cache/purge \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"all": true}'

# Purge cache for a specific model
curl -X POST https://api.allroutes.ai/v1/cache/purge \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o"}'

Per-Request Cache Control

Disable caching for individual requests using the cache_control field:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "What time is it?"}],
  "cache_control": {
    "no_cache": true
  }
}

Field	Type	Description
`no_cache`	boolean	Skip cache lookup and do not store the response
`no_store`	boolean	Allow cache lookup but do not store the response
`max_age`	integer	Override TTL for this specific response (seconds)

Cost Savings

Typical cache savings by workload type:

Workload	Cache Hit Rate	Cost Savings
FAQ / Support bots	60-80%	55-75%
Classification tasks	50-70%	45-65%
Templated generation	40-60%	35-55%
Code completion	20-40%	15-35%
Creative writing	5-15%	5-10%

Best Practices

Leave semantic caching enabled for conversational workloads -- even slight query variations will produce cache hits.
Tune the similarity threshold -- lower values (0.85) increase hit rates but may return less relevant cached responses. Higher values (0.95+) are more precise.
Exclude real-time models from caching where freshness matters (e.g., models used with web search plugins).
Monitor cache headers to track savings and optimize your configuration.

Anthropic Prompt Caching

For Anthropic models, the gateway also forwards block-level cache_control markers so you can take advantage of Claude's native prompt caching (90% input-token discount on cache hits). Mark the long, stable parts of your prompt:

{
  "model": "claude-sonnet-4-20250514",
  "messages": [
    {
      "role": "system",
      "content": [
        {"type": "text", "text": "<long system prompt>", "cache_control": {"type": "ephemeral"}}
      ]
    },
    {"role": "user", "content": "What does the system prompt say?"}
  ]
}

This is independent of AllRoutes' 3-tier cache -- they stack.