← Back to Blog
Announcements2026-03-01·6 min read

Introducing AllRoutes.ai: A Unified AI API Gateway

We built AllRoutes because integrating four LLM providers in one product means writing four error handlers, four retry policies, and four cost-tracking pipelines. Here's the architecture we shipped instead.

When we started building AI features into our internal tooling at the end of 2024, the integration story was already messy. OpenAI's SDK used messages[].content as either a string or a list of content parts. Anthropic's SDK split the system prompt into a top-level field. Google's Vertex SDK wanted protobuf-shaped requests behind an OAuth2 service account. Each one had a different streaming format, a different rate-limit error shape, and a different idea of what "tool calls" looked like on the wire.

We tried abstraction layers. We tried writing a thin wrapper. We tried just accepting the duplication. None of it scaled past two providers, and by the time we needed five we had three engineers spending half their week on glue code.

AllRoutes.ai is the gateway we wished existed back then. Today it routes traffic to 100+ models behind a single OpenAI-compatible endpoint, and it does so with smart routing, semantic caching, BYOK, and usage analytics built into the request path itself, not bolted on afterwards.

The shape of the problem

Most teams hit the same three walls when they wire up multiple LLM providers:

  1. Schema drift. Every provider has a slightly different request and response shape. You either pick one provider and lock in, or you write if/else branches that grow forever.
  2. Operational drift. Each provider has its own auth scheme, its own quota mechanics, and its own incident behavior. A 429 from Anthropic does not mean the same thing as a 429 from OpenAI.
  3. Cost drift. GPT-4o and Claude Sonnet 4 cost different amounts per million tokens, those prices change, and their quality on your specific traffic mix is not what the leaderboard says it is.

A naive proxy solves none of these. AllRoutes solves all three by treating the gateway as a programmable layer rather than a passthrough.

What ships in v1

The public API is OpenAI-compatible. Pointing your existing client at https://api.allroutes.ai/v1 and swapping the API key gets you running in under a minute:

from openai import OpenAI

client = OpenAI(
    api_key="allroutes_sk_...",
    base_url="https://api.allroutes.ai/v1",
)

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Behind that endpoint sits a Go gateway built on Gin, a Python adapter service that normalizes provider-specific quirks, and a fleet of background workers handling billing reconciliation, webhook delivery, analytics rollups, health monitoring, quality eval, and the marketplace. Eighteen Compose services in total. Postgres for transactional state, Redis for hot caches, NATS for inter-service messaging, ClickHouse for analytics events.

The headline features:

  • Smart routing. Pick a strategy per request: cost-optimized, latency-optimized, quality-optimized, or strict failover. The router uses real-time pricing, p95 latency observed in the last hour, and quality scores from our evaluator workers.
  • Semantic caching. Embedding-based cache that recognizes paraphrased prompts. Caches in two tiers: exact match for the obvious wins, and ANN-based similarity for the long tail.
  • BYOK. Bring your own provider keys, encrypted per-user with envelope encryption. We never see plaintext provider keys after upload.
  • Compass Radar. Our quality observability layer that scores live traffic against rubrics you define and flags regressions when a provider quietly downgrades a model.

Compass Radar in particular

The thing we are proudest of is Compass Radar. Most teams treat LLM quality as a deploy-time concern: you pick a model, you eyeball some outputs, you ship. Compass Radar runs evaluations continuously against a sliding window of your real traffic. It catches the case where Anthropic ships a quiet update to claude-sonnet-4 and the average task-completion score on your customer-support workload drops three points overnight. You get a webhook before your users start complaining.

The way this works in practice: every Nth request in your account (configurable, default 1%) gets shadow-evaluated against an LLM-as-judge rubric you wrote in plain English. The judge runs on a held-out model so the eval signal is independent of the production model. Scores get rolled up into ClickHouse, and the Radar UI shows you per-model trendlines with confidence intervals.

What we are not

AllRoutes is not a model hosting service. We do not run weights. Every request is a request to a real upstream provider, with the latency and reliability properties of that provider. Our job is to make the layer above the provider better, not to replace the provider.

AllRoutes is also not a fine-tuning platform, an agent framework, or a vector database. There are good products in each of those spaces and we integrate with them rather than competing.

What is next

The roadmap for the next quarter is heavy on the routing engine. We want quality-aware routing that learns per-customer, per-task what actually works, rather than using a global quality score. We want better support for tool-calling differences across providers. We want native MCP support so an MCP-aware client can pull tool schemas from the gateway directly.

If you have been juggling provider SDKs for the last year, give AllRoutes a try. The free tier covers most prototyping workloads, and the quickstart is genuinely five minutes. We would also love to hear what is broken about your current setup.

← Back to Blog