Updates, engineering deep-dives, and guides from the AllRoutes.ai team.
We built AllRoutes because integrating four LLM providers in one product means writing four error handlers, four retry policies, and four cost-tracking pipelines. Here's the architecture we shipped instead.
We benchmarked cost-optimized routing across 18 production workloads. The median saving was 41% and the p10 was 18%. Here is how the router actually decides, and why naive 'cheapest model wins' loses money.
Exact-match caches miss almost everything. We built a two-tier semantic cache on top of a HNSW index that catches paraphrased prompts at 92% precision. Here is the architecture and the gotchas.
Bring-your-own-key sounds simple until you realize you have a database full of plaintext OpenAI keys. Here is the envelope encryption design we shipped, why we picked KMS over local keys, and what we got wrong the first time.