Works with everything you already use

LangChain talks to AllRoutes via the standard OpenAI chat model with our base URL. Smart routing, semantic cache, and BYOK all light up under the hood — your ChatOpenAI keeps the same interface.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="claude-sonnet-4-6:nitro",
    base_url="https://api.allroutes.ai/v1",
    api_key=os.environ["ALLROUTES_KEY"],
)

LlamaIndex's OpenAI LLM and OpenAIEmbedding both accept our base URL. You get LlamaIndex's vector / graph indices plus AllRoutes' routing, caching, and observability.

from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-4o:nitro",
    api_base="https://api.allroutes.ai/v1",
    api_key=os.environ["ALLROUTES_KEY"],
)

Our streaming endpoint returns standard SSE that Next 15 Route Handlers can pipe straight to the client. Works in the Edge Runtime as well as Node.

export async function POST(req: Request) {
  const upstream = await fetch("https://api.allroutes.ai/v1/chat/completions", {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.ALLROUTES_KEY}` },
    body: await req.text(),
  });
  return new Response(upstream.body, { headers: { "Content-Type": "text/event-stream" } });
}

Use @ai-sdk/openai with the AllRoutes base URL. Server actions, edge functions, and Next.js Route Handlers all work without modification — streaming included.

import { createOpenAI } from "@ai-sdk/openai";

const allroutes = createOpenAI({
  apiKey: process.env.ALLROUTES_KEY!,
  baseURL: "https://api.allroutes.ai/v1",
});

Set OTEL_EXPORTER_OTLP_ENDPOINT to your Datadog Agent and AllRoutes' built-in span emitter writes routing decisions, provider latencies, and cache hit-rates directly into your existing APM views.

We attach traceparent to every upstream provider call and emit spans for gateway.route, provider.call, and cache.lookup. Configure your collector with the standard OTLP endpoint — no AllRoutes-specific receiver needed.

Add ALLROUTES_KEY as a project env variable, then call our API from any Vercel function — Edge Runtime, Serverless, or Edge Middleware. The streaming endpoint plays well with both Response body forwarding and the AI SDK helpers.

Use AllRoutes' embeddings endpoint with text-embedding-3-large or any other model — same interface as OpenAI. Pinecone receives standard float vectors; routing happens transparently on our side.

emb = client.embeddings.create(
  model="text-embedding-3-large",
  input=chunks,
)
pinecone_index.upsert([(id, e.embedding) for id, e in zip(ids, emb.data)])