Analytics

Built-in dashboards for cost, latency, quality, and usage breakdowns by model, key, user, and session.

Overview

The AllRoutes dashboard ships with analytics for every request that flows through the gateway. Out of the box you get:

Cost -- spend over time, broken down by model, provider, key, user, or session
Latency -- p50, p95, p99 distributions per model and per route
Throughput -- requests per minute, tokens per second
Quality -- prompt/completion token ratios, finish-reason breakdowns, healing trigger rate
Cache -- hit rate, savings, similarity score distribution
Errors -- 4xx/5xx counts, top error types, retry-after distribution

Data is computed in ClickHouse from the same trace pipeline that drives Broadcast, so dashboard numbers always match what your external observability tools see.

Where to Find It

In the dashboard:

Overview -- one-page summary across all keys and models
Models -- drill into per-model cost, latency, quality
Keys -- per-key budget tracking and usage
Users -- per-end-user tracking (when user field is set on requests)
Sessions -- session-grouped traces (when session_id is set)

Tagging Requests

Analytics dimensions are populated from the request body. To get rich breakdowns, tag every request:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Hello"}],
  "user": "usr_123",
  "session_id": "sess_456",
  "metadata": {
    "feature": "support-bot",
    "environment": "production",
    "tenant_id": "acme-corp"
  }
}

Field	Indexed As	Use For
`user`	`user`	Per-end-user usage and abuse detection
`session_id`	`session`	Conversation-level cost and replay
`metadata.*`	Custom dimensions	Feature/team/tenant breakdowns

metadata keys appear automatically as filterable dimensions in the dashboard.

Programmatic Access

GET https://api.allroutes.ai/v1/analytics/usage

Query Param	Type	Description
`start_date`	string	ISO 8601
`end_date`	string	ISO 8601
`granularity`	string	`hour`, `day`, `week`, `month`
`group_by`	string	`model`, `provider`, `key`, `user`, `session` (comma-separated)
`model`	string	Filter to a single model
`key_id`	string	Filter to a single API key

curl "https://api.allroutes.ai/v1/analytics/usage?start_date=2026-04-01&end_date=2026-04-30&granularity=day&group_by=model" \
  -H "Authorization: Bearer allroutes_sk_..."

Response

{
  "data": [
    {"date": "2026-04-01", "model": "gpt-4o", "requests": 1234, "input_tokens": 456789, "output_tokens": 123456, "cost_usd": 12.34, "p50_latency_ms": 850, "p95_latency_ms": 2400, "cache_hit_rate": 0.32},
    {"date": "2026-04-01", "model": "claude-sonnet-4-20250514", "requests": 567, "input_tokens": 234567, "output_tokens": 78901, "cost_usd": 8.76, "p50_latency_ms": 920, "p95_latency_ms": 2800, "cache_hit_rate": 0.28}
  ]
}

Per-Request Inspection

Every request gets a unique request_id (returned in the X-AllRoutes-Request-ID header and in the response body). Look up the full trace at:

GET https://api.allroutes.ai/v1/analytics/requests/:request_id

{
  "id": "req_xyz789",
  "model": "gpt-4o",
  "provider": "openai",
  "user": "usr_123",
  "session_id": "sess_456",
  "metadata": {"feature": "support-bot"},
  "latency_ms": 1234,
  "input_tokens": 28,
  "output_tokens": 156,
  "cost_usd": 0.0032,
  "cache": {"status": "miss", "tier": null, "savings_usd": 0},
  "guardrails": {"pii_redacted": 0, "injection_score": 0.02},
  "fallbacks": [],
  "created_at": "2026-04-05T14:30:00Z"
}

By default, prompt and response content are not stored. Enable content retention for debugging by toggling Settings > Analytics > Store Content in the dashboard.

Custom Dashboards

For deeper exploration, Broadcast the same trace stream to BigQuery, Snowflake, or Datadog. The schema matches the API responses above, so SQL queries are straightforward:

SELECT
  toDate(created_at) AS day,
  model,
  count(*) AS requests,
  sum(cost_usd) AS spend,
  quantile(0.95)(latency_ms) AS p95_ms,
  countIf(cache.status = 'hit') / count(*) AS cache_hit_rate
FROM allroutes_traces
WHERE created_at >= now() - INTERVAL 30 DAY
GROUP BY day, model
ORDER BY day DESC, spend DESC;

Quality Metrics

The dashboard surfaces two quality proxies that don't require LLM-as-judge evaluation:

Healing rate -- % of responses that triggered response-healing to fix malformed JSON / markdown
Refusal rate -- % of responses where finish_reason = "content_filter" or the response body matches refusal patterns

For deeper quality evaluation (LLM-as-judge, regression detection), use the trace stream with Langfuse or LangSmith.

Best Practices

Always tag with user and session_id -- without them you can only see aggregate numbers
Use metadata consistently -- pick a stable schema (environment, feature, tenant_id) and apply it to every request
Set per-key budget alerts -- in the dashboard under Keys > Alerts to get a webhook before a key blows its monthly cap
Review the cache hit rate weekly -- a sudden drop usually means a code path stopped using a stable system prompt