Self-Hosted Models

Connect Ollama, vLLM, TGI, or any OpenAI-compatible endpoint to AllRoutes for unified routing, caching, and observability.

Overview

AllRoutes can route requests to your own self-hosted models alongside cloud providers. Connect any OpenAI-compatible endpoint -- Ollama, vLLM, Text Generation Inference (TGI), or a custom server -- and AllRoutes treats it as a first-class provider with full routing, caching, guardrails, and observability support.

Supported Runtimes

Runtime	Protocol	Use Case
Ollama	OpenAI-compatible	Local development, small-scale inference
vLLM	OpenAI-compatible	High-throughput production serving
TGI (Text Generation Inference)	OpenAI-compatible	Hugging Face model serving
LocalAI	OpenAI-compatible	Multi-model local inference
LiteLLM	OpenAI-compatible	Proxy with model translation
Custom server	OpenAI-compatible	Any server implementing `/v1/chat/completions`

Any server that accepts the OpenAI chat completions format works with AllRoutes.

Adding a Custom Endpoint

Via Dashboard

Navigate to Settings > Custom Endpoints
Click Add Endpoint
Enter the base URL (e.g., http://gpu-server.internal:11434/v1)
Optionally add an API key if your endpoint requires authentication
Click Test & Save

Via API

curl -X POST https://api.allroutes.ai/v1/endpoints \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ollama-local",
    "base_url": "http://gpu-server.internal:11434/v1",
    "api_key": "",
    "provider_type": "ollama",
    "health_check": true,
    "auto_discover": true
  }'

Response:

{
  "id": "ep_abc123",
  "name": "ollama-local",
  "base_url": "http://gpu-server.internal:11434/v1",
  "provider_type": "ollama",
  "status": "healthy",
  "models_discovered": [
    "llama3.1:70b",
    "codellama:34b",
    "mistral:7b"
  ],
  "created_at": "2026-04-05T10:00:00Z"
}

Auto-Discovery

When auto_discover is enabled, AllRoutes queries your endpoint's /v1/models route to automatically detect all available models. Discovered models appear in the dashboard and can be used in API requests like any cloud model.

Auto-discovery runs:

On creation -- immediately when the endpoint is added
Every 5 minutes -- periodic refresh to detect new or removed models
On demand -- trigger a refresh via the API

curl -X POST https://api.allroutes.ai/v1/endpoints/ep_abc123/discover \
  -H "Authorization: Bearer allroutes_sk_..."

Health Checks

AllRoutes monitors your custom endpoints with periodic health checks:

Interval: Every 30 seconds
Method: Lightweight GET /v1/models request
Timeout: 5 seconds
Failure threshold: 3 consecutive failures mark the endpoint as unhealthy

When an endpoint is unhealthy, AllRoutes automatically routes requests to fallback models (cloud providers or other healthy endpoints).

Health Status

Status	Description
`healthy`	Endpoint is responding normally
`degraded`	Elevated latency or intermittent errors
`unhealthy`	Endpoint is down; traffic is rerouted
`unknown`	Initial state before first health check

Check endpoint health:

curl https://api.allroutes.ai/v1/endpoints/ep_abc123/health \
  -H "Authorization: Bearer allroutes_sk_..."

{
  "status": "healthy",
  "latency_ms": 12,
  "last_check": "2026-04-05T14:30:00Z",
  "uptime_percent": 99.8,
  "models_online": 3
}

Using Self-Hosted Models

Once connected, use your self-hosted models by their discovered name with a self/ prefix:

{
  "model": "self/llama3.1:70b",
  "messages": [{ "role": "user", "content": "Hello!" }]
}

Or reference the endpoint name directly:

{
  "model": "ollama-local/llama3.1:70b",
  "messages": [{ "role": "user", "content": "Hello!" }]
}

Fallback to Cloud

Combine self-hosted and cloud models in a fallback chain:

{
  "models": ["self/llama3.1:70b", "meta-llama/llama-3.1-70b-instruct", "gpt-4o"],
  "messages": [{ "role": "user", "content": "Summarize this document." }]
}

AllRoutes tries your self-hosted model first (zero cost), then falls back to a cloud-hosted version, then to GPT-4o.

Connecting Ollama

# Start Ollama (default port 11434)
ollama serve

# Pull a model
ollama pull llama3.1:70b

# Add to AllRoutes
curl -X POST https://api.allroutes.ai/v1/endpoints \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ollama",
    "base_url": "http://localhost:11434/v1",
    "provider_type": "ollama",
    "auto_discover": true
  }'

Connecting vLLM

# Start vLLM with OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --port 8000

# Add to AllRoutes
curl -X POST https://api.allroutes.ai/v1/endpoints \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "vllm-llama",
    "base_url": "http://gpu-server:8000/v1",
    "provider_type": "vllm",
    "auto_discover": true
  }'

Connecting TGI

# Start TGI with OpenAI-compatible API
docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-70B-Instruct

# Add to AllRoutes
curl -X POST https://api.allroutes.ai/v1/endpoints \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "tgi-llama",
    "base_url": "http://gpu-server:8080/v1",
    "provider_type": "tgi",
    "auto_discover": true
  }'

Managing Endpoints

Method	Endpoint	Description
`POST`	`/v1/endpoints`	Add a new custom endpoint
`GET`	`/v1/endpoints`	List all custom endpoints
`GET`	`/v1/endpoints/:id`	Get endpoint details
`PUT`	`/v1/endpoints/:id`	Update an endpoint
`DELETE`	`/v1/endpoints/:id`	Remove an endpoint
`GET`	`/v1/endpoints/:id/health`	Check endpoint health
`POST`	`/v1/endpoints/:id/discover`	Trigger model auto-discovery

Benefits

Zero cost -- no per-token charges for self-hosted models
Unified API -- same OpenAI-compatible format for self-hosted and cloud models
Full features -- caching, guardrails, plugins, and broadcast work on self-hosted models
Automatic failover -- fall back to cloud providers when self-hosted models are down
Centralized observability -- monitor self-hosted and cloud usage in one dashboard