Configuration

Auto AI Router is configured via a YAML file passed with the -config flag.

See the full example in config.yaml.example.

Full Example

server:
  port: 8080
  max_body_size_mb: 100
  response_body_multiplier: 10
  response_compatibility: native
  request_timeout: 60s
  write_timeout: 60s
  idle_timeout: 2m
  idle_conn_timeout: 120s
  max_idle_conns: 200
  max_idle_conns_per_host: 20
  logging_level: info
  master_key: "sk-your-master-key-here"
  default_models_rpm: -1
  credential_name_as_team_id: false
  model_prices_link: ""
  model_prices_sync_interval: 5m
  response_headers:
    mode: passthrough

fail2ban:
  max_attempts: 3
  ban_duration: permanent
  error_codes: [401, 403, 429, 500, 502, 503, 504]
  # error_code_rules:
  #   - code: 429
  #     max_attempts: 5
  #     ban_duration: 5m

monitoring:
  prometheus_enabled: true
  log_errors: false
  errors_log_path: "logs/logs.jsonl"

credentials:
  - name: "openai_main"
    type: "openai"
    api_key: "sk-proj-xxxxx"
    base_url: "https://api.openai.com"
    rpm: 100
    tpm: 50000

  - name: "vertex_ai"
    type: "vertex-ai"
    project_id: "your-project-id"
    location: "global"
    credentials_file: "path/to/service-account.json"
    rpm: 100
    tpm: 50000

  - name: "gemini_studio"
    type: "gemini"
    api_key: "os.environ/GEMINI_API_KEY"
    base_url: "https://generativelanguage.googleapis.com"
    rpm: 60
    tpm: -1

  - name: "air_fallback"
    type: "air"
    base_url: "http://backup-router.local:8080"
    api_key: "sk-remote-master-key"
    rpm: 200
    tpm: 100000
    is_fallback: true

models:
  - name: "gpt-4o"
    credential: openai_main
    rpm: 100
    tpm: 50000
  - name: "gemini-2.5-pro"
    credential: vertex_ai
    rpm: 100
    tpm: 50000

litellm_db:
  enabled: false
  is_required: false
  database_url: "os.environ/LITELLM_DATABASE_URL"
  max_conns: 25
  min_conns: 5
  health_check_interval: 10s
  connect_timeout: 5s
  auth_cache_ttl: 20s
  auth_cache_size: 10000
  log_queue_size: 5000
  log_batch_size: 100
  log_flush_interval: 5s
  include_team_spend_in_user_spend: true
  log_retry_attempts: 3
  log_retry_delay: 1s

Server Parameters

The allowlist mode forwards Content-Type, Cache-Control, Retry-After, Content-Disposition, Content-Range, Last-Modified, and Location. Transport headers are generated by the router. Other upstream response headers are removed.

When credential_name_as_team_id is enabled, spend logs use the selected provider credential name as team_id only when the authenticated key does not provide a team_id. An explicit team assignment always takes precedence. This is intended for billing-aggregator AIR instances that group spend by provider credential; leave it disabled for embedded routers and migration helpers.

model_prices_sync_interval controls how often the price source is re-read after the initial startup load. It accepts any Go duration string (30s, 15m, 1h) and is ignored when model_prices_link is empty. A failed refresh keeps the previously loaded prices and is retried on the next tick; a missing or non-positive value falls back to 5m. See Model Pricing for guidance on picking a value.

When PostgreSQL or Kafka spend logging is enabled, every routed model must have a price entry. If neither its real provider name nor its alias can be priced, the router returns 503 Model pricing unavailable before contacting the provider. Use an explicit zero-price entry for intentionally free models.

tiktoken_enabled controls the local prompt/completion token estimator used as a fallback for streaming responses when a provider doesn't report usage (e.g. the stream is cut before the final usage chunk, or the provider omits token counts entirely), and for budget-reservation cost estimates ahead of the request. Two different costs apply while it's on: the prompt-token estimate and the final completion-token BPE count are lazy — computed only if a stream actually finishes without provider-reported usage — so they're free for providers that do report usage. However, per-chunk delta-text accumulation runs on every streaming chunk of every stream while this flag is on, regardless of whether the provider ultimately reports usage, since the accumulator has to keep pace with the stream in case it's needed at the end.

Set to false to skip local estimation entirely if all your configured providers always report usage and you'd rather not carry any tiktoken-related per-chunk cost. This has three effects, not just spend logging: spend logs show 0 prompt/completion tokens (instead of an estimate) for streaming requests where a provider omits usage; the streaming-fallback total that feeds rateLimiter.ConsumeTokens/ConsumeModelTokens is zeroed the same way; and budget-reservation cost estimates (server.litellm_db.enforce_budget_reservation) undercount, reflecting only the completion-token portion — reservation itself still runs, it just can't account for prompt-token cost without local tokenization.

On SIGTERM the router reports not ready, keeps serving for shutdown_delay so that the load balancer stops sending it new requests, then stops accepting and gives the requests still in flight shutdown_timeout to finish. Requests still running then are cancelled and handled as if their clients had disconnected, so their spend is still logged; with drain_upstream_on_abort a cancelled request keeps reading the provider for up to a minute to capture its real usage, as after a real disconnect, and the router waits for that. The router then flushes queued spend logs to Kafka and PostgreSQL and closes its connections. To restart without cutting requests, set shutdown_timeout to the longest a request may run (e.g. your ingress route timeout), and give the process at least shutdown_delay + shutdown_timeout + 30s to stop (Kubernetes: terminationGracePeriodSeconds), or + 90s with drain_upstream_on_abort. The extra time covers the cancelled requests and the spend-log flush; if the process is killed first, requests still running are cut and spend entries still queued in memory are lost.

max_connection_lifetime bounds how long a provider connection takes new requests. A pooled connection is otherwise reused for as long as requests keep coming, which under steady load is until something cuts it, so a proxy on the way (proxy_url) can never drain its tunnels: each carries the router's own TLS session with the provider, and restarting the proxy cuts the requests on them. With max_connection_lifetime set, no request is sent on a connection older than that, and each connection closes as soon as the request on it ends; no request is cut. A proxy that stops taking new tunnels but keeps its open ones for max_connection_lifetime plus the longest request therefore restarts without cutting anything. The cost is a new connection (TCP, the proxy's CONNECT, TLS) for about average request duration / max_connection_lifetime of the requests: with 7.6 s requests and 10m, 1.3%.

Fail2Ban Parameters

The fail2ban block bans a credential + model pair after repeated failures at configured HTTP status codes, so a broken or rate-limited upstream stops eating a share of round-robin traffic. See Fail2Ban for the full parameter reference, per-error-code rules, and per-credential overrides (for upstreams — e.g. resellers/aggregators — that signal failure with a different status code than the rest of the pool).

Retry Parameters

The retry block decides which upstream errors are replayed on the next credential of the same model. By default, self-hosted vllm credentials do not retry 400. See Retries and Request Cleanup for status code sets, body markers and per-provider and per-credential overrides.

Monitoring Parameters

The /health endpoint is always available and cannot be disabled or reconfigured.

Credentials

Each credential defines a connection to an LLM provider. See Providers for details on each type.

Common fields for all credentials:

Outbound proxy per credential

Set one proxy_url on a credential to route its upstream requests through a fixed proxy:

credentials:
  - name: openai_main
    type: openai
    api_key: os.environ/OPENAI_API_KEY
    base_url: https://api.openai.com
    proxy_url: os.environ/OPENAI_PROXY_URL

For example, set OPENAI_PROXY_URL=http://user:password@proxy.example.com:3128. Supported schemes: http, https, socks5, and socks5h. Percent-encode special characters in usernames and passwords. Both SOCKS5 schemes resolve destination hostnames through the proxy.

The setting applies to ordinary and streaming provider requests, Vertex AI OAuth token acquisition and refresh, and /health and /v1/models discovery for air and proxy credentials. An explicit URL overrides HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. When omitted or empty, the existing environment-based behavior is preserved. Invalid URLs are rejected during config loading without exposing proxy credentials in the error. If the proxy fails, the request fails through the usual credential retry/fallback flow; it does not retry the same credential directly. Proxy lists and rotation are not supported.

Fixed upstream request headers

By default the router forwards the client's own headers (except authentication, hop-by-hop, client-IP and internal AIR headers) to the provider, including its User-Agent. Set request_headers on a credential to send fixed values instead, for example when the provider's WAF rejects some client User-Agents (Novita's Cloudflare answers Python-urllib/* with 403 error code: 1010):

credentials:
  - name: novita
    type: openai
    api_key: os.environ/NOVITA_API_KEY
    base_url: https://api.novita.ai/openai
    request_headers:
      User-Agent: "auto-ai-router/1.0"      # replaces the client's User-Agent
      X-Provider-Token: os.environ/NOVITA_TOKEN
      X-Client-Hint: ""                     # empty: never send this header

• The configured value replaces every value the client sent under the same name (names are case-insensitive). An empty or blank value removes the header; for User-Agent this sends none at all rather than Go's default Go-http-client.
• Applies to every request sent with the credential: regular and streaming HTTP requests, retries (a retry on another credential uses that credential's headers) and the native WebSocket handshake of /v1/responses.
• Values support os.environ/VAR. A variable that is unset or blank is a config error, not a silent removal. Surrounding spaces and tabs are trimmed.
• Values are never logged: startup logs list header names only, and the debug dump of outbound headers shows [credential request_headers] instead.
• Only for direct providers. air and proxy credentials reject the field: the remote router would treat the headers as client headers and forward them to every provider behind it. Set them on the provider credential of the router that actually calls the provider.
• Headers the router manages itself are rejected at load time: Authorization, X-Api-Key, X-Goog-Api-Key, Host, Content-*, Transfer-Encoding, Expect, Accept-Encoding, Anthropic-Beta, Anthropic-Version, Origin, hop-by-hop headers (Connection, Keep-Alive, Proxy-Connection, Proxy-Authenticate, Proxy-Authorization, TE, Trailer, Upgrade), and the Air-* and Sec-WebSocket-* families. Invalid header names and values with control characters (CR, LF, NUL, ...) are rejected too.

Scoped credential visibility

Credential scopes let one shared AIR instance expose different providers to different API keys. If scopes is omitted or empty, the credential remains available to everyone. If scopes is set, a request can use the credential only when the request API key has at least one matching scope. denied_scopes always wins over scopes.

credentials:
  - name: shared-openai
    type: openai
    api_key: "os.environ/OPENAI_KEY"
    base_url: "https://api.openai.com/v1"
    rpm: -1

  - name: team-a-openai
    type: openai
    api_key: "os.environ/TEAM_A_OPENAI_KEY"
    base_url: "https://api.openai.com/v1"
    rpm: -1
    scopes: [team-a]

  - name: all-except-team-a
    type: openai
    api_key: "os.environ/SHARED_KEY"
    base_url: "https://api.openai.com/v1"
    rpm: -1
    denied_scopes: [team-a]

For LiteLLM DB auth, request scopes are read from API key metadata:

{
  "air_scopes": ["team-a"],
  "air_denied_scopes": ["premium"]
}

If air_scopes is not set, AIR falls back to the LiteLLM API key name (key_name, then key_alias). For example, a key named team-a gets the team-a scope automatically. The master key bypasses scope filters.

For DB-loaded credentials, add the same fields to LiteLLM_CredentialsTable.credential_info:

{
  "custom_llm_provider": "openai",
  "air_scopes": ["team-a"],
  "air_denied_scopes": ["premium"]
}

Scope filtering applies to routing, retry/fallback selection, /health, /v1/models, /trace, /vhealth, and /vtrace.

In both LiteLLM API-key metadata and LiteLLM_CredentialsTable.credential_info, air_forbidden_scopes is accepted as an alias for air_denied_scopes.

Set reasoning_only: true on a credential when its provider accepts only reasoning traffic:

credentials:
  - name: anthropic-reasoning
    type: anthropic
    api_key: "os.environ/ANTHROPIC_API_KEY"
    base_url: "https://api.anthropic.com"
    reasoning_only: true

The filter recognizes enabled reasoning_effort, reasoning, thinking, thinking_budget, and thinking_level parameters. Requests that omit reasoning or explicitly disable it skip the credential during initial selection, sticky routing, retries, and fallback selection. For DB-loaded credentials, use "air_reasoning_only": true in credential_info.

Models

The models section binds specific models to credentials and optionally sets per-model rate limits.

models:
  - name: "gpt-4o"
    credential: openai_main
    rpm: 100
    tpm: 50000

By default, all models are available through all credentials. Use the models section to restrict which credentials serve which models.

A model entry can also set reasoning_effort_map, which rewrites the client's reasoning effort into a value the model accepts (e.g. {minimal: low, high: medium, default: low}). See Reasoning effort mapping.

By default, models can also be declared directly inside a credential via the models: field — they are automatically extracted and added to the global models list with the credential name pre-filled.

See Load Balancing for details on multi-credential routing.

Set supports_vision: false on a model that cannot accept images; what happens to images sent to it is configured in the top-level vision_fallback section. See Vision Fallback.

models:
  - name: "glm-text"
    credential: vllm_glm
    supports_vision: false

vision_fallback:
  mode: describe            # reject | strip | describe
  describe_model: qwen-vl

YAML Anchors for Models

When many credentials share the same set of models, YAML anchors eliminate repetition. Define a template once with &anchor-name and reference it with *anchor-name.

List anchor in x-model-templates

The x-model-templates top-level key is a dedicated namespace for anchor definitions. It is not processed by the router — its sole purpose is to hold anchors so they can be referenced elsewhere.

x-model-templates:
  vertex-base-models: &vertex-base-models
    - name: gemini-2.5-flash
      rpm: 100
      tpm: 50000
    - name: gemini-2.5-pro
      rpm: 50
      tpm: 100000

credentials:
  - name: "vertex_v1"
    type: "vertex-ai"
    project_id: "proj-1"
    location: "global"
    credentials_file: "keys/proj-1.json"
    rpm: 100
    models: *vertex-base-models   # expands to the full list

  - name: "vertex_v2"
    type: "vertex-ai"
    project_id: "proj-2"
    location: "global"
    credentials_file: "keys/proj-2.json"
    rpm: 100
    models: *vertex-base-models   # same list, credential set to "vertex_v2"

Each model copy automatically gets the parent credential name injected, so no manual credential: field is needed inside the template.

Single-model anchor

An anchor can also target a single model mapping and be used as an item in a models: list:

x-model-templates:
  flash: &flash
    name: gemini-2.5-flash
    rpm: 100
    tpm: 50000

credentials:
  - name: "vertex_v1"
    type: "vertex-ai"
    project_id: "proj-1"
    location: "global"
    credentials_file: "keys/proj-1.json"
    rpm: 100
    models:
      - *flash               # single model from anchor
      - name: gemini-2.5-pro # inline model
        rpm: 50
        tpm: 100000

Expanding a list anchor inside the top-level models: section

A list anchor can be expanded inline within the top-level models: sequence. The router flattens the result so all items end up as a flat list:

x-model-templates:
  shared-models: &shared-models
    - name: gemini-2.5-flash
      credential: vertex_v1
      rpm: 100
      tpm: 50000
    - name: gemini-2.5-pro
      credential: vertex_v1
      rpm: 50
      tpm: 100000

models:
  - *shared-models        # expands and flattens both items into the list
  - name: gpt-4o
    credential: openai_main
    rpm: 60
    tpm: 80000

Supported combinations