Prometheus Metrics

Enable Prometheus metrics in the config:

monitoring:
  prometheus_enabled: true

Metrics are available at /metrics.

The same metrics can also be pushed to an OTLP collector instead of (or in addition to) being scraped — see OpenTelemetry.

Available Metrics

Per-Key Metrics

Opt-in request counters per LiteLLM API key. Requires litellm_db.enabled: true (keys are identified through LiteLLM token validation) and prometheus_enabled or otel.enabled; otherwise the flag is ignored with a startup warning.

monitoring:
  prometheus_enabled: true
  key_metrics:
    enabled: true
    # Owner labels on auto_ai_router_key_info. Default (unset): all but
    # user_email; [] disables owner labels (the info series keeps only `key`).
    info_labels: [key_alias, user_id, team_id, team_alias, organization_id]
    max_keys: 5000 # distinct keys; later keys are counted as key="__other__" (0 = unlimited)
    idle_ttl: 24h # drop series of keys without traffic this long (0 = never)

• key is the first 12 hex characters of the key's sha256 hash (find the key with WHERE token LIKE '<key>%'), or master for the router master key. The full hash is never exposed: LiteLLM auth accepts a hashed token as a bearer credential.
• status is the HTTP status the client received, so budget (402) and key RPM/TPM (429) rejections are counted. Requests rejected during authentication (unknown, expired or blocked key) have no key and are not counted. A streaming response that failed mid-stream after a 200 counts as 200. A client that disconnected before any status was sent (or aborted a WebSocket turn) counts as 499, as in nginx; exclude it from error rates with status!="499".
• Every authenticated API call counts, including GET /v1/models. Each WebSocket Responses turn counts as one request; the 101 upgrade itself is not counted.
• Owner labels are only on auto_ai_router_key_info; join them in PromQL on (key, instance). Every replica exports its own info series for the keys it served, so joining on key alone fails with "many-to-many matching" as soon as there is more than one replica. With otel.enabled only (no /metrics scrape), make sure the OTLP pipeline still gives each replica a distinct instance label (e.g. from service.instance.id), or the join fails the same way. Changing a key's alias replaces its info series, so the join stays one-to-one.
• The join drops key="__other__" (no info series), so once max_keys is reached every joined panel silently undercounts. The bundled Grafana dashboards show it as the "Overflow (__other__)" value of the Key RPM stat; alert on sum(rate(auto_ai_router_key_requests_total{key="__other__"}[5m])) > 0 and raise max_keys.

info_labels values: key_alias, user_id, user_email, team_id, team_alias, organization_id. /metrics is usually unauthenticated inside the cluster, so user_email is opt-in.

Example queries:

# Requests per minute per key (top 10), with alias
topk(10, sum by (key, key_alias) (
  rate(auto_ai_router_key_requests_total[5m]) * 60
  * on (key, instance) group_left (key_alias) auto_ai_router_key_info
))

# Requests per minute by team
sum by (team_alias) (
  rate(auto_ai_router_key_requests_total[5m]) * 60
  * on (key, instance) group_left (team_alias) auto_ai_router_key_info
)

# Error share per user (4xx + 5xx, client disconnects excluded)
sum by (user_id) (
  rate(auto_ai_router_key_requests_total{status=~"[45]..",status!="499"}[5m])
  * on (key, instance) group_left (user_id) auto_ai_router_key_info
)
/
sum by (user_id) (
  rate(auto_ai_router_key_requests_total[5m])
  * on (key, instance) group_left (user_id) auto_ai_router_key_info
)

Proxy Credential Exclusion

Proxy credentials are not included in Prometheus metrics. Their statistics are available through the /health endpoint and are synchronized from the remote /health endpoint every 30 seconds.

Scrape Configuration

Example Prometheus scrape config:

scrape_configs:
  - job_name: 'auto-ai-router'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:8080']