Responses API

Auto AI Router implements the OpenAI Responses API and routes requests natively to Anthropic, Comet API, Vertex AI, and AWS Bedrock — without converting through Chat Completions format as an intermediary.

Endpoints

GPT-6 Astra

Configure gpt-6-astra on an existing OpenAI-compatible credential. OpenAI credentials forward Responses requests directly to /v1/responses. For example, with an existing credential named openai_main:

models:
  - name: gpt-6-astra
    credential: openai_main

For GPT-6 requests, the router removes temperature, top_p, and top_logprobs. Responses requests also drop message.output_text.logprobs from include, preserving other include values. Chat Completions requests drop logprobs and use the existing max_tokens to max_completion_tokens conversion. max_output_tokens is preserved. See the OpenAI migration guide.

Use Responses for tool calling. Native HTTP forwarding preserves async tool flags, additional_tools with required or named tool_choice, configuration_update input items, prompt_cache_breakpoint, and prompt_cache_options. Reasoning values such as max and pro are forwarded for the upstream to interpret; the router does not add provider-specific reasoning aliases. Cache options do not guarantee a cache hit.

Prices still come from server.model_prices_link or the LiteLLM database. Add the model's applicable rates there before using budget enforcement. Input, output, cache read/write, and reasoning usage use the existing billing pipeline.

Request Parameters

All standard Responses API parameters are supported. The table below lists the full set recognized by the router:

Not all providers support every parameter. See the Provider Support table below.

service_tier is ignored and removed by Auto AI Router. Clients cannot select an upstream service tier. The effective tier is controlled by the router/provider configuration.

Content Types

The input array accepts items of different types. Supported ContentPart types within messages:

Anthropic-backed input_file

For Anthropic, Comet API, and ProMan routes that use the Anthropic-compatible native Responses converter:

AIR does not implement a Files API resolver or OpenAI/LiteLLM file ID to Anthropic file ID mapping. file_id values are provider/credential scoped and are not forwarded to Anthropic-backed routes.

Malformed PDF data URIs, non-base64 encodings, empty document sources, and unsupported document source types are rejected as client request validation errors (400 Bad Request) rather than internal conversion failures.

Input items can also be function call / function call output items for multi-turn tool use:

{"type": "function_call", "call_id": "call_abc", "name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}
{"type": "function_call_output", "call_id": "call_abc", "output": "{\"temp\":22}"}

Multi-Turn Conversations

Storing Responses

Set "store": true to persist a response. A stored response can be retrieved later:

curl http://localhost:8080/v1/responses/resp_01abc... \
  -H "Authorization: Bearer sk-your-key"

Continuing a Conversation

Pass previous_response_id to continue from a prior response. The router reconstructs the previous output as input context before sending to the provider:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-your-key")

first = client.responses.create(
    model="claude-sonnet-4-20250514",
    input="What is the capital of France?",
    store=True,
)

second = client.responses.create(
    model="claude-sonnet-4-20250514",
    input="And what language do they speak there?",
    previous_response_id=first.id,
    store=True,
)

Streaming

Add "stream": true to receive Server-Sent Events. The event sequence follows the Responses API specification:

response.created
response.in_progress
  response.output_item.added
  response.content_part.added
  response.output_text.delta  (repeated)
  response.output_text.done
  response.content_part.done
  response.output_item.done
response.completed
[DONE]

stream = client.responses.create(
    model="gemini-2.5-flash",
    input="Tell me about Paris",
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

WebSocket Protocol

The router accepts WebSocket connections on GET /v1/responses (with Upgrade: websocket header). By default, each turn uses the existing HTTP/SSE provider path. Enable native upstream WebSockets for models whose providers support them:

models:
  - name: gpt-6-astra
    credential: openai_main
    websocket_responses: true

Native Upstream Mode

Native mode connects directly to the configured provider's WebSocket Responses endpoint. It supports OpenAI and existing proxy/air credentials with native Responses passthrough. All credentials serving an opted-in model must support WebSockets. For a custom deployment name, also set passthrough_responses: true if native Responses is not auto-detected.

Send response.create to begin. Once response.created arrives, GPT-6 Astra accepts response.steer during generation:

{"type":"response.steer","previous_response_id":"resp_1","input":"Keep the answer shorter."}

Steering accepts user input, and the upstream creates its successor response automatically. The router tracks and bills each response separately. If response.steer.pending requests a tool result, send another response.create with the original previous_response_id and the required function_call_output. Async tool flags, cache options, and configuration_update items in response.create.input follow the HTTP passthrough rules. Support for these features still depends on the provider; Azure WebSocket transport does not imply Azure supports steering. See OpenAI steering.

Native sessions use one model and credential, one active response, and at most one queued steer. Authentication, model access, credential scope, rate limits, and budget checks run for each create/steer request. Provider usage, including cache usage, feeds the existing billing pipeline; continuation reservations include an estimate of prior context.

stream and background are removed and store is forced to false. Responses remain on the upstream connection: router HTTP retrieval and cross-connection continuation are unavailable. Reconnect with full input history after an upstream disconnect, credential change, one hour, or 128 admitted responses. There is no automatic provider fallback inside an established session. By default, disconnecting closes the upstream and uses estimated usage for unfinished responses. With drain_upstream_on_abort: true, the router keeps reading for the existing drain grace period to collect final provider usage.

The remaining examples also apply to the default HTTP/SSE bridge; its local and persistent response-store behavior is described separately below.

Connection

const ws = new WebSocket("ws://localhost:8080/v1/responses", {
  headers: { "Authorization": "Bearer sk-your-key" }
});

Sending a Request

Send a JSON message with "type": "response.create" and any standard Responses API fields:

{
  "type": "response.create",
  "model": "claude-sonnet-4-20250514",
  "input": "Hello! What is 2+2?",
  "stream": true
}

In the default bridge, type is stripped before forwarding HTTP. Native mode sends response.create over the upstream WebSocket.

Receiving Events

The server sends each SSE event as a plain JSON text message (no data: prefix, no [DONE]). Turn completion is signaled by a terminal event (response.completed, response.failed, response.incomplete, error).

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  if (data.type === "response.output_text.delta") {
    process.stdout.write(data.delta);
  } else if (data.type === "response.completed") {
    console.log("\nDone");
  } else if (data.type === "error") {
    console.error(data.error.message);
  }
};

Error Events

HTTP errors are converted to structured WebSocket error events:

{
  "type": "error",
  "sequence_number": 0,
  "error": {
    "code": "api_error",
    "message": "Rate limit exceeded",
    "type": "server_error",
    "param": null
  }
}

Connection-Local Cache (HTTP/SSE Bridge)

When store: false is explicitly set, completed responses are cached in connection-local memory for the duration of the WebSocket connection. This allows previous_response_id continuations within the same session without a persistent store. The cache is cleared on reconnect.

When store is absent or true, the persistent response store handles continuations across reconnects.

Multi-Turn Example

// First turn
ws.send(JSON.stringify({
  type: "response.create",
  model: "claude-sonnet-4-20250514",
  input: "What is the capital of France?",
  store: false,
}));

// Wait for response.completed, capture response ID, then:
ws.send(JSON.stringify({
  type: "response.create",
  model: "claude-sonnet-4-20250514",
  input: "What language do they speak there?",
  previous_response_id: "<id from first turn>",
  store: false,
}));

Compact API

POST /v1/responses/compact summarizes a conversation into a single compaction item. This is useful for reducing context size while preserving essential information.

Request

curl -X POST http://localhost:8080/v1/responses/compact \
  -H "Authorization: Bearer sk-your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "input": [
      {"role": "user", "content": "What is photosynthesis?"},
      {"role": "assistant", "content": "Photosynthesis is the process by which plants..."}
    ]
  }'

Requirements:

• model is required
• Request body limit: 10 MB

Response

{
  "id": "resp_01abc...",
  "object": "response.compaction",
  "created_at": 1234567890,
  "output": [
    {
      "type": "compaction",
      "id": "compact_01xyz...",
      "encrypted_content": "<summary of the conversation>"
    }
  ],
  "usage": {
    "input_tokens": 120,
    "output_tokens": 45,
    "total_tokens": 165
  }
}

The encrypted_content field contains the model's summary. Use this item in input for subsequent requests to continue the conversation from the compacted context.

Native vs Passthrough Mode

The router uses two modes for Responses API requests:

Native mode is used automatically for Anthropic, Comet API, Vertex AI, and AWS Bedrock. Passthrough is used for OpenAI and other providers that already speak Responses API natively.

The mode can be overridden via model configuration:

models:
  - name: "my-model"
    passthrough_responses: true  # force passthrough

Provider Support

Retry and Fallback

When a provider credential returns a rate-limit error (429), the router automatically tries the next available credential of the same type. The original HTTP error code is preserved in the final response — the client receives 429 (not 502) when all credentials of the appropriate type are exhausted.

When no credentials are available at all, the router returns 503 Service Unavailable.