Streaming

Set stream: true on any chat completion request and Onlist returns the response as Server-Sent Events (SSE) instead of a single JSON body. The wire format is identical to OpenAI and OpenRouter, so your existing streaming client works unchanged after you point it at https://onlist.io.

The SSE wire format

When stream: true, the response Content-Type is text/event-stream. The body is a sequence of data: lines, each carrying one ChatCompletionChunk JSON object. The stream ends with a final usage-only chunk followed by the literal sentinel data: [DONE]:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":11,"completion_tokens":2,"total_tokens":13}}

data: [DONE]

Two details matter:

• The final usage chunk is the last data: frame before [DONE]. It carries choices: [] (empty) and a populated usage object. Content chunks before it do not include usage.
• data: [DONE] is a plain string sentinel, not JSON. Stop reading when you see it; do not try to parse it.

The official OpenAI SDKs handle both of these for you. If you parse the stream by hand, guard against the [DONE] sentinel and accumulate choices[].delta.content across chunks.

Examples

The request body is a standard chat completion plus stream: true. The optional provider object works the same way as on non-streaming requests.

from openai import OpenAI

client = OpenAI(
    base_url="https://onlist.io/v1",
    api_key="YOUR_ONLIST_TOKEN",
)

stream = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Stream a haiku about routing."}],
    stream=True,
)

for chunk in stream:
    # The terminal usage chunk has an empty choices list.
    if chunk.choices:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)
    elif chunk.usage:
        print(f"\n[tokens: {chunk.usage.total_tokens}]")

The -N flag disables curl's output buffering so you see chunks as they arrive.

The terminal usage chunk is always emitted

Onlist rewrites only three things in your request in place: the model, the provider pin, and stream_options.include_usage. It otherwise forwards the request to the upstream provider byte-faithfully, relaying the chunks back unchanged. The usage object in the terminal chunk carries prompt_tokens, completion_tokens, and total_tokens. Note that usage.cost is not included anywhere in the stream: cost is reported out of band. See Usage & cost for how to retrieve it.

Heartbeats while waiting for the first token

Reasoning models can think for a long time before emitting their first token. If the connection stays completely silent during that window, edge proxies (Onlist's edge closes silent streaming connections after ~100 seconds) or your own HTTP client may kill the request before any output arrives.

To keep the connection alive, Onlist commits the response early on slow requests: after roughly 15 seconds without a first token it sends the 200 status line, the SSE headers, and then an SSE comment frame every ~15 seconds until real data starts:

: ONLIST PROCESSING

: ONLIST PROCESSING

data: {"id":"chatcmpl-...","object":"chat.completion.chunk", ...}

Lines starting with : are comments in the SSE specification — compliant parsers drop them at the protocol layer, so they never reach your application code. The official OpenAI and Anthropic SDKs (and any client that works with OpenRouter, which uses the same mechanism) handle them transparently. If you parse the stream by hand, skip any line that begins with :.

Two consequences to be aware of:

• Errors after heartbeats begin arrive in-stream over HTTP 200. Once the status line has been sent it cannot be changed, so if every upstream attempt fails after heartbeats started, the terminal error is delivered as a final SSE frame instead of an HTTP error status. For OpenAI-format endpoints the frame is a data: payload with a top-level error key and no [DONE] sentinel (SDKs raise it as an API error):

  data: {"error":{"message":"...","type":"upstream_error","code":502}}
  

  For the Claude-native endpoint it is a standard Anthropic error event and no message_stop:

  event: error
  data: {"type":"error","error":{"type":"api_error","message":"..."}}
  

• Non-streaming requests cannot be heartbeated. A single JSON body has no place to put padding bytes, so a non-streaming call to a slow reasoning model can still be cut off by the ~100-second edge idle limit. For requests that may think longer than that, use stream: true.

Route id is written before the first byte

X-Onlist-Route-Id identifies the routing decision for the request. On a streaming response it is flushed with the first byte the platform sends — which on slow requests may be a heartbeat comment rather than model output — so you can read it as soon as bytes start arriving; you do not have to wait for the stream to finish.

This is the same opaque UUID returned on non-streaming responses, and it is your key for correlating a stream to its settled cost and metadata after the fact. It is not the body id field (that is the upstream completion id).

For the full header reference and rate-limit behavior, see the pages below.

Next steps

• Usage & cost: Read token counts and correlate a stream to its out-of-band cost via the route id.
• Limits & headers: X-Onlist-Route-Id, X-Onlist-Warnings, and Retry-After behavior in full.