Streaming

Set stream: true on any chat completion request and Onlist returns the response as Server-Sent Events (SSE) instead of a single JSON body. The wire format is identical to OpenAI and OpenRouter, so your existing streaming client works unchanged after you point it at https://onlist.io.

The SSE wire format

When stream: true, the response Content-Type is text/event-stream. The body is a sequence of data: lines, each carrying one ChatCompletionChunk JSON object. The stream ends with a final usage-only chunk followed by the literal sentinel data: [DONE]:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":11,"completion_tokens":2,"total_tokens":13}}

data: [DONE]

Two details matter:

• The final usage chunk is the last data: frame before [DONE]. It carries choices: [] (empty) and a populated usage object. Content chunks before it do not include usage. A stream can also end without one; see The terminal usage chunk.
• data: [DONE] is a plain string sentinel, not JSON. Stop reading when you see it; do not try to parse it.

The official OpenAI SDKs handle both of these for you. If you parse the stream by hand, guard against the [DONE] sentinel and accumulate choices[].delta.content across chunks.

With web search on, one more chunk arrives between the last content chunk and the usage chunk. It carries the sources in choices[0].delta.annotations and no content. If the provider's stream has no usage chunk, the annotations chunk is the last data frame.

With the openrouter:web_search server tool, the model may be called several times before it answers, with a search between calls. You still get one stream in the shape above: the text and reasoning of every call arrive as they are generated, while the search calls and the finish_reason, usage chunk and data: [DONE] of the calls that searched are not sent. If the text so far does not end with a line break, \n\n comes before the next call's text. The one usage chunk at the end carries the sum over all calls.

Examples

The request body is a standard chat completion plus stream: true. The optional provider object works the same way as on non-streaming requests.

from openai import OpenAI

client = OpenAI(
    base_url="https://onlist.io/v1",
    api_key="YOUR_ONLIST_TOKEN",
)

stream = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Stream a haiku about routing."}],
    stream=True,
)

for chunk in stream:
    # The terminal usage chunk has an empty choices list.
    if chunk.choices:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)
    elif chunk.usage:
        print(f"\n[tokens: {chunk.usage.total_tokens}]")

The -N flag disables curl's output buffering so you see chunks as they arrive.

The terminal usage chunk

The usage chunk comes from the provider. Most providers send it once include_usage is set, but a provider whose upstream ignores the option sends none, and then the stream has no usage chunk: the last content chunk (or the web search annotations chunk) is followed directly by data: [DONE]. Write your client so that it also works without one.

Onlist rewrites only three things in your request in place: the model, the provider pin, and stream_options.include_usage. It otherwise forwards the request to the upstream provider byte-faithfully, relaying the chunks back unchanged. The one exception is web search. With the web plugin or :online, Onlist also adds the results to your last user message, removes plugins, and adds the annotations chunk and the search count in usage. With the openrouter:web_search server tool, it replaces the tool with a search function, removes max_tool_calls and stop_server_tools_when, and joins the model's calls into one stream as described above. The usage object in the terminal chunk carries prompt_tokens, completion_tokens, and total_tokens. Note that usage.cost is not included anywhere in the stream: cost is reported out of band. See Usage & cost for how to retrieve it.

Heartbeats while waiting for the first token

Reasoning models can think for a long time before emitting their first token. If the connection stays completely silent during that window, edge proxies (Onlist's edge closes silent streaming connections after ~100 seconds) or your own HTTP client may kill the request before any output arrives.

To keep the connection alive, Onlist commits the response early on slow requests: after roughly 15 seconds without a first token it sends the 200 status line, the SSE headers, and then an SSE comment frame every ~15 seconds until real data starts:

: ONLIST PROCESSING

: ONLIST PROCESSING

data: {"id":"chatcmpl-...","object":"chat.completion.chunk", ...}

With the web search plugin or :online, this wait starts only after the search. The search can take up to about 30 seconds and sends nothing at all, not even the headers, so the first byte can arrive about 45 seconds after the request. With the openrouter:web_search server tool, searches run between model calls; when the stream has been quiet for about 15 seconds during one, Onlist sends a comment line to keep the connection open.

Lines starting with : are comments in the SSE specification — compliant parsers drop them at the protocol layer, so they never reach your application code. The official OpenAI and Anthropic SDKs (and any client that works with OpenRouter, which uses the same mechanism) handle them transparently. If you parse the stream by hand, skip any line that begins with :.

Two consequences to be aware of:

• Errors after heartbeats begin arrive in-stream over HTTP 200. Once the status line has been sent it cannot be changed, so if every upstream attempt fails after heartbeats started, the terminal error is delivered as a final SSE frame instead of an HTTP error status. For OpenAI-format endpoints the frame is a data: payload with a top-level error key and no [DONE] sentinel (SDKs raise it as an API error):

  data: {"error":{"message":"...","type":"upstream_error","code":502}}
  

  For the Claude-native endpoint it is a standard Anthropic error event and no message_stop:

  event: error
  data: {"type":"error","error":{"type":"api_error","message":"..."}}
  

• Non-streaming requests cannot be heartbeated. A single JSON body has no place to put padding bytes, so a non-streaming call to a slow reasoning model can still be cut off by the ~100-second edge idle limit. For requests that may think longer than that, use stream: true.

Route id is written before the first byte

X-Onlist-Route-Id identifies the routing decision for the request. On a streaming response it is flushed with the first byte the platform sends — which on slow requests may be a heartbeat comment rather than model output — so you can read it as soon as bytes start arriving; you do not have to wait for the stream to finish.

This is the same opaque UUID returned on non-streaming responses, and it is your key for correlating a stream to its settled cost and metadata after the fact. It is not the body id field (that is the upstream completion id).

For the full header reference and rate-limit behavior, see the pages below.

Next steps

• Usage & cost: Read token counts, and find a stream's out-of-band cost in the usage log.
• Limits & headers: X-Onlist-Route-Id, X-Onlist-Warnings, and Retry-After behavior in full.