Streaming

Set stream: true to receive the answer one chunk at a time. Chat Completions streams and Responses streams both work this way.

When to use it

You show the answer as it arrives.
A request may take minutes, for example a long answer or a large file. A stream stays open while Olive works, where a request that waits for the whole answer can time out.
You want to stop a long answer early. You are billed for the tokens generated up to the moment you disconnect.

The SDK samples assume the client from the Quickstart.

curl https://api.avey.ai/llm/chat/completions \
  -H "Authorization: Bearer $AVEY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "olive",
    "messages": [{ "role": "user", "content": "Summarise the management of stable angina." }],
    "stream": true
  }'

The Go samples reuse the Quickstart client. Add the imports each sample uses.

Chat Completions chunks

Each chunk carries the new text in choices[0].delta.content. The stream ends with data: [DONE].

Reasoning arrives in delta.reasoning_content before the content. Usage arrives once, on a last chunk after the content, whether or not you set stream_options.include_usage. Its choices holds one entry with an empty delta, not an empty array as on OpenAI.

for chunk in stream:
    if chunk.usage:
        print(f"\ntokens: {chunk.usage.total_tokens}")
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    reasoning = getattr(delta, "reasoning_content", None)
    if reasoning:
        print(reasoning, end="")
    if delta.content:
        print(delta.content, end="")

Responses streams

Set stream=True on responses.create. Each event has a type. The answer text arrives in response.output_text.delta events.

curl -N https://api.avey.ai/llm/responses \
  -H "Authorization: Bearer $AVEY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "olive",
    "input": "Summarise the management of stable angina.",
    "stream": true
  }'

Event	What it carries
`response.output_text.delta`	The next piece of the answer, in `delta`
`response.completed`	The finished response. The stream ends here.
`response.incomplete`	The response, stopped early, for example at the output limit. The stream ends here.
`error`	A failure after the stream started. The stream ends here.

A Responses stream has no data: [DONE] line. It ends with one of the last three events. Read type and ignore the events you do not handle.

Heartbeats

During gaps between events, for example while a file is read or a tool runs, the server sends an SSE comment line, : heartbeat, about every second. It keeps the connection open. The SDKs and standard SSE parsers skip comments. If you parse the stream yourself, skip every line that starts with :.

Limits

Limit	Value
Context window	65,536 tokens for the input, reasoning and answer together
Output	16,384 tokens at most
Read timeout	700 seconds without a byte. Heartbeats keep a stream under it.

Errors

An error before the first event is a normal HTTP error with a JSON body. See Errors.

An error after the stream started cannot change the status code, so it arrives in the stream:

API	What you receive
Chat Completions	A last `data:` line holding an `error` object. No `data: [DONE]` follows.
Responses	An `event: error` event.

The message is generic, such as the request failed; retry later. Treat the text received so far as incomplete, and retry.

Things to know

A response that stops at the output limit has status incomplete.
Rate limits apply when the stream opens. A request over a limit returns 429 before any event is sent. See Rate limits.