Streaming

Pass stream: true. Frames are chat.completion.chunk, terminated by data: [DONE].

stream = client.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

const stream = await client.chat.completions.create({
  model: "llama-3.1-8b",
  messages: [{ role: "user", content: "Count to five." }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

The stream opens with a role frame, then content frames, then a frame carrying finish_reason:

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

Token counts in the stream

Ask for them with stream_options:

stream = client.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
    stream_options={"include_usage": True},
)

You then get one extra frame, after the final choice frame, with an empty choices array:

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":9,"total_tokens":23}}

If your loop reads chunk.choices[0] unconditionally it will find nothing on that frame. That is OpenAI’s shape, and the SDKs handle it; hand-rolled parsers usually need a guard.

A stream that failed part way sends no usage frame. A usage block on a broken stream would read as a completed request.

Tool calls

They stream. delta.tool_calls accumulates by index: the first frame carries the id, type and function name, and later frames append a few characters of function.arguments at a time. Reassembling them is the client’s job, and every SDK already does it. See Tool calling.

When a stream fails part way

Once the response headers are sent there is no status code left to fail with, so a mid-stream failure closes the stream with finish_reason: "error" rather than "stop".

Check it. A parser that reads only content cannot tell a truncated answer from a complete one, and that is the whole reason this exists.

Disconnecting

Billing is otherwise identical whether you stream or not. Same reservation, same settle, same ledger entry, same usage record. Send the same prompt at temperature: 0 with and without stream and the two cost the same.

Not available for every model

A small number of models are served through a path where streaming is not implemented. They are filtered out of anywhere that offers streaming, and asking for it explicitly is a clean error rather than a reservation followed by a failure.

Idempotency and streaming do not combine

Idempotency-Key with stream: true is a 400. Replaying a stream from cache would mean buffering every response on the chance of a retry. Refusing is honest; accepting the header and ignoring it would let you believe you were protected from a double charge when you were not. See Idempotency.