Streaming

Turn it on#

Add "stream": true to the request body and the response becomes text/event-stream (SSE). The official SDKs wrap this for you:

from openai import OpenAI

client = OpenAI(api_key="sk-...", base_url="https://api.flyrouter.ai/v1")

stream = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=[{"role": "user", "content": "Write a short poem about autumn"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Event formats by protocol#

OpenAI Chat Completions: each chunk is a data: {...} line, the text delta is in choices[0].delta.content, and the stream ends with data: [DONE]. To get token usage, add "stream_options": {"include_usage": true} and the last chunk carries usage.

Anthropic Messages: events arrive as message_start → content_block_start → several content_block_delta → content_block_stop → message_delta → message_stop. Text deltas are in delta.text of content_block_delta, and usage is in message_delta.usage.

from anthropic import Anthropic

client = Anthropic(api_key="sk-...", base_url="https://api.flyrouter.ai")

with client.messages.stream(
    model="claude-sonnet-5-5",
    max_tokens=4096,
    messages=[{"role": "user", "content": "Write a short poem about autumn"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()
print("\n", final.usage)

Gemini: call streamGenerateContent with ?alt=sse. See the Gemini native API.

Things to know#

• Large models can go seconds to tens of seconds without emitting text while they think. That's normal, so don't disconnect early. Set your client's read timeout to 10 minutes or more.
• Always stream when max_tokens is large (tens of thousands or more). Non-streaming requests that big tend to time out while waiting.
• If you disconnect mid-stream, the part already generated is still billed.