Streaming

Server-sent events in the OpenAI chunk format, including how a mid-stream failure is reported.

Set stream: true and the response is text/event-stream, one chat.completion.chunk per frame, terminated by data: [DONE]. This is the OpenAI format exactly, so the SDK stream iterators work unchanged.

curl needs -N (--no-buffer). Without it curl buffers the body and you will conclude Atlas is not streaming when it is.

The frame sequence

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1757376000,"model":"atlas-mid-1","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk",...,"choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk",...,"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk",...,"choices":[],"usage":{"prompt_tokens":14,"completion_tokens":9,"total_tokens":23}}

data: [DONE]

A role frame opening the assistant turn.
Content deltas, one or more.
A finish frame carrying finish_reason and an empty delta.
A usage frame with an empty choices array.
data: [DONE].

Usage on a stream

The usage frame arrives whether or not you asked for it. Atlas always sends stream_options: {"include_usage": true} upstream, because metering must not depend on what a caller happened to request, and the resulting frame is passed through to you unchanged.

So your delta-handling code must tolerate a chunk whose choices is []. Indexing choices[0] unconditionally is the most common way a working OpenAI integration breaks on its own last frame — and here it breaks even if you never set include_usage.

Setting stream_options: {"include_usage": true} explicitly is still worth doing: it documents the intent, and it is what your code would need against a provider that honors the flag literally.

The usage frame is the last frame before [DONE]:

{"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":9,"total_tokens":23}}

Where a model emits a reasoning prelude, those tokens are counted inside completion_tokens. Never add completion_tokens_details.reasoning_tokens to completion_tokens; you would double-count.

Keep-alive comments

If a generation is silent for 15 seconds, Atlas writes an SSE comment line to keep the connection and any intermediary alive:

: atlas-heartbeat

Comments are not events: the SDK stream parsers skip them, and they are never billed. If you are parsing the wire yourself, ignore any line beginning with :.

When a stream fails mid-generation

This is the part worth reading before you ship.

Once frames have been sent, the HTTP status is already 200 and cannot be changed. So a failure after that point is delivered in band: a final data: frame carrying an error envelope, and then no [DONE].

data: {"id":"chatcmpl-...","choices":[{"index":0,"delta":{"content":"One, two"},"finish_reason":null}]}

data: {"error":{"message":"...","type":"server_error","param":null,"code":"backend_error","request_id":"req_01k4v9m2..."}}

The frame carries no event: field, because that is what the OpenAI API itself sends — the OpenAI SDK's stream iterator checks each decoded frame for an error key and raises APIError from it, so a try/except around your loop already fires with no Atlas-specific code.

The missing [DONE] is deliberate. A client that ignores the error frame and keys only off the sentinel would record a truncated generation as a completed one — a bug that surfaces as an inexplicably short answer rather than as a failure. Treat "stream ended without [DONE]" as an error, whatever else you do.

Deploys

When the gateway is asked to shut down, every in-flight stream receives a 503 error frame (code: "server_shutting_down") inside the grace period, by the same in-band path. Your existing retry-on-503 logic handles a rollout as a retryable blip.

Behind a proxy

If your traffic passes through a proxy you control, and streamed tokens arrive in one batch at the end, the proxy is buffering. Atlas sets the headers that turn this off — x-accel-buffering: no, and cache-control: no-cache, no-store, no-transform — but an intermediary configured to ignore them will still buffer. There is no content-length on a streamed response; the body is chunked.