Streaming Responses Guide

This guide covers best practices for handling streaming responses across different platforms and languages.

Why stream?

Better UX: Users see tokens appear in real time instead of waiting 2-5 seconds for a full response
Lower perceived latency: Users see the first token as soon as the model starts generating (latency varies by model and region)
Memory efficient: Process tokens as they arrive instead of buffering the full response

Python (OpenAI SDK)

stream = client.chat.completions.create(
    model="gpt-oss-20b",
    messages=[{"role": "user", "content": "Explain transformers"}],
    stream=True
)
 
full_response = ""
for chunk in stream:
    content = chunk.choices[0].delta.content or ""
    full_response += content
    print(content, end="", flush=True)

Python (async)

import asyncio
 
async def stream_response():
    stream = await async_client.chat.completions.create(
        model="gpt-oss-20b",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )
 
    async for chunk in stream:
        content = chunk.choices[0].delta.content or ""
        print(content, end="", flush=True)
 
asyncio.run(stream_response())

Node.js

const stream = await client.chat.completions.create({
  model: 'gpt-oss-20b',
  messages: [{ role: 'user', content: 'Hello' }],
  stream: true,
});
 
for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content || '';
  process.stdout.write(content);
}

React (Next.js)

'use client'
import { useState } from 'react'
 
export function Chat() {
  const [output, setOutput] = useState('')
  const [loading, setLoading] = useState(false)
 
  async function handleSubmit(prompt: string) {
    setLoading(true)
    setOutput('')
 
    const response = await fetch('/api/chat', {
      method: 'POST',
      body: JSON.stringify({ prompt }),
    })
 
    const reader = response.body!.getReader()
    const decoder = new TextDecoder()
 
    while (true) {
      const { done, value } = await reader.read()
      if (done) break
 
      const text = decoder.decode(value)
      setOutput(prev => prev + text)
    }
 
    setLoading(false)
  }
 
  return <div>{output}{loading && <span className="animate-pulse">▊</span>}</div>
}

A streaming request can fail in two places. Before the stream starts it fails with a normal HTTP error status, so the create call raises. Once the stream has started, the error arrives inside the stream, or the connection drops, and it is raised while you iterate. Handle both:

from openai import APIConnectionError, APIStatusError
 
try:
    stream = client.chat.completions.create(
        model="gpt-oss-20b",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True,
    )
except APIStatusError as e:
    # Failed before the stream started: e.status_code is the HTTP status
    print(f"Request failed: {e.status_code} {e.message}")
except APIConnectionError as e:
    print(f"Could not reach the API: {e}")
else:
    try:
        for chunk in stream:
            content = chunk.choices[0].delta.content or ""
            process(content)
    except Exception as e:
        # An error event inside the stream, or a dropped connection
        print(f"Stream interrupted: {e}")
        # Optionally retry or fall back to non-streaming

Best practices

Always flush output — use flush=True in Python or process.stdout.write in Node
Show a cursor — display a blinking cursor while streaming for better UX
Handle [DONE] — the stream ends with data: [DONE]; your parser must handle this
Set timeouts — if no token arrives in 30 seconds, the connection may be stale
Buffer by word — for display, buffer until a space character for smoother text rendering