AI Dictionary of Terms

Streaming

A response delivery method where AI outputs are transmitted token-by-token (or chunk-by-chunk) as they are generated, rather than waiting for the complete response — dramatically reducing perceived latency and creating a more natural, interactive user experience.

The Simple Version

Imagine ordering food at a restaurant. In the traditional approach (non-streaming), you wait 20 minutes for the entire meal to be prepared, then it arrives all at once.

In streaming, the waiter brings dishes as they’re ready — appetizer first, then soup, then main course. You start enjoying your meal much sooner, even though the total preparation time is the same.

Streaming works the same way with AI. Instead of waiting 10 seconds for a complete response, you see the first words appear in milliseconds, with new words flowing in continuously. The total generation time is the same, but the experience feels instant and responsive.

This is why ChatGPT, Claude, and other chat interfaces feel so responsive — they’re streaming tokens to you as they’re generated.

Detailed Explanation

Streaming leverages the autoregressive nature of language models. Since LLMs generate text one token at a time, each token can be sent to the client immediately after generation, without waiting for subsequent tokens.

Streaming Technologies:

1. Server-Sent Events (SSE):

2. WebSockets:

3. HTTP Chunked Transfer:

How Streaming Works (OpenAI Example):

Client → Server: "Tell me a story"
Server → Client: "Once" (token 1)
Server → Client: " upon" (token 2)
Server → Client: " a" (token 3)
Server → Client: " time" (token 4)
...
Server → Client: "[DONE]" (completion signal)

Streaming vs. Non-Streaming:

Aspect Non-Streaming Streaming
Time to First Token Full generation time Milliseconds
User Experience Waiting, then complete response Progressive, interactive
Network Efficiency Single large response Many small chunks
Cancellation Cannot cancel mid-generation Can stop early
Implementation Simpler More complex

Benefits of Streaming:

Challenges:

Key Characteristics

Business Context

Streaming is essential for modern AI user experiences:

Why It Matters:

Enterprise Applications:

Implementation Considerations:

Cost Implications:

Real-World Analogy

A live sports broadcast vs. a recorded replay. In a live broadcast, you see the action unfold in real-time — every play, every moment. In a replay, you wait for the entire highlight package. Streaming AI responses is like live broadcasting: you experience the content as it’s created, making it feel immediate and engaging.

Code Example

# Streaming responses with OpenAI API
from openai import OpenAI

client = OpenAI()

# Non-streaming (traditional)
print("=== Non-Streaming ===")
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku about AI"}],
    stream=False
)
print(response.choices[0].message.content)
# Waits for complete response, then prints all at once

# Streaming (progressive)
print("\n=== Streaming ===")
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku about AI"}],
    stream=True  # Enable streaming
)

# Process tokens as they arrive
for chunk in stream:
    if chunk.choices[0].delta.content:
        # Print each token immediately (no newline)
        print(chunk.choices[0].delta.content, end="", flush=True)

print()  # Final newline

# Streaming with cancellation
print("\n=== Streaming with Cancellation ===")
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a long essay about AI"}],
    stream=True,
    max_tokens=500
)

token_count = 0
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
        token_count += 1
        
        # Cancel after 20 tokens (simulating user cancellation)
        if token_count >= 20:
            print("\n[User cancelled generation]")
            break

Common Misconceptions

Sources & Further Reading