AI Dictionary of Terms

Context Window

The maximum number of tokens (text units) that a language model can process in a single input-output interaction, defining the “working memory” available for the model to understand and generate text.

The Simple Version

Imagine you’re having a conversation with someone, but they can only remember the last 10 minutes of what you’ve said. If you talk for an hour, they’ll forget the beginning.

A context window is like that memory limit for AI. If a model has a 4,000-token context window, it can only “see” the most recent 4,000 tokens of your conversation or document. Anything beyond that is invisible to the model — it’s as if it doesn’t exist.

This is why you can’t paste an entire book into ChatGPT and ask it to analyze the whole thing. The book is longer than the context window, so the model can only see a portion of it.

Detailed Explanation

The context window is a fundamental architectural constraint of transformer-based language models, determined by the model’s positional encoding mechanism and attention mechanism.

How Context Windows Work:

  1. Tokenization: Input text is converted into tokens (roughly 0.75 words per token in English)
  2. Context Limit: The model can only process up to its maximum context window size
  3. Sliding Window: As new tokens are added, older tokens “fall off” the beginning
  4. Shared Space: The context window includes both input (prompt + retrieved context) AND output (model’s response)

Context Window Sizes (2026):

Why Context Windows Matter:

Technical Constraints:

Key Characteristics

Business Context

Context windows directly impact enterprise AI architecture and costs:

Strategic Implications:

Practical Considerations:

Cost Example:

Real-World Analogy

A desk workspace. A small desk (4K context) can only hold a few papers at once — you need to constantly shuffle things around. A large conference table (1M context) can hold hundreds of documents, letting you see everything at once. The larger workspace is more powerful but also more expensive and harder to manage.

Code Example

# Checking context window and token usage
import tiktoken

# Load tokenizer for GPT-4
encoding = tiktoken.encoding_for_model("gpt-4")

# Example text
text = """
The history of artificial intelligence began in antiquity, with myths, stories, 
and rumors of artificial beings endowed with intelligence or consciousness 
by master craftsmen. The study of logic and formal reasoning from antiquity 
to the present led directly to the invention of the programmable digital 
computer in the 1940s...
""" * 100  # Repeat to make it longer

# Count tokens
tokens = encoding.encode(text)
print(f"Text length: {len(text)} characters")
print(f"Token count: {len(tokens)} tokens")
print(f"Approximate words: {len(tokens) * 0.75:.0f} words")

# Check if it fits in context window
gpt4_context = 128000  # GPT-4o context window
if len(tokens) > gpt4_context:
    print(f"WARNING: Text exceeds context window by {len(tokens) - gpt4_context} tokens")
else:
    print(f"Fits in context window with {gpt4_context - len(tokens)} tokens to spare")

Common Misconceptions

Sources & Further Reading