The number of requests, tokens, or operations an AI system can process per unit of time (e.g., requests per second, tokens per second) — a critical metric for understanding system capacity, scalability, and cost-efficiency at scale.
Imagine a highway.
You can have a fast highway (low latency) with only one lane (low throughput), or a slower highway with 10 lanes (high throughput). For AI systems, you need to optimize both depending on your use case.
Throughput answers the question: “How much work can this system handle?” If you need to process 10,000 customer queries per hour, you need a system with sufficient throughput.
Throughput is a fundamental capacity metric that determines how much work an AI system can handle in a given time period.
Throughput Metrics:
1. Requests Per Second (RPS):
2. Tokens Per Second (TPS):
3. Queries Per Second (QPS):
Factors Affecting Throughput:
Throughput vs. Latency Trade-offs:
| Strategy | Latency | Throughput | Use Case |
|---|---|---|---|
| Single request, no batching | Low | Low | Real-time chat |
| Dynamic batching | Medium | High | API serving |
| Large batch processing | High | Very High | Offline analysis |
Throughput Optimization Techniques:
1. Batching: Group multiple requests and process them together on the GPU.
2. Model Parallelism:
3. Quantization: Reduce model precision (FP16 → INT8 → INT4) to process more tokens per second.
4. Speculative Decoding: Use a small model to draft tokens, verify with large model in parallel.
5. Caching: Cache frequent queries to avoid reprocessing (dramatically increases effective throughput).
Throughput is critical for enterprise AI planning and cost management:
Why Throughput Matters:
Throughput Requirements by Use Case:
Cost Implications:
Scaling Strategies:
Monitoring and Capacity Planning:
A restaurant kitchen.
Each approach has trade-offs. A fine dining restaurant prioritizes quality (low throughput, high attention). A fast-food chain prioritizes speed and volume (high throughput, standardized). Your AI system should match the throughput to your needs.
# Measuring throughput with concurrent requests
import asyncio
import time
from openai import AsyncOpenAI
client = AsyncOpenAI()
async def make_request(prompt: str):
"""Make a single API request."""
response = await client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
max_tokens=50
)
return response.choices[0].message.content
async def measure_throughput(num_requests: int, concurrency: int):
"""Measure throughput with concurrent requests."""
prompts = [f"Request {i}: What is {i} + {i}?" for i in range(num_requests)]
start_time = time.time()
# Process requests with limited concurrency
semaphore = asyncio.Semaphore(concurrency)
async def limited_request(prompt):
async with semaphore:
return await make_request(prompt)
tasks = [limited_request(prompt) for prompt in prompts]
results = await asyncio.gather(*tasks)
end_time = time.time()
total_time = end_time - start_time
rps = num_requests / total_time
print(f"Requests: {num_requests}")
print(f"Concurrency: {concurrency}")
print(f"Total Time: {total_time:.2f}s")
print(f"Throughput: {rps:.2f} requests/second")
print("-" * 40)
# Test with different concurrency levels
async def main():
await measure_throughput(100, concurrency=1) # Sequential
await measure_throughput(100, concurrency=10) # 10 concurrent
await measure_throughput(100, concurrency=50) # 50 concurrent
asyncio.run(main())
# Expected: Higher concurrency = higher throughput (up to API rate limits)
Reality: Throughput and latency are often trade-offs. Maximizing throughput (large batches) increases latency. You need to balance based on your use case.
Reality: Throughput scales sub-linearly due to communication overhead, memory bottlenecks, and diminishing returns. Doubling GPUs doesn’t always double throughput.
Reality: Even small systems need to understand throughput to plan capacity, optimize costs, and ensure they can handle growth.