AI Dictionary of Terms

Latency

The time delay between when a request is sent to an AI system and when the first response (or complete response) is received — a critical performance metric that directly impacts user experience and system responsiveness.

The Simple Version

Imagine calling a friend and asking them a question. Latency is how long it takes for them to start answering.

For AI systems, latency is the time between you hitting “send” and seeing the AI’s response appear. In conversational AI, high latency makes the system feel slow and unresponsive. In batch processing, latency matters less since you’re not waiting interactively.

Detailed Explanation

Latency in AI systems is measured at multiple points and has significant implications for architecture and user experience.

Types of Latency:

1. Time to First Token (TTFT):

2. Time Between Tokens (Inter-Token Latency):

3. Total Latency:

Factors Affecting Latency:

Latency Benchmarks (2026):

Latency vs. Quality Trade-offs:

Key Characteristics

Business Context

Latency is a critical factor in enterprise AI deployment and user experience:

Why Latency Matters:

Latency Requirements by Use Case:

Optimization Strategies:

Monitoring and Alerting:

Real-World Analogy

Ordering food at a restaurant.

Each has its place. You wouldn’t want fine dining latency when you’re hungry and in a hurry, but you’d accept it for a celebration dinner. Similarly, AI latency requirements depend on the context.

Code Example

# Measuring latency for AI API calls
import time
from openai import OpenAI

client = OpenAI()

def measure_latency(prompt: str, model: str = "gpt-4o"):
    """Measure time to first token and total latency."""
    
    start_time = time.time()
    
    # Streaming response to measure TTFT
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True
    )
    
    first_token_time = None
    token_count = 0
    
    for chunk in response:
        if chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.time()
            token_count += 1
    
    end_time = time.time()
    
    ttft = first_token_time - start_time
    total_time = end_time - start_time
    tokens_per_second = token_count / (total_time - ttft) if total_time > ttft else 0
    
    print(f"Model: {model}")
    print(f"Time to First Token: {ttft*1000:.0f}ms")
    print(f"Total Latency: {total_time*1000:.0f}ms")
    print(f"Tokens Generated: {token_count}")
    print(f"Generation Speed: {tokens_per_second:.1f} tokens/sec")
    print("-" * 40)

# Test with different prompts
measure_latency("What is 2+2?")
measure_latency("Write a detailed essay about the history of artificial intelligence.")

Common Misconceptions

Sources & Further Reading