AI Dictionary of Terms

Speculative Decoding

An inference optimization technique that uses a small, fast “draft” model to generate multiple candidate tokens, which are then verified in parallel by a larger, more accurate “target” model — achieving the quality of the large model with the speed approaching the small model.

The Simple Version

Imagine you’re writing a document with a very fast but occasionally inaccurate assistant, and a very accurate but slow editor.

Without speculative decoding: You wait for the slow editor to write each word. It’s accurate but takes forever.

With speculative decoding: The fast assistant quickly drafts 5-10 words. The slow editor reviews all of them at once (in parallel), accepting the correct ones and fixing any mistakes. You get the editor’s accuracy with the assistant’s speed.

Speculative decoding does the same for AI. A small, fast model (draft model) generates several tokens quickly. The large, accurate model (target model) verifies them all at once. If the draft was right, you’ve generated multiple tokens in the time it takes to generate one. If the draft was wrong, the target model corrects it.

This technique can achieve 2-3x speedup while maintaining the exact same output quality as the large model.

Detailed Explanation

Speculative decoding exploits the fact that verifying N tokens takes roughly the same time as generating 1 token in autoregressive decoding. By having a draft model propose multiple tokens, the target model can verify them in parallel.

The Process:

Step 1: Draft Phase

Step 2: Verification Phase

Step 3: Acceptance/Rejection

Mathematical Insight:

Draft Model Selection:

Key Factors for Success:

1. Draft Quality:

2. Draft Length (K):

3. Hardware Utilization:

Variants:

1. Standard Speculative Decoding:

2. Medusa:

3. SpecInfer:

4. EAGLE:

Key Characteristics

Business Context

Speculative decoding is critical for reducing inference costs at scale:

Why It Matters:

Enterprise Applications:

Cost Example:

Implementation Considerations:

Popular Implementations:

Real-World Analogy

A chef and sous-chef preparing a meal. The sous-chef (draft model) quickly preps ingredients based on the recipe. The head chef (target model) reviews everything at once, accepting correct prep and fixing mistakes. The meal is prepared much faster because the head chef doesn’t have to do everything from scratch — they just verify and correct.

Code Example

# Speculative decoding with Hugging Face Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load target model (large, accurate)
target_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load draft model (small, fast)
draft_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-70b-hf")

# Speculative decoding function
def speculative_decode(
    prompt: str,
    target_model,
    draft_model,
    tokenizer,
    max_new_tokens: int = 100,
    K: int = 5  # Number of tokens to draft
):
    """Generate text using speculative decoding."""
    
    inputs = tokenizer(prompt, return_tensors="pt").to(target_model.device)
    generated_ids = inputs.input_ids
    
    for _ in range(max_new_tokens // K):
        # Step 1: Draft phase - generate K tokens with draft model
        draft_outputs = draft_model.generate(
            generated_ids,
            max_new_tokens=K,
            do_sample=False,  # Greedy for deterministic drafting
            return_dict_in_generate=True,
            output_scores=True
        )
        draft_tokens = draft_outputs.sequences[0, -K:]
        
        # Step 2: Verification phase - verify all K tokens with target model
        # Target model processes all K tokens in parallel
        with torch.no_grad():
            target_outputs = target_model(
                torch.cat([generated_ids, draft_tokens.unsqueeze(0)], dim=1)
            )
            target_logits = target_outputs.logits[0, -K-1:-1]  # Logits for K positions
            target_probs = torch.softmax(target_logits, dim=-1)
        
        # Step 3: Acceptance/rejection
        accepted_tokens = []
        for i in range(K):
            draft_token = draft_tokens[i]
            target_token = target_probs[i].argmax()
            
            # Accept if draft matches target's most likely token
            if draft_token == target_token:
                accepted_tokens.append(draft_token)
            else:
                # Reject and use target's token
                accepted_tokens.append(target_token)
                break  # Stop at first mismatch
        
        # Append accepted tokens
        generated_ids = torch.cat([
            generated_ids,
            torch.tensor(accepted_tokens, device=generated_ids.device).unsqueeze(0)
        ], dim=1)
        
        # Stop if we generated enough tokens
        if len(accepted_tokens) < K:
            break
    
    return tokenizer.decode(generated_ids[0], skip_special_tokens=True)

# Usage
prompt = "Once upon a time in a land far away,"
result = speculative_decode(prompt, target_model, draft_model, tokenizer)
print(result)
# Achieves 2-3x speedup compared to standard decoding
# while producing identical output quality

Common Misconceptions

Sources & Further Reading