AI Dictionary of Terms

Model Serving

The infrastructure and software systems that deploy trained machine learning models into production, making them accessible via APIs for real-time predictions — the critical layer between trained models and end-user applications.

The Simple Version

Imagine you’ve trained a brilliant data scientist (the model). They know everything about your domain and can answer any question. But they’re sitting in a back room with no phone, no email, no way for customers to reach them.

Model serving is like giving that data scientist a phone, an email address, and a receptionist to handle calls. It makes the model accessible to users through APIs, handles multiple requests at once, manages load, and ensures reliability.

Without model serving, you have a trained model that can’t be used. With model serving, you have a production AI system that can serve millions of users.

Detailed Explanation

Model serving encompasses the entire infrastructure stack that makes trained models available for inference in production environments.

Core Components:

1. Model Loading:

2. Request Handling:

3. Inference Execution:

4. Scaling and Load Balancing:

5. Monitoring and Observability:

Model Serving Frameworks:

1. vLLM:

2. Text Generation Inference (TGI):

3. NVIDIA Triton Inference Server:

4. TorchServe:

5. Ray Serve:

6. SGLang:

Key Features of Production Serving:

1. Batching:

2. Streaming:

3. Caching:

4. Quantization:

5. Model Versioning:

6. Security:

Key Characteristics

Business Context

Model serving is the bridge between trained models and business value:

Why It Matters:

Enterprise Considerations:

Build vs. Buy:

Cost Comparison:

When to Self-Host:

When to Use Cloud APIs:

Infrastructure Requirements:

Real-World Analogy

A restaurant kitchen. The chef (model) can cook amazing dishes, but without the kitchen infrastructure (ovens, prep stations, waitstaff, ordering system), they can’t serve customers. Model serving is the kitchen infrastructure that enables the chef to serve hundreds of customers efficiently, handling orders, managing timing, and ensuring quality.

Code Example

# Deploying a model with vLLM (production-grade serving)
from vllm import LLM, SamplingParams

# Initialize the serving engine
llm = LLM(
    model="meta-llama/Llama-2-70b-chat-hf",
    tensor_parallel_size=4,  # Use 4 GPUs
    gpu_memory_utilization=0.9,
    max_model_len=4096,
    quantization="awq",  # Quantized for efficiency
    enable_prefix_caching=True,  # Cache shared prefixes
)

# Define sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=500,
)

# Serve multiple requests (batching)
prompts = [
    "Explain quantum computing in simple terms:",
    "Write a haiku about artificial intelligence:",
    "What are the benefits of renewable energy?",
]

# Generate responses (vLLM handles batching automatically)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt}")
    print(f"Response: {generated_text}")
    print("---")

# vLLM also provides an OpenAI-compatible API server
# Run: python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-70b-chat-hf
# Then use standard OpenAI client to query it

# Alternative: Using Hugging Face TGI (Text Generation Inference)
# Deploy with Docker:
# docker run --gpus all -p 8080:80 \
#   ghcr.io/huggingface/text-generation-inference:latest \
#   --model-id meta-llama/Llama-2-70b-chat-hf

# Query the deployed model
import requests

response = requests.post(
    "http://localhost:8080/generate",
    json={
        "inputs": "What is the capital of France?",
        "parameters": {
            "max_new_tokens": 50,
            "temperature": 0.7,
        }
    }
)

print(response.json()["generated_text"])

Common Misconceptions

Sources & Further Reading