AI Dictionary of Terms

Quantization

A model optimization technique that reduces the numerical precision of a neural network’s weights and activations (e.g., from 32-bit floating point to 8-bit integers), dramatically shrinking model size and speeding up inference with minimal loss in accuracy.

The Simple Version

Imagine you have a high-resolution photograph that’s 50MB. It looks beautiful, but it’s huge and slow to load. If you compress it to a JPEG, it becomes 2MB — still looks great, but loads 25x faster. You traded a tiny bit of quality for massive gains in speed and size.

Quantization does the same thing to AI models. Instead of storing each number in the model with 32 bits of precision (like 3.14159265358979), it uses fewer bits — maybe 8 bits (just 3) or even 4 bits (just 3.1). The model becomes 4-8x smaller and runs much faster, while still giving nearly identical answers.

This is why you can now run a 70-billion parameter model like Llama 2 on a laptop — quantization makes it fit.

Detailed Explanation

Neural networks store their “knowledge” as billions of numerical weights. By default, these weights are stored as 32-bit floating point numbers (FP32), which can represent values with extreme precision. Quantization reduces this precision.

Precision Levels:

Two Main Approaches:

1. Post-Training Quantization (PTQ):

2. Quantization-Aware Training (QAT):

Modern Quantization Techniques:

GPTQ (Post-Training):

AWQ (Activation-Aware Weight Quantization):

GGUF (llama.cpp format):

BitsAndBytes (NF4 / FP4):

Key Characteristics

Business Context

Quantization is the key enabler for cost-effective enterprise AI deployment:

Why It Matters:

Enterprise Applications:

ROI Example:

Popular Quantized Model Ecosystems:

Real-World Analogy

A master chef’s recipe written in three versions:

The essence is preserved, but the format is optimized for the situation.

Code Example

# Loading a quantized model using Hugging Face + bitsandbytes
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "meta-llama/Llama-2-7b-hf"

# Configure 4-bit quantization (NF4)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NormalFloat 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16, # Compute in higher precision
    bnb_4bit_use_double_quant=True        # Quantize the quantization constants too
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Load quantized model (uses ~3.5GB instead of ~28GB for FP16)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    device_map="auto"
)

# Compare memory usage
print(f"Model loaded with 4-bit quantization")
print(f"Memory usage: ~3.5GB (vs ~28GB for FP16)")

# Generate text
prompt = "Explain quantization in one sentence:"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Common Misconceptions

Sources & Further Reading