AI Dictionary of Terms

Learning Rate

A critical hyperparameter that controls the size of the steps taken during model optimization — determining how much the model’s weights are adjusted in response to the estimated error at each iteration, with values that are too high causing instability and values that are too low causing slow convergence.

The Simple Version

Imagine you’re trying to find the lowest point in a valley while blindfolded. You can feel the slope under your feet and take steps downhill.

The learning rate is the “step size” for your AI model as it learns. Get it right, and training is fast and stable. Get it wrong, and training either fails completely or takes impractically long.

Detailed Explanation

The learning rate (often denoted as η or α) is a scalar that multiplies the gradient when updating model parameters:

parameter_new = parameter_old - learning_rate × gradient

Effects of Different Learning Rates:

Too High:

Too Low:

Just Right:

Learning Rate Schedules: Modern training rarely uses a fixed learning rate. Common schedules include:

1. Step Decay:

2. Exponential Decay:

3. Cosine Annealing:

4. Warmup:

5. One-Cycle Policy:

Typical Learning Rate Ranges:

Learning Rate Finding:

Key Characteristics

Business Context

Learning rate directly impacts training costs and model quality:

Cost Implications:

Enterprise Considerations:

Common Pitfalls:

Best Practices:

Real-World Analogy

Driving a car toward a destination:

The learning rate is like your speed — too fast or too slow both cause problems. The right speed gets you there efficiently.

Code Example

# Learning rate schedules in PyTorch
import torch
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt

# Simple model
model = nn.Linear(10, 1)
optimizer = optim.Adam(model.parameters(), lr=0.001)

# 1. Step Decay Scheduler
scheduler_step = optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)

# 2. Exponential Decay Scheduler
scheduler_exp = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)

# 3. Cosine Annealing Scheduler
scheduler_cosine = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)

# 4. Warmup + Cosine Annealing (common for Transformers)
def get_warmup_cosine_scheduler(optimizer, warmup_steps, total_steps):
    def lr_lambda(step):
        if step < warmup_steps:
            return step / warmup_steps
        progress = (step - warmup_steps) / (total_steps - warmup_steps)
        return 0.5 * (1 + torch.cos(torch.tensor(progress * 3.14159)))
    return optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)

scheduler_warmup = get_warmup_cosine_scheduler(optimizer, warmup_steps=100, total_steps=1000)

# Demonstrate learning rate over training
lrs = []
for epoch in range(100):
    # Training step would go here
    optimizer.step()
    
    # Record learning rate
    lrs.append(optimizer.param_groups[0]['lr'])
    
    # Update scheduler
    scheduler_cosine.step()

# Plot learning rate schedule
plt.plot(lrs)
plt.title("Cosine Annealing Learning Rate Schedule")
plt.xlabel("Epoch")
plt.ylabel("Learning Rate")
plt.grid(True)
plt.show()

# Learning rate range test (finding optimal LR)
def lr_range_test(model, train_loader, start_lr=1e-7, end_lr=1, num_iters=100):
    """Find optimal learning rate by gradually increasing it."""
    optimizer = optim.SGD(model.parameters(), lr=start_lr)
    
    lr_mult = (end_lr / start_lr) ** (1 / num_iters)
    lrs = []
    losses = []
    
    current_lr = start_lr
    for i in range(num_iters):
        optimizer.param_groups[0]['lr'] = current_lr
        
        # Training step (simplified)
        # ... forward, loss, backward ...
        
        lrs.append(current_lr)
        # losses.append(loss.item())
        
        current_lr *= lr_mult
    
    return lrs, losses

# Learning rate finder helps identify the LR where loss decreases fastest
# Typically use LR slightly before the minimum loss point

Common Misconceptions

Sources & Further Reading