AI Dictionary of Terms

LSTM (Long Short-Term Memory)

A specialized type of Recurrent Neural Network (RNN) designed to learn long-term dependencies by using a gating mechanism that controls which information to keep, forget, or update, solving the vanishing gradient problem that plagued earlier RNNs.

The Simple Version

Imagine you’re reading a mystery novel. You need to remember clues from the first chapter to understand the plot twist in the final chapter. But you also need to forget irrelevant details — like what the character had for breakfast — so your brain doesn’t get overloaded.

An LSTM works the same way. It has a special “memory cell” that can hold information for a long time. But it also has three “gates” that act like security guards:

This allows the LSTM to remember important things from long ago (like a character’s name from chapter 1) while forgetting irrelevant details (like the weather on page 50).

Detailed Explanation

LSTMs were introduced by Hochreiter & Schmidhuber in 1997 to address the vanishing gradient problem in vanilla RNNs, where gradients become too small to update weights effectively over long sequences.

Architecture Components:

Mathematical Flow (simplified):

  1. Forget Gate: f_t = sigmoid(W_f · [h_{t-1}, x_t] + b_f)
  2. Input Gate: i_t = sigmoid(W_i · [h_{t-1}, x_t] + b_i)
  3. Candidate: C_tilde_t = tanh(W_C · [h_{t-1}, x_t] + b_C)
  4. Update Cell State: C_t = f_t * C_{t-1} + i_t * C_tilde_t
  5. Output Gate: o_t = sigmoid(W_o · [h_{t-1}, x_t] + b_o)
  6. Hidden State: h_t = o_t * tanh(C_t)

Key Advantages over Vanilla RNNs:

Limitations:

Key Characteristics

Business Context

While Transformers dominate modern NLP, LSTMs remain valuable in specific enterprise scenarios:

Where LSTMs excel:

Business considerations:

When to choose LSTM vs. Transformer:

Real-World Analogy

A librarian managing a reading room. The librarian has a long-term memory (cell state) of all the books. When a new patron arrives (new input), the librarian decides:

This selective memory allows the librarian to serve thousands of patrons over a day without getting confused.

Code Example

# LSTM for time series prediction using PyTorch
import torch
import torch.nn as nn

class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(LSTMModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        
        # LSTM layer
        self.lstm = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True
        )
        
        # Fully connected layer for output
        self.fc = nn.Linear(hidden_size, output_size)
        
    def forward(self, x):
        # Initialize hidden state and cell state
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        
        # Forward propagate LSTM
        out, _ = self.lstm(x, (h0, c0))
        
        # Get output from last time step
        out = self.fc(out[:, -1, :])
        
        return out

# Create model for stock price prediction
# Input: 5 features (open, high, low, close, volume)
# Hidden: 64 LSTM units
# Layers: 2 stacked LSTM layers
# Output: 1 (predicted next day's price)
model = LSTMModel(
    input_size=5,
    hidden_size=64,
    num_layers=2,
    output_size=1
)

# Test with sample data
# Batch of 32 sequences, each 30 days long, with 5 features
batch_size = 32
sequence_length = 30
input_features = 5

sample_input = torch.randn(batch_size, sequence_length, input_features)
predictions = model(sample_input)

print("Input shape:", sample_input.shape)  # [32, 30, 5]
print("Output shape:", predictions.shape)  # [32, 1]
print("Sample prediction:", predictions[0].item())

Common Misconceptions

Sources & Further Reading