AI Dictionary of Terms

Positional Encoding

A technique used in Transformer models to inject information about the relative or absolute position of tokens in a sequence, since the self-attention mechanism itself is inherently order-agnostic.

The Simple Version

A way to tell an AI the order of words in a sentence. Because Transformers look at all words at once, they don’t inherently know that “The dog bit the man” is different from “The man bit the dog.” Positional encoding adds a “location tag” to each word so the model understands the sequence.

Detailed Explanation

Unlike Recurrent Neural Networks (RNNs) which process data sequentially, Transformers process entire sequences in parallel. To preserve the sequential nature of language, positional encodings (often using sine and cosine functions of different frequencies) are added to the input token embeddings. This allows the attention mechanism to calculate relationships based on relative distances between tokens.

Key Characteristics

Order Preservation: Crucial for syntax and grammar; without it, a bag-of-words model results.
Extrapolation: Learned positional embeddings struggle with sequences longer than those seen in training, whereas sinusoidal encodings can theoretically extrapolate to infinite lengths.
Learned vs. Fixed: Models like BERT use learned positional embeddings, while the original Transformer paper used fixed sinusoidal functions.

Business Context

Document Understanding: Enables models to process long-form text, legal contracts, and code where the order of information is strictly legally or logically binding.
Time-Series Analysis: Adapted for financial or sensor data where the chronological order of data points dictates the predictive outcome.

Real-World Analogy

Numbering the pages of a manuscript before shredding it and handing it to a team of researchers. Even if they read the pages out of order, the page numbers allow them to reconstruct the original narrative flow.

Code Example

# Conceptual: Generating sinusoidal positional encodings (PyTorch style)
import torch
import math

def get_positional_encoding(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe.unsqueeze(0) # Shape: (1, seq_len, d_model)

# Added to token embeddings before passing to the Transformer encoder.

Common Misconceptions

Myth: Transformers know the order of words naturally.
Reality: Self-attention is permutation invariant. Without positional encoding, it treats sentences as unordered sets.
Myth: Positional encoding only works for text.
Reality: It is heavily used in vision transformers (ViTs) to encode the 2D spatial position of image patches.

Sources & Further Reading

Vaswani, A., et al. Attention Is All You Need. NeurIPS 2017