AI Dictionary of Terms

Activation Function

A mathematical function applied to the output of each neuron in a neural network that introduces non-linearity, enabling the network to learn complex patterns and relationships that cannot be captured by linear transformations alone.

The Simple Version

Imagine you’re voting on whether to go to a party. Each friend gives you a reason (input), and you weight how important each reason is. But you don’t just add up the weighted reasons — you apply a decision rule: “If the total score is above 7, I’ll go. Otherwise, I won’t.”

That decision rule is like an activation function. Without it, the neural network would just be a series of linear equations (addition and multiplication), which can only learn straight-line relationships. Activation functions introduce the “decision rules” that let the network learn complex, non-linear patterns.

Common activation functions include:

Detailed Explanation

Activation functions are applied after each linear transformation (weights × inputs + bias) in a neural network. They determine whether a neuron should “fire” (activate) based on its input.

Why Non-Linearity Matters: Without activation functions, a neural network with multiple layers is mathematically equivalent to a single-layer network. No matter how many layers you add, the network can only learn linear relationships. Activation functions break this limitation, enabling the network to approximate any function (Universal Approximation Theorem).

Common Activation Functions:

1. ReLU (Rectified Linear Unit):

f(x) = max(0, x)

2. GELU (Gaussian Error Linear Unit):

f(x) = x · Φ(x)  (where Φ is cumulative distribution of standard normal)

3. SiLU / Swish:

f(x) = x · sigmoid(x)

4. Sigmoid:

f(x) = 1 / (1 + exp(-x))

5. Tanh (Hyperbolic Tangent):

f(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))

6. Leaky ReLU:

f(x) = max(αx, x)  (where α is small, e.g., 0.01)

7. Softmax:

f(x_i) = exp(x_i) / Σ exp(x_j)

Choosing Activation Functions:

Use Case Recommended Activation
Hidden layers (CNNs, feedforward) ReLU or GELU
Transformers GELU
Binary classification output Sigmoid
Multi-class classification output Softmax
RNNs / LSTMs Tanh (hidden), Sigmoid (gates)
When ReLU dying is a problem Leaky ReLU or GELU

Key Characteristics

Business Context

While activation functions are a technical detail, understanding them helps interpret model behavior and training dynamics:

Why It Matters:

Enterprise Implications:

Real-World Analogy

A bouncer at a club. The bouncer decides who gets in based on their input (appearance, ID, etc.). Different bouncers have different rules:

The bouncer’s rule (activation function) determines how inputs are transformed into outputs.

Code Example

# Common activation functions in PyTorch
import torch
import torch.nn as nn
import matplotlib.pyplot as plt

# Create sample input
x = torch.linspace(-5, 5, 100)

# 1. ReLU
relu = nn.ReLU()
y_relu = relu(x)

# 2. GELU
gelu = nn.GELU()
y_gelu = gelu(x)

# 3. Sigmoid
sigmoid = nn.Sigmoid()
y_sigmoid = sigmoid(x)

# 4. Tanh
tanh = nn.Tanh()
y_tanh = tanh(x)

# 5. Leaky ReLU
leaky_relu = nn.LeakyReLU(negative_slope=0.1)
y_leaky = leaky_relu(x)

# Neural network with activation functions
class SimpleNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 64)
        self.activation = nn.GELU()  # Activation after linear layer
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 1)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.activation(x)  # Non-linearity introduced here
        x = self.fc2(x)
        x = self.activation(x)
        x = self.fc3(x)
        return x  # No activation on output (for regression)

# For classification, use sigmoid or softmax on output
class ClassificationNetwork(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.fc1 = nn.Linear(10, 64)
        self.activation = nn.ReLU()
        self.fc2 = nn.Linear(64, num_classes)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.activation(x)
        x = self.fc2(x)
        x = torch.softmax(x, dim=1)  # Softmax for multi-class
        return x

print("Activation functions demonstrated successfully")

Common Misconceptions

Sources & Further Reading