A phenomenon in deep neural networks where the gradients used to update weights become exponentially smaller as they are backpropagated through many layers, causing the early layers to stop learning entirely.
When a neural network is too deep, the “error signal” gets diluted as it travels backward. By the time the signal reaches the first few layers, it’s so tiny that those layers don’t update at all, rendering them useless.
During backpropagation, gradients are calculated using the chain rule, which involves multiplying derivatives layer by layer. If the activation function (like Sigmoid or Tanh) has derivatives less than 1 (e.g., Sigmoid’s max derivative is 0.25), multiplying these small numbers repeatedly results in a gradient that approaches zero. This makes training deep networks impossible without specific architectural interventions.
Trying to pass a message through a long line of translators, where each translator is only allowed to pass on 10% of what they heard. By the 10th translator, the message is completely gone.
# Conceptual: Why Sigmoid causes vanishing gradients
import numpy as np
def sigmoid_derivative(x):
s = 1 / (1 + np.exp(-x))
return s * (1 - s) # Max value of this is 0.25 at x=0
# If we have a 10-layer network, and the derivative is 0.25 at each layer:
gradient = 1.0
for _ in range(10):
gradient *= 0.25
print(f"Gradient after 10 layers: {gradient}")
# Output: 0.0000009536... (Effectively zero, weights will not update)