Derivatives and the chain rule

Training works by nudging millions of numbers in whichever direction makes the model slightly less wrong, and a derivative is what says which direction that is. This section covers the idea as a slope, gradients as the same idea in many dimensions at once, and the chain rule that carries a correction back through a stack of layers. You will not compute these by hand in practice, but when a training run refuses to improve the explanation is almost always here.

A derivative is a rate of change#

If you nudge the input slightly, how much does the output move? That is the derivative. In training, the question is always: if I nudge this parameter, how much does the loss move?

Make that concrete with a function you can compute in your head. Take f of x equals x squared, and stand at x equals 3, where f is 9. Move a whole step to x equals 4 and f becomes 16, so the output rose by 7 for an input rise of 1: a rate of 7. Move a tenth of a step instead, to 3.1, and f becomes 9.61, a rise of 0.61 over 0.1, which is a rate of 6.1. A hundredth of a step gives 6.01. The smaller the nudge, the closer the measured rate gets to 6, and 6 is the derivative of x squared at 3. The formula 2x is a shortcut for that measurement, not a separate idea.

def f(x):
    return x ** 2

for h in (1.0, 0.1, 0.01, 0.00001):
    print(h, (f(3 + h) - f(3)) / h)

# 1.0      7.0
# 0.1      6.100000000000012
# 0.01     6.009999999999849
# 1e-05    6.000009999951316

The sign carries as much information as the number. A derivative of 6 says the output goes up when the input goes up. A derivative of -6 says it goes down. Zero says that, right here, moving the input a little does nothing — which is what being at the bottom of a valley feels like, and also the top of a hill, and also a flat stretch. That ambiguity is worth holding on to: it is why training can stall in places that are not the answer.

Gradients#

With millions of parameters, the derivative with respect to each one is a partial derivative, and the collection of all of them is the gradient. It points in the direction of steepest increase, so training steps in the opposite direction.

A partial derivative is the same measurement with one difference: nudge one input and hold every other one still. Take g of x and y equals x squared plus 3 y squared, at the point x equals 1, y equals 2, where g is 13. Nudge x alone and the rate is 2x, which is 2. Nudge y alone and the rate is 6y, which is 12. So the gradient at that point is the pair [2, 12]: the function is six times more sensitive to y than to x here. That is all a gradient is — one number per input, each one answering how much does the output move if I nudge only this.

def g(x, y):
    return x ** 2 + 3 * y ** 2

eps = 1e-5
x, y = 1.0, 2.0
print(g(x, y))                                      # 13.0
print((g(x + eps, y) - g(x - eps, y)) / (2 * eps))  # 2.0000000000131024
print((g(x, y + eps) - g(x, y - eps)) / (2 * eps))  # 12.000000000078613

Now the claim that makes training work: of all the directions you could step, the gradient points where the function rises fastest, so its negative is where it falls fastest. Test it rather than believing it. Take a step of the same tiny length in four different directions from that point and compare how much g drops.

import numpy as np

grad = np.array([2.0, 12.0])
here = np.array([1.0, 2.0])
step = 0.01

directions = {
    "against the gradient": -grad / np.linalg.norm(grad),
    "along -x only": np.array([-1.0, 0.0]),
    "along -y only": np.array([0.0, -1.0]),
    "some other way": np.array([-0.6, -0.8]),
}
for name, d in directions.items():
    print(name, round(g(*(here + step * d)) - g(*here), 6))

# against the gradient  -0.121361
# along -x only         -0.0199
# along -y only         -0.1197
# some other way        -0.107772

loss = 2.4

gradient tells you, for every parameter:
  increase this one and loss goes up      -> decrease it
  increase this one and loss goes down    -> increase it
  changing this one does nothing          -> leave it

Take a small step against the gradient. Repeat a few million times.

The chain rule#

A network is functions composed with functions. The chain rule says the derivative of a composition is the product of the derivatives along the path. That is what lets a loss at the output be attributed back to a weight in the first layer.

input -> layer1 -> layer2 -> layer3 -> loss

effect of a layer1 weight on the loss
  = its effect on layer1's output
  x layer1's effect on layer2
  x layer2's effect on layer3
  x layer3's effect on the loss

Work one all the way through, because the rule only becomes obvious once you have. Start with a single weight w equal to 0.5, an input x equal to 2, and a target of 1. Three steps: a equals w times x, which is 1.0; b equals tanh of a, which is 0.7616; and the loss is b minus the target, squared, which is 0.0568. Now go backwards, one local derivative per step. The loss changes with b at a rate of 2 times b minus target, which is -0.4768. b changes with a at a rate of 1 minus b squared, which is 0.4200. And a changes with w at a rate of x, which is 2. Multiply the three: -0.4768 times 0.4200 times 2 is -0.4005.

So increasing w decreases the loss here, at a rate of about 0.4 per unit of w. That number was produced with no calculus beyond three memorised derivatives and a multiplication, and the whole of backpropagation is that procedure with more steps and matrices in place of single numbers.

import math

w, x, target = 0.5, 2.0, 1.0

a = w * x                          # 1.0
b = math.tanh(a)                   # 0.7615941559557649
loss = (b - target) ** 2           # 0.056837346474444175

dloss_db = 2 * (b - target)        # -0.4768116880884703
db_da = 1 - b * b                  #  0.41997434161402614
da_dw = x                          #  2.0
print(dloss_db * db_da * da_dw)    # -0.4004973495576554

def loss_of(w_value):              # the numerical check, always worth doing
    return (math.tanh(w_value * x) - target) ** 2

print((loss_of(w + 1e-6) - loss_of(w - 1e-6)) / 2e-6)    # -0.4004973495549957

The step, and how big it should be#

The gradient gives a direction, not a distance. Gradient descent multiplies it by a learning rate and subtracts. Everything that can go wrong with that choice shows up in a function small enough to follow by hand: take the loss to be 5 w squared, whose gradient is 10 w, and start at w equals 1 where the loss is 5. The update is w minus learning rate times 10 w, so each step multiplies w by 1 minus ten times the learning rate.

loss(w) = 5 w^2      gradient = 10 w      start at w = 1

lr = 0.05   multiplier 0.5      1, 0.5, 0.25, 0.125, 0.0625, 0.0312     converging
lr = 0.19   multiplier -0.9     1, -0.9, 0.81, -0.729, 0.6561, -0.5905  bouncing, still shrinking
lr = 0.21   multiplier -1.1     1, -1.1, 1.21, -1.331, 1.4641, -1.6105  bouncing, growing: diverged
lr = 0.002  multiplier 0.98     1, 0.98, 0.9604, 0.9412, 0.9224         technically working

The multiplier is 1 - 10 x lr. It flips sign at lr = 0.1 and exceeds
1 in size at lr = 0.2, which is exactly where divergence begins.

for lr in (0.05, 0.19, 0.21):
    w = 1.0
    history = [w]
    for _ in range(8):
        gradient = 10 * w              # d/dw of 5 w^2
        w = w - lr * gradient
        history.append(round(w, 4))
    print(lr, history)

# 0.05 [1.0, 0.5, 0.25, 0.125, 0.0625, 0.0312, 0.0156, 0.0078, 0.0039]
# 0.19 [1.0, -0.9, 0.81, -0.729, 0.6561, -0.5905, 0.5314, -0.4783, 0.4305]
# 0.21 [1.0, -1.1, 1.21, -1.331, 1.4641, -1.6105, 1.7716, -1.9487, 2.1436]

Two lessons generalise from those nine lines. The threshold depends on the curvature of the loss and not on the learning rate alone: the 5 in 5 w squared is what put the limit at 0.2, and a loss ten times as curved would diverge at a tenth of the rate. That is why a learning rate copied from another project with different data or a different model size so often fails. And the symptom is diagnostic: a loss that oscillates with growing amplitude means the rate is above the threshold, while a loss that falls smoothly but barely means it is far below.

What this explains#

The chain rule being a product explains the two classic training failures directly, which is the practical payoff of understanding it.

• Vanishing gradients: many factors below one multiply towards zero, so early layers stop learning. This is why deep networks needed residual connections and better activations.
• Exploding gradients: factors above one multiply upwards, the update is enormous, and the loss becomes NaN. This is why gradient clipping exists.

How far to go#

You need comprehension, not fluency in manual differentiation. Frameworks compute gradients automatically, and nobody in industry differentiates by hand. The point is being able to reason about why training is misbehaving.