Derivatives you can check in code

Training a model means nudging numbers until a loss goes down, and a derivative is what tells you which way to nudge and by how much. This section takes that idea from something you can feel — change the input a little, see how far the output moves — through to the derivatives of the particular functions that keep appearing in machine learning, and the chain rule that strings them together. Every result here is checked numerically in code, so you never have to take one on faith.

Nudge and measure#

def f(x):
    return x ** 2

def numerical_derivative(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

print(numerical_derivative(f, 3.0))     # about 6.0, and by hand d/dx x^2 = 2x = 6

A derivative says how much the output changes per unit change in the input, for a tiny change. You can always estimate it by nudging the input both ways and measuring. That makes every hand-derived formula checkable.

Nudging both ways rather than one way matters, and so does the size of the nudge. Too large and you are measuring the average slope across a curve rather than the slope at a point; too small and the two function values are so close that floating point subtraction throws away the digits that carried the answer. The usable window is wide but it is not unlimited.

for h in (1e-2, 1e-5, 1e-8, 1e-12, 1e-15):
    print(h, (f(3 + h) - f(3 - h)) / (2 * h))

# 0.01    5.999999999999872
# 1e-05   6.000000000039306
# 1e-08   5.999999963535174
# 1e-12   6.000533403494046
# 1e-15   5.329070518200751      three digits of the answer are gone

The derivatives ML uses constantly#

x^2              ->  2x
e^x              ->  e^x
log(x)           ->  1 / x
sigmoid(x)       ->  sigmoid(x) x (1 - sigmoid(x))
relu(x)          ->  1 if x > 0 else 0

softmax followed by cross-entropy, with respect to the logits:
                 ->  p - y       (predicted probabilities minus the one-hot target)

The last line is worth memorising. The gradient of the standard classification loss with respect to the logits is simply the predicted probabilities minus the true labels. The model is pushed up on the right class by how much it underestimated it, and down on the others.

Confirm it on three numbers rather than trusting it. Logits of 2.0, 1.0 and 0.1 become probabilities 0.659, 0.242 and 0.099 after a softmax. If the correct class is the second one, the target vector is [0, 1, 0], so p minus y is [0.659, -0.758, 0.099]. Positive on the two wrong classes, meaning push those logits down; negative on the right one, meaning push it up, and by the largest amount because that is where the model was most wrong.

import numpy as np
import torch
import torch.nn.functional as F

logits = np.array([2.0, 1.0, 0.1])
e = np.exp(logits - logits.max())          # subtract the max: no overflow
probs = e / e.sum()
target = np.array([0.0, 1.0, 0.0])         # the true class is index 1

print(probs.round(6))                       # [0.659001 0.242433 0.098566]
print((probs - target).round(6))            # [ 0.659001 -0.757567  0.098566]

t = torch.tensor(logits, requires_grad=True)
loss = F.cross_entropy(t.unsqueeze(0), torch.tensor([1]))
loss.backward()
print(t.grad.numpy().round(6))               # [ 0.659001 -0.757567  0.098566]
print(loss.item())                           # 1.4170300162778335

The chain rule through two steps#

import math

x, w, target = 2.0, 0.5, 1.0
z = w * x                     # step 1
p = 1 / (1 + math.exp(-z))    # step 2: sigmoid
loss = (p - target) ** 2      # step 3

dloss_dp = 2 * (p - target)
dp_dz = p * (1 - p)
dz_dw = x
dloss_dw = dloss_dp * dp_dz * dz_dw     # multiply along the chain
print(z, p, loss)              # 1.0 0.7310585786300049 0.07232948812851325
print(dloss_dw)                # -0.21150837113706686

Read the three factors as a sentence. The loss is 0.0723 and the prediction is 0.731 against a target of 1, so the first factor is negative: the prediction needs to go up. The second factor, 0.197, is the sigmoid's derivative at this point, and it is a dampener — no matter how wrong the prediction is, this factor can never exceed 0.25. The third is the input, 2, which is why an input of zero gives that weight no gradient at all. Increase w and the loss falls at a rate of 0.21.

Check against autograd#

w_t = torch.tensor(0.5, requires_grad=True)
p_t = torch.sigmoid(w_t * 2.0)
((p_t - 1.0) ** 2).backward()
print(w_t.grad.item())        # -0.2115083485841751

That seven-digit agreement is the standard you want from a gradient check. Order-of-magnitude agreement means you have the shape of the derivative right and a factor wrong somewhere. A sign flip means you subtracted in the wrong order. Agreement on some entries and not others usually means an axis is being summed that should not be.

Why sigmoid ruins deep networks, in one line of arithmetic#

The sigmoid's derivative is largest at the middle, where it is exactly 0.25, and it falls away fast on either side. At an input of 2 it is 0.105; at 4 it is 0.0177; at 8 it is 0.000335. Since the chain rule multiplies one of these per layer, ten sigmoid layers at their very best multiply the gradient by 0.25 to the power of ten, which is 0.00000095. The early layers of such a network receive a signal a million times weaker than the last layer, before any of the other factors are taken into account.

import math

for z in (0, 1, 2, 4, 8):
    s = 1 / (1 + math.exp(-z))
    print(z, round(s, 6), round(s * (1 - s), 8))

# 0  0.5       0.25         the best it ever is
# 1  0.731059  0.19661193
# 2  0.880797  0.10499359
# 4  0.982014  0.01766271
# 8  0.999665  0.00033524   effectively no gradient at all

print(0.25 ** 10)      # 9.5367431640625e-07
print(0.25 ** 20)      # 9.094947017729282e-13

What to memorise, and what to skip#

Five derivatives are worth having by heart because they turn up in every explanation: x squared goes to 2x, e to the x is its own derivative, log x goes to 1 over x, sigmoid goes to sigmoid times one minus sigmoid, and softmax with cross-entropy goes to p minus y. Add the chain rule, which is multiply the local derivatives along the path, and the numerical check, which is nudge and measure. That is the entire working set.

Skip the rest of a calculus course without hesitation: integration by parts, trigonometric derivatives, limits and epsilon-delta proofs, series expansions, differential equations. None of them appears in an engineering interview for these roles and none is needed to reason about training. If a paper uses one, it will be in a derivation you are reading rather than implementing.