Reading the maths in papers
By the end: You can decode common notation, work through an equation by checking shapes and plugging in numbers, and turn it into code.
- Notation you will see constantly
- Shapes first
- Plug in small numbers
- Turn it into ten lines of code
New to reading papers? Research is published as papers: eight or ten page documents, most of them free on arXiv, each reporting one result. Reading one is not reading it front to back. It is a triage — the abstract and the figures to find what is being claimed, the method section to see what was actually done, the results table and the baseline it was compared against to judge whether the claim survives, and the full text only if the paper has earned it. Why it matters
The equations in a machine learning paper are usually a compressed description of about ten lines of code, and the notation is a small fixed vocabulary that recurs almost everywhere. This section gives you that vocabulary, then a method that works on any equation you meet: check the shape of everything in it, substitute small numbers and compute by hand, then write the code. The rest of the section does exactly that to the equation you will meet most often, at the pace you should do it yourself.
A small dictionary#
E_{x ~ p}[f(x)] average of f(x) over x drawn from p -> a mean over samples
argmax_x f(x) the x that makes f largest -> the best choice, not the value
nabla_theta L gradient of L w.r.t. parameters -> what loss.backward() computes
KL(p || q) how different q is from p -> not symmetric
sum_i / prod_i add up / multiply together over i -> a loop, or an axis argument
x ~ p x is drawn from the distribution p -> a sample
|| x || length of the vector x -> np.linalg.norm
x . y or x^T y dot product -> (x * y).sum()
1[condition] 1 if the condition holds, else 0 -> a boolean used as a number
theta / phi parameters being learned -> the weights
hat{y} a predicted value, not the true one -> model output
log p(x) log probability -> always logs, to avoid underflowShapes before meaning#
For every symbol, write down its shape: scalar, vector of length d, matrix n by d. Do this before trying to understand what the equation means, because most equations become obvious once the shapes are known, and a shape that does not line up is proof you have misread something rather than proof the maths is hard. Take scaled dot-product attention, written in every transformer paper as softmax of Q K-transpose over the square root of d-k, times V.
Q [n_q, d] one row per query
K [n_k, d] one row per key
V [n_k, d_v] one row per value, and there are as many values as keys
Q K^T [n_q, d] @ [d, n_k] -> [n_q, n_k] every query against every key
/ sqrt(d) shape unchanged a scalar divide
softmax(...) [n_q, n_k], each row summing to 1 the rows are the weights
... @ V [n_q, n_k] @ [n_k, d_v] -> [n_q, d_v] one output row per queryPlug in numbers#
Now make it concrete with the smallest example that still has structure: one query, three keys, two dimensions. Choose values you can compute in your head — a query that points along the first axis, and keys that point along the first, the second, and diagonally between them.
Q = [[1, 0]] one query, pointing along the first axis
K = [[1, 0], [0, 1], [1, 1]] three keys: same direction, perpendicular, between
V = [[10, 0], [0, 10], [5, 5]]
Q K^T = [[1, 0, 1]] query matches key 1 and key 3, not key 2
/ sqrt(2) = [[0.707, 0, 0.707]]
softmax = [[0.401, 0.198, 0.401]] the two matching keys share most of the weight
@ V = [[6.017, 3.983]] mostly value 1 and value 3, a little of value 2Then push it to the extremes, which is where the understanding actually arrives. Multiply Q by ten: the scores become 10, 0, 10, the softmax becomes 0.4998, 0.0004, 0.4998, and the output is a hard selection rather than a blend — which is why the scale of the scores matters and why the division by the square root of d is in the equation at all. Without it, larger d makes the dot products larger, the softmax sharper, and the gradients smaller.
Code it#
The last step is to write the ten lines and check them against the numbers you computed by hand. If they agree, you have read the equation correctly; if they do not, one of the two is wrong and finding out which is the fastest learning available.
import numpy as np
def attention(Q, K, V):
d = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d) # [n_q, n_k]
scores = scores - scores.max(-1, keepdims=True) # subtract max: no overflow
w = np.exp(scores)
w = w / w.sum(-1, keepdims=True) # rows sum to 1
return w @ V # [n_q, d_v]
Q = np.array([[1.0, 0.0]])
K = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
V = np.array([[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]])
print(np.round(attention(Q, K, V), 3))
# [[6.017 3.983]]The same four steps handle equations that look far worse. The DPO objective below is a page of derivation in its paper and four lines here, and the notation is all in the dictionary above: an expectation is a mean, pi and ref are two models' log probabilities of the same text, and the subscripts w and l are the preferred and rejected responses.
# L = -E[ log sigma( beta * ((log pi(y_w|x) - log ref(y_w|x))
# - (log pi(y_l|x) - log ref(y_l|x))) ) ]
import torch.nn.functional as F
def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1): # all log probs, shape [batch]
margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))
return -F.logsigmoid(margin).mean()When it is genuinely hard
Some papers do require real background — a measure-theoretic bound, a convergence proof. The judgement is whether the maths is load-bearing for what you need. If you want to use the method, the loss function and the algorithm box are load-bearing and the proof in the appendix is not. Read the parts you will implement, and skip the rest without guilt until a result depends on it.
Where people go wrong
Skipping the equations and implementing from the prose. The details that matter, such as a sign, a normalisation or an expectation over the wrong distribution, are in the equations.
If this comes up in an interview
How do you approach a dense equation in a paper you need to implement?
Identify every symbol and its shape, rewrite it in words, substitute small numbers and check extreme cases, then write it as code with shape comments and test it on a case where I know the answer.
Build this: Pick three equations from papers you care about. For each, write the symbol table with shapes, a hand-worked example with tiny numbers, and a short implementation with a test.
Watch it explained
This is one section of Reading papers, 19 hours across 4 sections. Read it in full to get the whole topic in order.