Vectors, matrices and what a multiply means

Nearly everything in machine learning is held as a list of numbers or a grid of them, and nearly every operation is one of those multiplied by another. This section covers what those objects represent, why the dot product gets used as a measure of similarity, and the rules about shapes that decide whether a multiplication is allowed at all. It is also the practical cure for the error you will hit most often, which is two things whose dimensions do not line up.

A vector is a list of numbers with a meaning#

A vector is an ordered list of numbers, written [2, 1] or [0.31, -0.08, 1.24, ...]. Ordered matters: the first number always means the same thing as the first number of every other vector you compare it with. That is the whole definition. Everything else said about vectors is a claim about what those numbers are for.

In machine learning a vector is usually a representation: an embedding of a word, the pixels of an image flattened into one row, the features of one customer. A row from a table with columns age, income and months_active becomes [34, 52000, 18]. An embedding from a model is 768 or 1536 numbers, and none of them individually means anything a person can read. Relationships between vectors do: which ones point the same way, which are far apart.

A matrix is a grid of numbers, rows and columns, written as a list of rows. Its shape is two numbers, rows first: a 2 by 3 matrix has two rows of three numbers each. Two very different things are stored this way and it is worth keeping them apart in your head. A batch of data is a matrix with one row per example and one column per feature, so 32 by 768 is thirty-two items, each described by 768 numbers. A weight matrix is a transformation, and its shape says what it turns into what: 768 by 512 takes something of width 768 and produces something of width 512.

scalar        7                        a single number, shape ()
vector        [2, 1]                   shape (2,)      one thing
matrix        [[1, 2, 3], [4, 5, 6]]   shape (2, 3)    2 rows, 3 columns
tensor        anything with more axes  shape (32, 128, 768)   batch, tokens, width

Read a shape right to left: the last number is the width of one item,
and everything before it counts how many items there are.

The dot product#

Multiply corresponding elements and add them up. That single operation is how similarity is measured everywhere in this field. A large dot product means the vectors point in a similar direction.

Do one by hand. Take a = [2, 1] and b = [4, 2]. The dot product is 2 times 4 plus 1 times 2, which is 10. Now take c = [-1, 2]: 2 times -1 plus 1 times 2 is 0. A dot product of zero means the two vectors are at right angles, and in the geometric picture that is exactly what it means: a and c share no direction at all. If the dot product is negative the vectors point away from one another.

The problem with the raw dot product is that it grows with length. The vector [20, 10] points in exactly the same direction as a = [2, 1], and its dot product with a is 50 rather than 10, purely because it is ten times longer. Cosine similarity divides that out: divide the dot product by both lengths and you get 1.0 for both b and [20, 10], because both point the same way as a. Cosine similarity is not a different measure from the dot product. It is the dot product of the two vectors after each has been shrunk to length 1.

import numpy as np

a = np.array([2.0, 1.0])
b = np.array([4.0, 2.0])        # same direction, twice as long
c = np.array([-1.0, 2.0])       # at right angles
long = np.array([20.0, 10.0])   # same direction, ten times as long

print(a @ b, a @ c, a @ long)                     # 10.0 0.0 50.0
print(np.linalg.norm(a))                          # 2.2360679... = sqrt(5)

def cosine(u, v):
    return (u @ v) / (np.linalg.norm(u) * np.linalg.norm(v))

print(cosine(a, b), cosine(a, c), cosine(a, long))
# 0.9999999999999998 0.0 0.9999999999999999

import numpy as np

a = np.array([1, 2, 3])
b = np.array([2, 3, 4])

np.dot(a, b)          # 1*2 + 2*3 + 3*4 = 20

# Cosine similarity is the dot product with length divided out,
# which is why it compares direction rather than magnitude.
np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Matrix multiplication, done by hand#

A matrix multiply is many dot products at once. Each output entry is one row of the first matrix dotted with one column of the second. That is why the inner dimensions must match: you cannot dot vectors of different lengths.

Nothing replaces doing one. Take A as 2 by 3 and B as 3 by 2. The result is 2 by 2, and each of its four entries is a separate dot product between a row of A, which has three numbers, and a column of B, which also has three.

A = [[1, 2, 3],        B = [[ 7,  8],
     [4, 5, 6]]             [ 9, 10],
                            [11, 12]]
     2 x 3                   3 x 2        ->  result is 2 x 2

entry [0,0] = row 0 of A . column 0 of B = 1*7  + 2*9  + 3*11 = 7 + 18 + 33 = 58
entry [0,1] = row 0 of A . column 1 of B = 1*8  + 2*10 + 3*12 = 8 + 20 + 36 = 64
entry [1,0] = row 1 of A . column 0 of B = 4*7  + 5*9  + 6*11 = 28 + 45 + 66 = 139
entry [1,1] = row 1 of A . column 1 of B = 4*8  + 5*10 + 6*12 = 32 + 50 + 72 = 154

A @ B = [[ 58,  64],
         [139, 154]]

import numpy as np

A = np.array([[1, 2, 3], [4, 5, 6]])          # 2 x 3
B = np.array([[7, 8], [9, 10], [11, 12]])     # 3 x 2

print(A @ B)
# [[ 58  64]
#  [139 154]]

print(B @ A)          # legal too, and a different size and meaning
# [[ 39  54  69]
#  [ 49  68  87]
#  [ 59  82 105]]

Two consequences follow from that hand calculation and they cover most of what you need. The inner dimensions must agree, because a dot product needs two lists of the same length. And the cost is one multiply-add per entry per inner element: an m by n times n by p multiply does m times n times p multiply-adds. A 4096 by 4096 weight matrix against a batch of 32 rows is about 537 million of them, which is why this operation, and not anything else in a network, is what GPUs are bought for.

(m x n) @ (n x p)  ->  (m x p)

The two n values must be equal. The result takes the outer pair.

(32 x 768) @ (768 x 512)  ->  (32 x 512)      valid
(32 x 768) @ (512 x 768)  ->  error           inner dims differ

Shape errors are the most common bug in machine learning code, and they are common for a reason that has nothing to do with the maths being hard. Every array in the program is a grid of anonymous numbers, so nothing except the shape says whether you are holding a batch of examples or a weight matrix, and nothing stops you passing one where the other belongs. The error, when you get one, names the two shapes and nothing else. Read it as a question: which of these two did I mean to transpose?

A = np.array([[1, 2, 3], [4, 5, 6]])     # 2 x 3
try:
    A @ A
except ValueError as e:
    print(e)
# matmul: Input operand 1 has a mismatch in its core dimension 0,
# with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3)

print((A @ A.T).shape)      # (2, 2)   row against row: 2 examples compared
print((A.T @ A).shape)      # (3, 3)   column against column: 3 features compared

Why a neural network is matrix multiplies#

A layer takes a batch of inputs, multiplies by a weight matrix, adds a bias and applies a non-linearity. The whole network is that chain repeated. Once you see it this way, the architecture diagrams become readable.

# batch of 32 items, each 768 numbers
x = np.random.randn(32, 768)
W = np.random.randn(768, 512)     # weights: in_features x out_features
b = np.random.randn(512)

h = np.maximum(0, x @ W + b)      # (32, 512), ReLU applied

Three things are worth noticing in those four lines. The batch dimension, 32, is untouched by the multiply and comes out the other side unchanged: the same weights are applied to every item independently, which is why a batch is just a way of keeping the hardware busy. The weight matrix is written in_features by out_features, so reading W as 768 by 512 tells you the layer narrows its input. And the bias has shape (512,) while the result has shape (32, 512), so adding them is not a matrix operation at all. It is broadcasting, which is the next thing to understand.

Broadcasting#

NumPy stretches smaller arrays to match larger ones when the shapes are compatible. It is what lets you add a bias of shape (512,) to a result of shape (32, 512). It is also a common source of silent bugs, because an unintended broadcast produces a valid array of the wrong shape rather than an error.

a = np.ones((32, 512))
b = np.ones(512)
(a + b).shape         # (32, 512)  intended

c = np.ones((32, 1))
(a * c).shape         # (32, 512)  probably intended

d = np.ones((1, 32))
(a.T * d).shape       # (512, 32)  almost certainly a bug

The rule NumPy applies is mechanical: line the shapes up from the right, and for each pair of axes either the two must be equal or one of them must be 1, in which case that one is repeated. So (32, 512) and (512,) line up as 512 against 512, fine. (32, 512) and (32,) line up as 512 against 32, which is neither equal nor 1, so that raises an error — a bias vector applied along the wrong axis crashes, which is the good case.

The bad case is the one below, and it is worth more attention than anything else on this page because it produces no error at all. A vector of predictions with shape (3,) minus a column of targets with shape (3, 1) lines up as 3 against 1, so the 1 is stretched: you get a 3 by 3 matrix of every prediction minus every target. The mean of its squares is a real number, computed without complaint, and it is wrong.

pred = np.array([1.0, 2.0, 3.0])          # shape (3,)
true = np.array([[1.0], [2.0], [3.0]])    # shape (3, 1) -- a column, from a dataframe

err = pred - true
print(err.shape)                # (3, 3)   not (3,)
print((err ** 2).mean())        # 1.3333333333333333

print(((pred - true.ravel()) ** 2).mean())   # 0.0 -- the predictions are perfect

The habit that prevents this costs nothing: after any operation that combines two arrays, print the shape of the result and check it against the shape you expected. A loss should be a scalar, a prediction vector should have one number per example. If a loss has a shape at all, you have this bug.

Where each of these shows up in real code#

operation             where you meet it
dot product           an embedding search scoring a query against a document
cosine similarity     the same search when document lengths differ
matrix multiply       every layer of every network: x @ W
the transpose         Q @ K.T in attention, scoring every token against every token
broadcasting          adding a bias, subtracting a mean, applying a mask
shapes                the error message you will read more often than any other

How far to go#

For ML engineering: shapes, transposes, broadcasting, dot products and matrix multiplication. That is enough to read papers and debug training. Eigenvalues, decompositions and change of basis matter for research and rarely for engineering.

What is worth memorising is short: the inner dimensions must match and the outer two survive; the dot product is elementwise multiply then sum; cosine is that with both lengths divided out; broadcasting stretches an axis of length 1. Everything else — determinants, matrix inverses computed by hand, Cramer's rule, row reduction — is what a first-year linear algebra course spends its time on and what you can skip entirely. If you ever need an inverse, and you rarely should, a solver does it better than the formula.