Linear algebra interview questions

What does a dot product tell you?

How aligned two vectors are. It is the sum of elementwise products, and dividing out the magnitudes gives cosine similarity, which compares direction only. It underlies embedding search, ranking and attention.

Why do matrix multiply shapes have to line up?

Each output entry is a row of the first dotted with a column of the second, and a dot product needs equal-length vectors. So the inner dimensions must match and the result takes the outer two.

What is broadcasting and why is it risky?

NumPy stretching a smaller array to match a larger one when shapes are compatible. It is convenient and it can silently produce a valid array of the wrong shape rather than an error, so the bug appears much later as bad results rather than a crash.

What is the difference between L1 and L2 regularisation?

L2 penalises squared weights and shrinks all of them slightly. L1 penalises absolute values and drives many weights to exactly zero, so it performs feature selection. The difference comes from the geometry of the two norms.

What does SVD give you?

A decomposition of any matrix into rotations and a scaling. Truncating to the largest singular values gives the best low-rank approximation, which underlies compression, PCA and parameter-efficient methods like LoRA.

How does PCA relate to eigenvectors?

PCA finds the eigenvectors of the data's covariance matrix. They are the directions of greatest variance, and the eigenvalues say how much variance each direction captures.

What does the dot product of two vectors tell you?

The sum of products of matching entries. Normalised by both lengths it is the cosine of the angle between them, a measure of how aligned they are. It underlies similarity search, attention scores and every layer of a neural network.

How is linear regression a linear algebra problem?

Stack the inputs into a matrix X with a column of ones, and the targets into a vector y. The best weights minimise the squared length of Xw minus y, the least squares problem, which has a closed-form solution and is what lstsq computes.

Explain PCA.

Centre the data, then find the orthogonal directions along which it varies most, in order. Projecting onto the first few keeps as much variance as possible in fewer dimensions. In practice it is computed with an SVD of the centred data matrix.

What gets marked up. Marked up for centring, the SVD connection, and knowing when to standardise.

Why does LoRA use so few trainable parameters?

It represents the update to a weight matrix as the product of two thin matrices of rank r. For a d by d matrix that is 2 times d times r numbers instead of d squared, a small fraction when r is small. It assumes the useful update is low rank.