What a neural network is
By the end: You can explain the architecture and build one from scratch.
- A layer as a matrix multiply plus a non-linearity
- Why the non-linearity is essential
- Activations and which to use
- Loss functions
New to neural networks? A neural network is a stack of simple steps that turn numbers into other numbers. Each layer takes the list of numbers it is given, multiplies and adds them in thousands of small ways, and passes the result on; the values doing that multiplying are called weights, and training means adjusting them. Between the layers sits a small bend — a step that is deliberately not a straight line — and without it the entire stack would collapse into one plain sum no matter how many layers you piled up. Why it matters
A neural network is a chain of very simple steps: multiply the numbers by a grid of weights, add an offset, then bend the result so it is not a straight line, and repeat. This section is about why that structure can represent almost anything, and why the bend is the part that matters, since without it the whole stack collapses into a single multiplication no matter how many layers you give it.
Task Output layer size Final activation Loss
regression, one number 1 none mean squared error
regression, outliers present 1 none Huber or L1
binary classification 1 none (logits) binary cross-entropy with logits
multi-class, one label number of classes none (logits) cross-entropy
multi-class, many labels number of classes none (logits) binary cross-entropy per class
next token vocabulary size none (logits) cross-entropyOne layer#
A layer takes an input vector, multiplies it by a weight matrix, adds a bias, and applies a non-linear function. That is the entire operation, repeated.
It is worth doing once with numbers small enough to check by hand, because every shape error later is a failure to have this picture. Three input features, two output units. The weight matrix is therefore three rows by two columns, one column per output unit, and the bias has one entry per output unit.
import numpy as np
x = np.array([[1.0, 2.0, -1.0]]) # [1, 3] one example, three features
W = np.array([[ 0.5, -0.2], # [3, 2] one column per output unit
[ 1.0, 0.3],
[-0.5, 0.8]])
b = np.array([0.1, -0.4]) # [2] one per output unit
z = x @ W + b # [1, 3] @ [3, 2] -> [1, 2]
print(z) # [[ 3.1 -0.8]]
print(np.maximum(0, z)) # [[ 3.1 0. ]]
# unit 0: 1.0*0.5 + 2.0*1.0 + (-1.0)*(-0.5) + 0.1 = 3.1
# unit 1: 1.0*(-0.2) + 2.0*0.3 + (-1.0)*0.8 - 0.4 = -0.8 -> ReLU sets it to 0One example was for legibility. In practice the first dimension is the batch, and nothing else changes: feeding 64 examples makes x shape [64, 3] and z shape [64, 2], with W and b untouched. That is why a batch costs almost nothing extra on a GPU. The same matrix multiply is issued once instead of 64 times.
def layer(x, W, b):
return relu(x @ W + b)
# A three-layer network is that, three times
h1 = layer(x, W1, b1)
h2 = layer(h1, W2, b2)
out = h2 @ W3 + b3 # no activation on the output layerinput [64, 784] a batch of 64 flattened 28x28 images
W1 [784, 256] -> h1 [64, 256] 784*256 = 200,704 weights
W2 [256, 128] -> h2 [64, 128] 256*128 = 32,768 weights
W3 [128, 10] -> out [64, 10] 128*10 = 1,280 weights
plus 394 biases
logits [64, 10] one score per class per exampleWhy the non-linearity matters#
Without it, stacking layers is pointless. Two matrix multiplies in a row are equivalent to one matrix multiply, so a hundred linear layers can represent exactly what one linear layer can. The non-linearity between them is what makes depth mean anything.
x @ A @ B = x @ (A @ B) = x @ C
Three layers collapse to one. Add ReLU between them and they
no longer collapse, which is where the expressive power comes from.That claim is usually asserted. It takes six lines to watch it happen, and the difference between having read it and having run it is what the follow-up question is for.
rng = np.random.default_rng(0)
A = rng.normal(0, 1, (4, 5))
B = rng.normal(0, 1, (5, 3))
C = rng.normal(0, 1, (3, 2))
X = rng.normal(0, 1, (100, 4))
deep = ((X @ A) @ B) @ C # three layers, no activation
single = X @ (A @ B @ C) # one 4x2 matrix
print(np.abs(deep - single).max()) # 3.55e-15 -> the same, to floating point
relu = lambda t: np.maximum(0, t)
deep_relu = relu(relu(X @ A) @ B) @ C
# Fit the best possible single linear map to what the ReLU network computes.
M, *_ = np.linalg.lstsq(X, deep_relu, rcond=None)
print(np.abs(deep_relu - X @ M).max()) # 1.8619 -> no linear map comes closeThe second half is the part that makes the point. It does not merely fail to find the equivalent matrix; it fits the best one that exists by least squares, and it is still wrong by 1.86 on outputs of roughly that size. There is no single matrix to find. Adding ReLU moved the network into a class of functions that no amount of linear algebra can reach.
This is a very common interview question
Why do we need activation functions. The answer is that without them the network is a single linear transformation regardless of depth. Saying it that precisely lands better than saying it adds non-linearity.
Activations#
The choice is nearly always made for you by where in the network you are, and the list below is in the order you will meet them rather than by merit.
Activation Formula Range Use it
ReLU max(0, x) 0 to inf hidden layers, the default, anything convolutional
GELU x * P(Z <= x) -0.17 to inf hidden layers of transformers
SiLU / Swish x * sigmoid(x) -0.28 to inf hidden layers, common in recent vision and LLMs
Leaky ReLU max(0.01x, x) -inf to inf hidden layers, when ReLUs are dying
Tanh (e^2x - 1)/(e^2x + 1) -1 to 1 rarely now; recurrent gates and some output heads
Sigmoid 1 / (1 + e^-x) 0 to 1 output only, binary and multi-label probabilities
Softmax normalised exp sums to 1 output only, one-label multi-classSigmoid and tanh saturate: past roughly plus or minus five the output stops moving, so its gradient goes to nearly zero and the weights feeding it stop being updated. Since backpropagation multiplies these gradients together on the way back, a stack of saturated sigmoids annihilates the signal before it reaches the early layers. This is the vanishing gradient problem, and it is the concrete reason deep networks were not trainable before ReLU.
def sigmoid(t): return 1 / (1 + np.exp(-t))
for v in (0.0, 2.0, 5.0, 10.0):
s = sigmoid(v)
print(v, round(s, 6), round(s * (1 - s), 8))
# x= 0.0 output 0.500000 gradient 0.25000000 the best it ever gets
# x= 2.0 output 0.880797 gradient 0.10499359
# x= 5.0 output 0.993307 gradient 0.00664806
# x=10.0 output 0.999955 gradient 0.00004540 effectively dead
print((sigmoid(5) * (1 - sigmoid(5))) ** 10) # 1.69e-22 through ten such layersReLU has gradient exactly 1 wherever it is active, so it multiplies nothing away, and it costs a comparison rather than an exponential. Its own failure is the mirror image: a unit whose input is always negative outputs zero, has zero gradient, and never recovers. That is the dying ReLU problem, and a learning rate set too high is the usual cause, because one enormous update pushes the unit's bias far negative and it never comes back.
Leaky ReLU fixes it bluntly by keeping a small slope for negatives. GELU and SiLU fix it smoothly, and are what transformers use: both are close to ReLU for large inputs but curve gently through zero, so a unit sitting slightly negative still receives a small gradient and can return. That is most of the practical difference, and it is worth perhaps a fraction of a per cent. Nobody has lost a job over picking GELU instead of SiLU.
Loss functions#
The loss is the only place where you say what you mean by a good answer. Everything else in training is machinery for lowering whatever number the loss produces, so choosing it wrongly means the machinery works perfectly towards the wrong target.
Mean squared error is for regression, and it squares, so a prediction ten units off counts a hundred times a prediction one unit off. That is right when large errors are genuinely much worse, and wrong when the data contains outliers, where it drags the whole model towards a handful of bad rows. Huber loss is the standard compromise: squared for small errors, linear for large ones, so outliers are noticed but not obeyed.
Cross-entropy is for classification, and it is the negative log of the probability assigned to the correct answer. Being right with probability 1 costs zero. Being wrong costs more and more without limit as the confidence rises, which is the behaviour you want and is precisely what squared error fails to provide.
for p in (0.9, 0.5, 0.1, 0.01, 0.001):
print(p, round((1 - p) ** 2, 5), round(-np.log(p), 4))
# predicted probability of the TRUE class
# p MSE cross-entropy
# 0.900 0.01000 0.1054
# 0.500 0.25000 0.6931
# 0.100 0.81000 2.3026
# 0.010 0.98010 4.6052
# 0.001 0.99800 6.9078MSE is bounded above by 1 here, so once a classifier is confidently wrong there is almost no gradient left to correct it. Cross-entropy is unbounded, so the more confidently wrong it is the harder it is pushed back. That is the whole argument for using it, and it is a better answer than reciting that cross-entropy is for classification.
Loss Task Notes
mean squared error regression assumes Gaussian noise, hates outliers
Huber / smooth L1 regression with outliers one threshold parameter to set
cross-entropy one label out of many feed it logits, never softmax output
binary cross-entropy with logits yes-or-no, or multi-label numerically stable version of the above
contrastive / InfoNCE embeddings learns that similar things sit close
CTC sequences without alignment speech and handwritingWhere people go wrong
Applying a softmax and then using a loss function that applies its own. Most frameworks expect raw scores and apply softmax internally, so doing it yourself first produces a subtly wrong gradient and a model that trains badly for no visible reason.
If this comes up in an interview
Why do neural networks need activation functions?
Without them, stacked linear layers collapse into a single linear transformation, since a matrix multiply of matrix multiplies is one matrix multiply. The non-linearity is what makes depth add expressive power.
Why is ReLU the default rather than sigmoid?
Sigmoid saturates at both ends, so gradients approach zero and early layers stop learning. ReLU does not saturate for positive inputs, keeping gradients alive, and it is cheaper to compute.
What is the dying ReLU problem?
A unit whose input is always negative outputs zero and has zero gradient, so it never updates and is permanently dead. Leaky ReLU and GELU avoid it by keeping a small gradient for negative inputs.
Which loss would you use for multi-class classification?
Cross-entropy, usually with softmax at the output. It penalises confident wrong predictions heavily, which is the behaviour you want, whereas squared error treats a confidently wrong answer too gently.
Build this: Build a two-layer network in NumPy for a classification task. Then remove the activation function and confirm it performs the same as a single linear layer.
Watch it explained
This is one section of Neural networks, 18 hours across 4 sections. Read it in full to get the whole topic in order.