Asked about Automatic differentiation from scratch

How does reverse-mode automatic differentiation work?

An answer that works

During the forward pass each operation records its inputs and how to compute its local derivative. Backward starts with a gradient of 1 at the output and visits nodes in reverse topological order, each passing its gradient times its local derivative to its inputs, summing contributions where a value was used more than once.

Then they push further

Why is reverse mode used for training rather than forward mode?

Training has one scalar output, the loss, and millions of inputs, the parameters. Reverse mode gets all parameter gradients in one backward pass; forward mode would need one pass per parameter.

Why must gradients be zeroed between training steps in PyTorch?

Gradients accumulate by design, because a value used in several places needs the sum of its gradients. Without zeroing, each step would add to the previous step's gradients.

What gets marked up

Marked up for topological order, summing gradients, and the reverse versus forward mode argument.

Reading an answer is not being able to give one. The section that teaches this is Automatic differentiation from scratch, 5 hours of Backpropagation.

Learn this properly

Also asked about Backpropagation

Your loss is not decreasing. How do you debug it?What does Adam do that SGD does not?Why use a learning rate warmup?What happens if you forget optimizer.zero_grad()?How does batch size relate to learning rate?Why AdamW rather than Adam with L2 regularisation?You have written a custom layer. How do you know its backward pass is right?What causes vanishing gradients and how are they addressed?