Asked about Batch size, schedules and regularisation in practice
How does batch size relate to learning rate?
An answer that works
Larger batches give less noisy gradients, which tolerate a larger learning rate, so the two are often scaled together. Changing one alone can destabilise or slow training.
Reading an answer is not being able to give one. The section that teaches this is Batch size, schedules and regularisation in practice, 4 hours of Backpropagation.
Learn this properlyAlso asked about Backpropagation
Your loss is not decreasing. How do you debug it?What does Adam do that SGD does not?Why use a learning rate warmup?What happens if you forget optimizer.zero_grad()?Why AdamW rather than Adam with L2 regularisation?You have written a custom layer. How do you know its backward pass is right?How does reverse-mode automatic differentiation work?What causes vanishing gradients and how are they addressed?