Asked about When the loss will not go down
Your model's training loss is flat from the first step. How do you debug it?
An answer that works
Check the initial loss against the expected value for uniform guessing, try to overfit one small batch, decode and inspect a batch with its labels, sweep the learning rate by factors of ten, and log gradient norms to see whether gradients reach the parameters at all.
Then they push further
It cannot overfit a single batch. What does that tell you?
The problem is in the model, the loss or the optimiser setup, not the data volume. Common causes are parameters not passed to the optimiser, a detached tensor, a wrong loss for the output format, or labels that do not match the inputs.
What gets marked up
Marked up for 'overfit one batch' as an early step and for a systematic order rather than random changes.
Reading an answer is not being able to give one. The section that teaches this is When the loss will not go down, 4 hours of PyTorch.
Learn this properlyAlso asked about PyTorch