Asked about What a neural network is
What is the dying ReLU problem?
An answer that works
A unit whose input is always negative outputs zero and has zero gradient, so it never updates and is permanently dead. Leaky ReLU and GELU avoid it by keeping a small gradient for negative inputs.
Reading an answer is not being able to give one. The section that teaches this is What a neural network is, 5 hours of Neural networks.
Learn this properlyAlso asked about Neural networks
Why do neural networks need activation functions?Why is ReLU the default rather than sigmoid?Which loss would you use for multi-class classification?Why do residual connections help deep networks?Why do transformers use layer norm rather than batch norm?Walk me through backpropagation for a two-layer network.Your validation loss starts rising while training loss keeps falling. What is happening and what do you do?