Asked about Derivatives you can check in code

What is the gradient of cross-entropy loss with softmax, with respect to the logits?

An answer that works

The predicted probability vector minus the one-hot target vector. It is simple and well-behaved, which is one reason softmax and cross-entropy are always used together.

Then they push further

Why is sigmoid a poor activation for hidden layers in deep networks?

Its derivative is at most 0.25 and near zero for large inputs, so multiplying many of them in the chain rule shrinks gradients towards zero in early layers. ReLU-family activations have a derivative of 1 for positive inputs.

What gets marked up

Marked up for p minus y without hesitation and for linking the sigmoid derivative to vanishing gradients.

Reading an answer is not being able to give one. The section that teaches this is Derivatives you can check in code, 4 hours of Calculus for ML.

Learn this properly

Also asked about Calculus for ML

What does the learning rate actually do, and how do you tell it is wrong?What is a gradient?Why do gradients vanish in deep networks?Your loss becomes NaN during training. What do you check?Are local minima the main problem in training deep networks?What does momentum do?Explain the Adam update.