Asked about Estimation, likelihood and uncertainty
What is maximum likelihood estimation?
An answer that works
Writing the probability of the data you actually observed as a function of the parameters, then choosing the parameters that make it largest. The data is fixed and the parameter is varied, which is the opposite of how the same formula is read as a probability.
Then they push further
Show it on something concrete.
The mean of a normal. The log-likelihood, dropping terms that do not involve the mean, is minus the sum of squared distances from it. Differentiating and setting to zero says the deviations must cancel, which happens only at the arithmetic average. So the sample mean is the MLE, and minimising squared error is maximum likelihood under normal noise.
Is the maximum likelihood estimate of the variance correct?
It is biased low. It measures spread around the sample mean rather than the true mean, and the sample mean sits in the middle of that same data, so the distances are too small by a factor of n minus one over n. On samples of five, a true variance of 4 comes out at about 3.2. Dividing by n minus one corrects it.
Where does the choice of loss come in?
The loss is the noise model. Normal noise gives squared error, a Bernoulli label gives cross-entropy, counts give Poisson loss, and heavier tails give absolute error. Choosing a loss is asserting how the data is noisy.
What gets marked up
Strong answers derive at least one estimator rather than defining the term, and name the squared-error connection without prompting.
Reading an answer is not being able to give one. The section that teaches this is Estimation, likelihood and uncertainty, 5 hours of Probability & statistics.
Learn this properlyAlso asked about Probability & statistics