Experiment tracking interview questions
3 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Knowing which run produced which result
Read the section that teaches thisIf this comes up in an interview
What would you log for a training run?
Hyperparameters, metrics per epoch, the git commit with a dirty flag, the data version as a content hash, the environment and package versions, the seed, and the resulting model as an artifact. The commit and data version are what make it reproducible; the rest is for comparison.
Then they push further
The result is in the notebook. Why is that not enough?
A notebook records the last state of each cell rather than the sequence that produced the number, so a printed 0.93 can sit under code that would no longer produce it. It records outputs and not inputs, it is one run when the comparison needs twenty, and it cannot be searched by result. The record has to be written by the code, to somewhere outside the notebook, on every run.
Why log the seed if you are not chasing exact reproducibility?
Because it is what lets you separate an effect from noise. Sweeping the seed alongside the hyperparameters gives every group in the results table its own spread, and a difference smaller than that spread is not a result.
What gets marked up. Marked up for the data version and the dirty flag, and for treating the seed as a swept parameter rather than a constant.
Two runs with the same seed give different results. Why?
GPU operations are not always deterministic: parallel reductions can vary in order and some algorithms are selected at runtime. You can force determinism at a speed cost, but usually it is better to accept the variance and report a mean across seeds.
Tracking that people actually use
Read the section that teaches thisIf this comes up in an interview
A model in production is behaving differently from what the team remembers testing. How do you find out what was trained?
Look up the model's run record: code commit, data version, config and metrics. If those were not logged, that is the finding and the fix. Then compare that run's evaluation with current production behaviour to see whether the model or the data changed.
Then they push further
The run record shows a data version, but that table has since been overwritten. Now what?
The run cannot be reproduced exactly. For the future, snapshot or version training data, with immutable files or a data versioning tool, and log the snapshot id.
What gets marked up. Marked up for naming code, data, config and environment, and for treating data versioning as the usual gap.
These are the questions for one topic. The whole question bank covers every topic on the site.