Asked about Evals in the loop
Where do you set the threshold that fails a build?
An answer that works
By measuring, not by taste. Run the suite ten times against an unchanged commit and see how far the score moves on its own; that spread is the floor. Then simulate the gate against no change and against a regression of the size you care about, and pick the threshold where false alarms are rare and real breaks are still caught. On a hundred-and-twenty case set, failing on any drop at all fires on about two runs in five of an unchanged system, which is how teams end up ignoring their own eval.
Then they push further
What if that threshold is too loose to catch the regressions you care about?
Then the set is too small, and no choice of threshold fixes it. At that size you cannot both catch an eight-point regression reliably and avoid failing clean builds. The pull request gate catches the large breaks, and a larger nightly set with a paired comparison catches the rest.
Some cases fail intermittently. What do you do with them?
Find them first: run the suite ten times unchanged and look for cases that gave both verdicts. Then read them, because there are three different causes. An ambiguous grader is the common one and is worth fixing. A case the system genuinely handles half the time is telling the truth and belongs in a reliability report rather than a gate. An inconsistent judge means the rubric needs work. Quarantine keeps them out of the gate while that is decided, but it is a holding pen, not a fix.
What gets marked up
Strong answers measure run-to-run variance before choosing anything and treat the threshold as a trade-off with numbers on both sides. Weak answers give a round number with no reasoning, or propose failing on any regression at all.
Reading an answer is not being able to give one. The section that teaches this is Evals in the loop, 6 hours of Evaluation.
Learn this properlyAlso asked about Evaluation