Asked about A week of improving one feature, worked through
Tell me how you would improve an AI feature that 'feels unreliable'.
An answer that works
Read a hundred or so traces and categorise the failures, count them, and build an eval around the top modes with a held-out split. Then make one change at a time, starting with the most common failure, keep what improves the tune set at acceptable cost, and check the held-out set at the end.
Then they push further
Your first idea is to switch to a bigger model. Why might that be wrong?
If the failures come from outdated documents or lost context, a bigger model reads the same bad inputs. It also multiplies cost. Fix the causes the traces show first, then test a bigger model as one change among others.
How do you know you did not overfit to your eval?
Keep a held-out split you look at rarely, and check that it moves with the tuning set. Then watch production failure rates after release, since real traffic is the final held-out set.
What gets marked up
Strong answers start from data, change one thing at a time, weigh cost, and use a held-out set. This question often stands in for a behavioural round, so a real example with numbers scores highest.
Reading an answer is not being able to give one. The section that teaches this is A week of improving one feature, worked through, 4 hours of Eval-driven development.
Learn this properlyAlso asked about Eval-driven development