Asked about Making it actually work
How do you handle a question your documents cannot answer?
An answer that works
Threshold on retrieval score so a weak best match does not proceed, instruct the model that declining is acceptable, and include no-answer cases in the eval set. A confident wrong answer costs more trust than a refusal.
Then they push further
How do you measure how well it declines?
Put unanswerable questions in the eval set on purpose and measure how often it answers them anyway, alongside how often it wrongly declines questions it could answer. Improving one usually worsens the other, so track both.
What gets marked up
Marked up for treating 'I do not know' as a correct answer with its own test cases, and for tracking wrong refusals too.
Reading an answer is not being able to give one. The section that teaches this is Making it actually work, 4 hours of Retrieval-augmented generation.
Learn this properlyAlso asked about Retrieval-augmented generation