Asked about Knowing whether the AI feature actually helps
You launched an AI writing assistant. How do you know it is helping?
An answer that works
Look at behaviour, not just ratings: how much of each draft survives editing, how often users regenerate, accept rates, and whether the time to finish the task dropped. Tie that to an outcome agreed before launch, with guardrails on complaints and cost.
Then they push further
Thumbs-up rate is 85%. Good news?
Hard to say. Only a small, unrepresentative share of users rate at all. I would check it against behaviour such as edit distance and abandonment, and read a sample of unrated sessions.
Users accept drafts but edit 70% of the text. What does that mean?
The drafts are a starting point but not good enough to send. Read the edits: they show exactly what users wanted instead, and they are excellent data for improving the prompt or building an eval.
What gets marked up
Marked up for behavioural signals and outcome metrics over ratings, and for treating user edits as data.
Reading an answer is not being able to give one. The section that teaches this is Knowing whether the AI feature actually helps, 4 hours of Product judgement for AI.
Learn this properlyAlso asked about Product judgement for AI