Checking LLM eval scores against what users actually do
The AI Product work group on testing eval scores against finished tasks, rephrased questions, edits and the time people spend checking an answer.
Updated
Build eval cases from failures users actually hit, then check whether a rising score comes with better outcomes in the product. Where the two disagree, the rubric needs work.
- Start from real tasks. Sachin Rekhi would sample real tasks, ask users whether the output helped them finish, and record what they had to correct or redo. Those failures become eval cases, grouped by task and user segment.
- Count the time and the checking. Ami Vora would add to the eval whether the person finished in the time they actually have and how much checking or retrying it took. Since an accepted answer can still create work, she would watch a few people use an answer and ask which claims they checked.
- Look for rephrased questions. Dennis Yang would label cases where a user repeats or rephrases the question right after a plausible answer, then track those requests after release. If the eval score rises and those requests rise too, he would revisit the rubric.
- Compare scores with the next action. Sam Stone prefers natural next actions, such as accepting, editing, saving or sharing, to thumbs-up prompts, and would inspect high-scoring outputs that get heavily edited or abandoned. Hemal Shah would test whether a user can trace a connector answer to a current source, and ask users which of two scored answers they could safely use.
Ask a follow-up
AI Product
Frequently Asked Questions
What should a first eval setup for an LLM product include?
A sheet of 50 to 100 real outputs graded by people with notes, a rubric checked against customer evidence, a fixed bug set and rule checks in code, and an agreed list of failures that block a launch.
Which parts of a product should become agentic first?
A frequent, bounded job inside a workflow people already use, where they can review and correct the result before it goes out. Prototype it on real inputs before committing to the full agent.
How can a working professional build AI skills that last?
Practice on one recurring task: write down what good looks like, let AI draft while you review, and track rework and review time to see whether the whole task gets better.
How is AI changing the product manager's job?
Execution and coordination get cheaper, so more of the job is choosing the problem, judging output and supervising agents. Sachin Rekhi still wants PMs talking to customers themselves, and Ami Vora wants them close to those conversations.
Which operations work can AI agents handle today?
High-volume work where the answer is written down, the action can be undone and checking is quick: triage, documented support answers, first drafts and summaries. Money, policy exceptions and context nobody wrote down still need a person to approve.
How will AI change services businesses?
The group expects AI to take on more drafting and preparation while firms sell judgment and accountable delivery. Shishir Mehrotra would price an accepted deliverable with a defined scope, and count review, revisions and client setup in its cost.
Why doesn't AI adoption show up in business results more often?
Usage is an input. Results come from changing one workflow and measuring its outcome, including the time spent checking and fixing AI output and whether a decision changed.
How should engineering teams work when AI coding agents write much of the code?
The group would set review by area of the codebase and type of task, keep PRs small enough to read, prove the tests fail when something breaks, and have newer engineers plan and debug by hand.
Where does AI speed up design and content work?
Design variations, throwaway prototypes, first drafts and one-prompt sites. Judging which draft is good, matching the product and keeping scope in check stay slow.
