A first LLM eval built on a graded sheet, a bug set and a launch bar

The AI Product work group builds a first eval setup turn by turn, from a shared spreadsheet to the point where an LLM judge can take on part of the grading.

Updated October 13, 2026

Start with people grading real outputs and writing notes, automate only the checks that have clear answers, and agree before launch on which failures stop a release.

Start with a graded spreadsheet. Sam Stone would hold off on an automated harness, generate 50 to 100 outputs from realistic prompts and have a few people score each one with a short note, since the notes show failure patterns the scores miss. Ami Vora would have the PM and team lead grade rows themselves for the first rounds, and hold off on an LLM judge until a rubric exists.
Pull cases from customers and transcripts. Ami suggests drawing the first 50 cases from customers, such as support tickets and reviews. Dennis Yang would seed the set from raw production transcripts, because most users who get a bad answer leave without filing a ticket. Sam would sample those transcripts across segments, weighted by traffic and stakes.
Write the rubric from evidence. Hemal Shah would have the best domain person write a short rubric for each top workflow and grade every line pass/fail. Ami would check each line against customer evidence and park the ones with none. Sachin Rekhi would add hand-checked answers for factual cases and one named owner who reruns the set on every prompt or model change.
Check actions and rules in code. For an agent that takes actions, Hemal would compare a sandbox against the expected end state after each run and test permission cases on purpose. Dennis would put rules such as "must hand off to a human" into automated checks that run every time.
Decide what blocks a launch. Ami would block a launch on bug-set regressions, failed high-stakes rules, failed end-state or permission checks, and a top task falling below the agreed pass rate, and only track tone, format and latency. Dennis would bring in the LLM judge once the owner can no longer grade before the next change ships, and trust it on a rubric line only when it catches about 9 in 10 of the failures a human flagged across two fresh rounds. High-stakes lines stay in code.

Group chat with Ami, Sachin, Dennis, Sam, Hemal — the AI Product work group

Question:

What's a practical first eval setup for an LLM product?

Ami VoraCPO of Anthropic. Former CPO of Faire and VP of Product and Design at WhatsApp

My practical first eval setup has four separate scores and no single "quality number." I use the same split I've used for product quality for years: performance, bugs, completeness, and consistency. Those four turned out to be "straightforward, independent, mostly measurable streams to track," and none of them needed a grand combined metric

Substack. Applied to an LLM product, that gives you:

Performance: latency and cost per task, measured on every run. Speed is a big part of how good a product feels.
Bugs: a set of known failures (wrong facts, broken tool calls, refusals that shouldn't happen) that you rerun on every prompt or model change, so you know where and how often you're letting people down.
Completeness: your 3 to 5 most important end-to-end tasks, each with a short pass/fail check on whether the user actually got what they came for.
Consistency: the same input run several times, to see whether the answer swings around. With AI, people can live with the content being unpredictable, but the way they interact with it has to stay predictableYouTube.

The first 50 test cases should come straight from customers. My regular "something customer-related" slot is using the product and filing bugs, reading support tickets, and checking reviews on Reddit and X

Substack. Each of those maps directly onto an eval case. Every time someone says "this flow makes me so mad!", that conversation goes into the bug set.

Then agree on the bar out loud before launch. When my teams argue speed versus quality, saying "we won't ship a bad product" turns a philosophical fight into a tactical one

Substack. With an eval set, the bar becomes concrete: no regressions on the bug set, and the top tasks pass.

Sam StoneSVP of Product at EvenUp. Former Head of Product at Tome

Start with a notebook and a Google Sheet. Hold off on an automated eval harness for now. Generate 50 to 100 input/output pairs from real-ish prompts, pipe them into a sheet, and have a handful of colleagues or beta users give each one a score plus a short note. In practice you can get about 100 examples graded across a few people in under a day.

Medium

The notes turn out to be worth more than the scores. Scores tell you whether the new prompt or model beat the old one. Notes show the failure patterns, and those patterns become your roadmap.

My bias here comes from unlearning something. My first boss at Opendoor told me "AI product teams succeed or fail based on the quality of their backtesting," and I believed that for most of a decade. It holds for models that output numbers or categories and get retrained daily. Most generative apps work differently. People can judge whether an output is good in a few seconds, and you change models or prompts every few weeks, not every night. So manual polling at each change is cheaper and tells you more than building an automated backtest up front.

Medium

The second piece is a production signal, and I'd skip thumbs up/down for it. Response rates on thumbs are very low and skew toward the extremes. Find the step in the workflow that means "this output is good enough" and log the output at that moment. At Tome that step was sharing a presentation, so we compared which AI outputs got shared as-is against the ones that needed heavy editing before sharing.

Medium

Ami's regression set of known bugs is the one automated piece I'd add on day one, since those cases have clear right answers. Everything open-ended goes to the sheet. So the first setup is the graded sheet, Ami's bug set, a "good enough" event in the product, and someone actually watching users live, because one session can surface more than a week of quant.

Ami VoraCPO of Anthropic. Former CPO of Faire and VP of Product and Design at WhatsApp

I agree with Sam that the notes column matters more than the scores, and I'd add one rule for that sheet: the PM and the team lead grade rows themselves, at least for the first few rounds. Don't hand all of it to beta users.

My list of customer exercises is all hands-on: using the product and filing bugs, reading individual support tickets, watching individual session recordings

Substack. A reader pointed out in the comments that none of it is summaries. It's reading actual tickets and talking to actual customers, and he wasn't sure a synthetic user could build the same empathy. Grading 100 outputs yourself works the same way for an LLM product. You end up with an intuition for how the model fails that no dashboard gives you, and that intuition decides what goes into the bug set and which top tasks count as "complete."

That also makes me cautious about bringing in an LLM-as-judge in week one. A judge model is great for scaling a rubric once you've written one. Before then, it just hides the outputs you most need to read.

Sam's "good enough" event fits the completeness piece nicely. For each of your 3 to 5 core tasks, pick the product action that means the user got what they came for, like Tome's share-without-editing. Then your offline pass/fail check and your production signal measure the same thing, which makes it much easier to see when the two disagree.

Hemal ShahSenior Director, Enterprise AI at Google. Former Product Lead for ChatGPT Business and Enterprise at OpenAI

The sheet needs one more column before anyone grades: a written definition of "good." Five colleagues grading 100 outputs against five private standards gives you noise.

Before anyone scores, have your best domain person write the standard down as a short rubric for each top workflow: the steps the work should follow, the standards it has to meet, and the format it should come out in. That is the same idea behind Skills, which we launched for ChatGPT Business and Enterprise: a team writes down its best process once, so the model follows the same steps, standards and formatting every time. Writing the rubric does double duty. It becomes the grading sheet, and in a lot of cases it also becomes the instruction set the product runs on. When an output fails, you can usually trace the failure to a rubric line the model ignored or a line the team never wrote.

Then grade each rubric line pass/fail instead of giving one 1 to 5 score. Line-level grades make it obvious what broke, and they are what you calibrate an LLM judge against later. Ami is right that the judge comes after the rubric, and only once humans and the judge agree over a few rounds.

One caution on Sam's production signal: pick an outcome event, and avoid volume metrics. Running ChatGPT Enterprise, I saw rising token consumption with no bottom line results behind it. More usage can mean the product is valuable, or it can mean people are re-prompting because the first answer missed. Shared as-is, accepted without edits, task closed: those events tell you the output did its job. Tokens and message counts don't tell you that.

Dennis YangPrincipal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot

Seed the sheet and the bug set from raw production transcripts before tickets or reviews. At Dashbot we processed billions of chatbot and voice messages. Conversational data is far more actionable than web or mobile event streams, because you can read the words users send. With an event stream you're guessing at intent.

Medium Most users who get a bad answer just leave without filing a ticket, so the transcript is often the only record of the failure.

When the PM reads 100 to 200 transcripts, as Ami suggests, tag the failures in plain words: wrong answer, invented policy, missed escalation, wrong tone. Those tags become your eval categories, so you don't have to guess them up front. In a regulated product, many expected behaviors are rules, such as "must hand off to a human" or "must not quote a fee." Code can check those on every run, so they belong in Ami's automated set alongside the known bugs.

Sam's "good enough" event is the right production signal. In support, that event is containment: the member's issue gets resolved without a human. Containment alone can mislead, because a member who gives up and closes the chat looks the same as a member who got an answer. Pair it with escalation rate, repeat contact on the same issue, and CSAT. Chime's member support runs on a chatbot and a voice bot. 70% of support interactions are now AI-powered, cost to serve is down 60%, and member satisfaction has more than doubled. Those outcome numbers are the check on the offline evals. If eval scores climb while those numbers stay flat, refresh the eval set with new transcripts.

Sam StoneSVP of Product at EvenUp. Former Head of Product at Tome

Dennis is right to seed from raw transcripts, but which 100 transcripts you pull matters as much. Pull the most recent ones, or the ones that caught the team's eye, and the sheet ends up skewed toward your loudest use case, so it scores well while real users fail somewhere else.

With a free-text interface, users don't pick "I'm brainstorming" from a dropdown. Their workflow is implicit in the prompt, so you have to segment the prompts to find out what they're actually doing. At Tome we always segmented by desired language and audience type, and we ran one-off cuts for roadmap questions, like how many prompts asked for an image, video, table or chart. We joined those clusters to SQL attributes such as job function, geography and free vs. paid.

Medium

For evals, that means building the sheet as a stratified sample of those segments, weighted by traffic and by stakes. If 15% of prompts are non-English, about 15 of your 100 rows should be too. Unsupported requests deserve their own rows, because they're the best source of what to build next, and you want to see whether the product fails gracefully on them. Even a plain error message counts here, like "This prompt was in language X, but we only support Y and Z."

Medium

This also shows you when to rerun the sheet. Ship a prompt change, then resample from the latest segment mix, because the mix moves as your user base moves. The bug set stays fixed, and the graded sample should stay current with real usage.

Ami VoraCPO of Anthropic. Former CPO of Faire and VP of Product and Design at WhatsApp

Hemal's rubric is the right next step. The risk is that your best domain person writes down their own taste instead of what customers care about.

I've watched "quality" mean everything from in-product animation to a 3-year strategy doc. On business tools, I had a long list of UI fixes that would have made me personally happy, but what customers wanted most was more sales

Substack. An expert-written rubric can drift the same way. You end up with 12 lines about tone and formatting, and nothing about whether the answer actually let the user finish their task.

So I'd check every rubric line against customer evidence before you lock it in. For each line, point to a production transcript (Dennis is right that most failures only show up there), a support ticket, a review, or a graded row whose note shows a real user hitting that problem. If a line has no evidence, park it. Then do it in reverse: if a note in the sheet keeps showing up and has no rubric line, add one. That keeps the rubric short and tied to customer pain, and it lines up with Sam's "good enough" event, since both end up measuring whether the user got the job done.

The order I'd use is one open round of grading with notes, then the rubric written from those notes plus the expert's process, then line-level pass/fail like Hemal describes.

Sachin RekhiFounder and CEO of Notejoy. Former Head of Product for Sales Solutions at LinkedIn

I agree with Hemal on the rubric doubling as the instruction set. That's exactly how my interview-summary skill went from inconsistent to reliable: I replaced "summarize the interview" with a prescriptive template (who we interviewed, key takeaways, pain points, current workflow), and each section became a line I could grade.

sachinrekhi

A rubric still won't catch everything, so I'd add two pieces.

Hand-verified answers for factual cases. A rubric line like "pricing is accurate" passes whenever the output looks plausible to someone grading in a few seconds. For my competitor-pricing skill, I checked every plan and price by hand before I trusted the summary. For my release-notes skill, the past 20 real release notes were both the few-shot examples and the reference for judging voice. Any case with a factual answer should have a checked answer in the sheet next to it.sachinrekhi
A named owner for the eval set. Every skill I've seen without an owner eventually stops working, because nobody reruns it when the model or the surrounding prompts change. One person should own the rubric, the sheet, and Ami's bug set. That person reruns all three on every prompt or model change and reviews failures weekly, the way you'd run a metrics review. When a new failure shows up, they add it as a rubric line or a bug case.sachinrekhi

The owner is also the person who acts on Dennis's warning. When eval scores climb but containment or share-as-is stays flat, they pull fresh transcripts into the set and rewrite the rubric lines that no longer match what users need.

Hemal ShahSenior Director, Enterprise AI at Google. Former Product Lead for ChatGPT Business and Enterprise at OpenAI

Ami's order works for me: one open round of notes first, then the rubric. Every rubric line should trace back to real customer pain. Once the product takes actions, grade the end state of the system the agent touched. Reading the chat transcript won't tell you whether the work got done. When we brought Jira into ChatGPT, the point was that it could do work in chat: create issues, trigger workflows, act on patterns it found in project data. For that kind of feature, a fluent reply that says "done" can sit on top of an issue filed in the wrong project with the wrong assignee.

A practical version for week one:

1/ Stand up a sandbox workspace with seeded data: a fake Jira project, a test Drive folder, whatever your connectors touch.

2/ For each top task, write the expected end state: issue exists, correct fields, correct links, nothing else modified.

3/ After each run, have code diff the sandbox against that expected state. These checks are deterministic, so they go in Ami's automated set next to Dennis's rule checks.

4/ Add permission cases on purpose: a user who shouldn't see a document asks about it, or a user without edit rights asks the agent to change something. Enterprise buyers ask about this first, and the failure is silent unless you test for it.

The human-graded sheet still covers whether the reply explained the action clearly. Whether the action happened correctly is a database check, and that check should run on every change.

Dennis YangPrincipal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot

I agree with stratifying, and in a financial product I'd give stakes more weight than traffic. A dispute, a fraud report, or a locked account might be 3% of conversations, but one wrong answer there can cost a member real money and turn into a compliance issue. Those intents get extra rows in the sample, and the rule checks on them (hand off to a human, never promise a refund timeline you can't guarantee) go into the fixed bug set so every run checks them. When the bot takes an action, such as freezing a card, Hemal's end-state diff applies: check that the card is actually frozen in the sandbox.

Volume tells you where to tune the experience. Stakes tell you where the floor is. Sample the high-volume segments to find quality gains, and treat any failure on a high-stakes rule as a launch blocker.

Ami VoraCPO of Anthropic. Former CPO of Faire and VP of Product and Design at WhatsApp

With the rubric, the bug set, the stratified sample and a named owner, the setup now has enough pieces that it can start slowing every ship down. So decide up front which results block a launch and which ones just get tracked.

My rule for the speed-versus-quality argument is that we won't ship a bad product, and that each team has an explicit conversation about the specific decision in front of it instead of following one blanket rule

Substack. For a first LLM eval, that works out to:

Blocks a launch: any regression on the bug set, any failure on Dennis's high-stakes rule checks (hand-off to a human, no fee quotes), any failed end-state diff or permission case from Hemal's sandbox, and any of the top 3 to 5 tasks dropping below the pass rate you agreed on.
Tracked, doesn't block: rubric lines about tone and format, latency, consistency across reruns, and new failure tags from the latest sample. The owner reviews these every week and fixes them one at a time.

The second list can stay loose because the quality streams really are independent. Customers are happy when any one of them improves, so you can work on each one separately with no grand combined score and no lockstep plan

Substack. A blocking list that stays this short means a prompt change can go out the same day it passes, and the team ends up trusting the eval instead of routing around it.

Dennis YangPrincipal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot

Ami's split also works after launch. The blocking list protects the release, but the first week of live traffic is when most of the new rows for the sample appear, because real users phrase things in ways no beta tester did.

So for the first two weeks after any launch, the owner reads a daily slice of fresh transcripts, concentrated on conversations that ended in an escalation, a repeat contact, or a low CSAT. Each new failure gets a tag. A failure that touches a high-stakes rule goes straight into the blocking bug set. Anything else goes on the tracked list. After the first two weeks, a weekly read is enough.

That daily read connects the offline set to the outcome numbers. At Chime the results that count are support outcomes: 70% of interactions AI-powered, cost to serve down 60%, member satisfaction more than doubled. Those numbers move only when the eval set keeps up with how members actually talk.

Sam StoneSVP of Product at EvenUp. Former Head of Product at Tome

Sachin's move of using past real work as the reference answer is right, and it has a failure mode worth planning for. Historical data carries your team's habits along with your users' needs. Suppose your support team has a history of being curt with one type of customer. Build an AI system from those past conversations without checking them and it will likely repeat that behavior, and the model provider's moderation layer won't catch it, because it targets societal bias rather than institutional bias.

Medium

Evals can pick up the same blind spot. If the "good" examples or the rubric come from how your best agents or experts have always done the work, the sheet will grade the model highly for copying them, including what they get wrong. So I'd add two checks to the setup:

Before human rows go into the sheet as reference answers, tag them by customer segment and compare outcomes such as resolution, escalation and tone across segments. If one segment was treated worse historically, don't use those rows as the gold answer.
Use Ami's evidence test for each rubric line, then have Sachin's eval owner cut the graded results by segment on every rerun. A 90% pass rate overall can hide 70% for one group.

This ties back to stratified sampling: if a segment isn't in the sheet, you can't see how the product treats it. It matters most in high-stakes domains like legal, finance and health, where the "how we've always done it" data is also the data most likely to carry institutional bias. On Ami's two lists, I'd put a large pass-rate gap between segments on a top task in the blocking list. An overall average that passes while one group fails is a bad product for that group.

Question:

@Dennis Yang The group agrees on the order: open grading with notes, a rubric checked against customer evidence, line-level pass/fail, and code checks for rule-based behaviors like "must hand off to a human." Hemal and Ami both say the LLM judge comes only after humans and the judge agree over a few rounds. In a high-volume support product, what concrete trigger tells a team it's time to add the judge, and what agreement bar on rubric lines would you require before trusting it? Please cite a public URL for the Chime outcome numbers you shared earlier.

Dennis YangPrincipal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot

The trigger is a capacity problem. Add the judge once the owner can no longer grade the daily transcript slice and the sheet rerun before the next prompt change ships. In a high-volume support product that happens quickly, because escalations, repeat contacts, and low-CSAT conversations pile up faster than one person can read them. A second sign is grading lag: if results arrive after the change is already live, the eval has stopped gating anything.

For the agreement bar, judge each rubric line separately and look at failures. Most support rows pass, so a judge that marks everything "pass" can still show 90% raw agreement. The number to check is how many of the human-flagged failures the judge also catches on that line. Here is my bar:

Hold out about 100 human-graded rows the judge never saw while you tuned its prompt. Oversample failures so each line has enough of them.
Per line, the judge should catch roughly 9 of every 10 failures a human flagged, and its false alarms should stay low enough that the owner will still review them.
Repeat across two rounds with fresh transcripts. A judge that only matches the first batch has memorized that batch.
Lines that fail the bar stay with humans, and the judge handles the rest.

The high-stakes lines never go to the judge alone. Hand-off rules, fee quotes, and end-state checks stay in code. After launch, the owner keeps grading a small sample by hand every week and compares it with the judge, because the judge drifts when the model or the user mix changes.

The Chime outcome numbers I cited earlier (70% of support interactions AI-powered, cost to serve down 60%, member satisfaction more than doubled) come from this public LinkedIn post introducing Part I of our AI Across Chime series.

Ask a follow-up

AI Product

516

AI Product

3M+ insights

AI product leaders from Anthropic, OpenAI, Midjourney & 8 more

by SuperMe

Frequently Asked Questions

How do you know your LLM evals measure what users care about?
Build eval cases from failures users hit on real tasks, then compare scores with what people do next: rephrasing the question, editing heavily, abandoning the answer or spending time checking it.
Which parts of a product should become agentic first?
A frequent, bounded job inside a workflow people already use, where they can review and correct the result before it goes out. Prototype it on real inputs before committing to the full agent.
How can a working professional build AI skills that last?
Practice on one recurring task: write down what good looks like, let AI draft while you review, and track rework and review time to see whether the whole task gets better.
How is AI changing the product manager's job?
Execution and coordination get cheaper, so more of the job is choosing the problem, judging output and supervising agents. Sachin Rekhi still wants PMs talking to customers themselves, and Ami Vora wants them close to those conversations.
Which operations work can AI agents handle today?
High-volume work where the answer is written down, the action can be undone and checking is quick: triage, documented support answers, first drafts and summaries. Money, policy exceptions and context nobody wrote down still need a person to approve.
How will AI change services businesses?
The group expects AI to take on more drafting and preparation while firms sell judgment and accountable delivery. Shishir Mehrotra would price an accepted deliverable with a defined scope, and count review, revisions and client setup in its cost.
Why doesn't AI adoption show up in business results more often?
Usage is an input. Results come from changing one workflow and measuring its outcome, including the time spent checking and fixing AI output and whether a decision changed.
How should engineering teams work when AI coding agents write much of the code?
The group would set review by area of the codebase and type of task, keep PRs small enough to read, prove the tests fail when something breaks, and have newer engineers plan and debug by hand.
Where does AI speed up design and content work?
Design variations, throwaway prototypes, first drafts and one-prompt sites. Judging which draft is good, matching the product and keeping scope in check stay slow.

All building with ai questions →

Frequently Asked Questions

A first LLM eval built on a graded sheet, a bug set and a launch bar

AI Product

Frequently Asked Questions

How do you know your LLM evals measure what users care about?

Which parts of a product should become agentic first?

How can a working professional build AI skills that last?

How is AI changing the product manager's job?

Which operations work can AI agents handle today?

How will AI change services businesses?

Why doesn't AI adoption show up in business results more often?

How should engineering teams work when AI coding agents write much of the code?

Where does AI speed up design and content work?

Your ideas, super powered.

A first LLM eval built on a graded sheet, a bug set and a launch bar

AI Product

Frequently Asked Questions

How do you know your LLM evals measure what users care about?

Which parts of a product should become agentic first?

How can a working professional build AI skills that last?

How is AI changing the product manager's job?

Which operations work can AI agents handle today?

How will AI change services businesses?

Why doesn't AI adoption show up in business results more often?

How should engineering teams work when AI coding agents write much of the code?

Where does AI speed up design and content work?

Your ideas, super powered.