# Building with AI: evals, agents and the work they change — Summary The AI Product work group answers questions about building with AI in group chats. Members cover LLM evals, what to make agentic and the skills that last. They also discuss how AI changes product, operations, engineering, design and services work, and why adoption doesn't always show up in results. Updated: 2026-10-13 HTML page: [Building with AI: evals, agents and the work they change]() Transcript: [Full Chat Transcript]() Work group: [AI Product]() ## Questions ### [A first LLM eval built on a graded sheet, a bug set and a launch bar]() Start with people grading real outputs and writing notes, automate only the checks that have clear answers, and agree before launch on which failures stop a release. - **Start with a graded spreadsheet.** Sam Stone would hold off on an automated harness, generate 50 to 100 outputs from realistic prompts and have a few people score each one with a short note, since the notes show failure patterns the scores miss. Ami Vora would have the PM and team lead grade rows themselves for the first rounds, and hold off on an LLM judge until a rubric exists. - **Pull cases from customers and transcripts.** Ami suggests drawing the first 50 cases from customers, such as support tickets and reviews. Dennis Yang would seed the set from raw production transcripts, because most users who get a bad answer leave without filing a ticket. Sam would sample those transcripts across segments, weighted by traffic and stakes. - **Write the rubric from evidence.** Hemal Shah would have the best domain person write a short rubric for each top workflow and grade every line pass/fail. Ami would check each line against customer evidence and park the ones with none. Sachin Rekhi would add hand-checked answers for factual cases and one named owner who reruns the set on every prompt or model change. - **Check actions and rules in code.** For an agent that takes actions, Hemal would compare a sandbox against the expected end state after each run and test permission cases on purpose. Dennis would put rules such as "must hand off to a human" into automated checks that run every time. - **Decide what blocks a launch.** Ami would block a launch on bug-set regressions, failed high-stakes rules, failed end-state or permission checks, and a top task falling below the agreed pass rate, and only track tone, format and latency. Dennis would bring in the LLM judge once the owner can no longer grade before the next change ships, and trust it on a rubric line only when it catches about 9 in 10 of the failures a human flagged across two fresh rounds. High-stakes lines stay in code. ### [Checking LLM eval scores against what users actually do]() Build eval cases from failures users actually hit, then check whether a rising score comes with better outcomes in the product. Where the two disagree, the rubric needs work. - **Start from real tasks.** Sachin Rekhi would sample real tasks, ask users whether the output helped them finish, and record what they had to correct or redo. Those failures become eval cases, grouped by task and user segment. - **Count the time and the checking.** Ami Vora would add to the eval whether the person finished in the time they actually have and how much checking or retrying it took. Since an accepted answer can still create work, she would watch a few people use an answer and ask which claims they checked. - **Look for rephrased questions.** Dennis Yang would label cases where a user repeats or rephrases the question right after a plausible answer, then track those requests after release. If the eval score rises and those requests rise too, he would revisit the rubric. - **Compare scores with the next action.** Sam Stone prefers natural next actions, such as accepting, editing, saving or sharing, to thumbs-up prompts, and would inspect high-scoring outputs that get heavily edited or abandoned. Hemal Shah would test whether a user can trace a connector answer to a current source, and ask users which of two scored answers they could safely use. ### [Where an existing product should put its first agent]() Start with a bounded job inside a workflow customers already use, where they can check and edit the result before it goes out, and widen what the agent does as you learn where people still step in. - **Pick a bounded job people can review.** Rachel Wolan would rank surfaces by whether the job is frequent and painful, whether the agent produces a unit the customer already works with, and whether the customer can review it under the right permissions before publishing. Sam Stone would start with draft-and-review work, like the demand packages EvenUp's legal customers edit before they send them. - **Weigh judgment against verification.** Aparna Chennapragada would also score how much judgment the job asks of the agent and how easily the user can check the result, starting from a supervised recurring task such as a weekly digest of customer-issue emails. She suggests writing the spec as the judgment the user wants, then testing it on a small set of real requests. - **Let real demand pick the job.** Dennis Yang would rank tasks by request volume, how often the current flow fails, whether the agent has reliable data and tools, and the cost of a wrong action, and keep sensitive account changes behind explicit confirmation. Rachel would look at what people already do with outside agents, as Webflow saw once it exposed its tools through MCP. - **Show up at the moment of need.** Shishir Mehrotra would build interventions that recognize what the person is doing and bring the context with them, such as Superhuman's Knowledge Checker catching an out-of-date number before someone sends it, ahead of a broad agent that waits for a prompt. - **Prototype, then give it an owner.** Sam would prototype against real inputs and collect 10 to 100 structured evaluations before the final ranking. Rachel would give the first surface a named owner and a decision date, and measure completed jobs, repeat use, human corrections and failed handoffs. ### [AI skills that outlast the next model release]() Practice on a task you already do every week. The skill that carries over is giving AI the right context, a bounded job and a clear standard, then judging what comes back. - **Turn a weekly task into a workflow.** Shishir Mehrotra suggests writing down the information you check, the judgment you make and what a good result must include, then letting an agent draft while you review, for a month. Mark Hull would run a weekly loop: state the user problem, draft with AI, test the result with someone who would use it and note where it failed. - **Write your standards down.** Sachin Rekhi gives Claude Code reference files with his frameworks for judging a strategy. He would have a PM pick one deliverable, write down the criteria for judging it and keep them in portable files for the next tool. Ami Vora would ask a colleague to find a case those criteria miss. - **Make the goal concrete.** Ami would write the AI goal into a performance review and protect time for it, the way she reserves an hour or two late on Friday afternoons for one growth goal. Aparna Chennapragada suggests three habits: use an approved tool on a recurring task, turn one task into a small prototype, and practice delegating a research brief and checking it. - **Judge the whole task and refine the instructions.** Mark would track rework, whether the result met the user's need and how long the review took, since a faster first draft counts only if the whole task improves. Shishir suggests describing intent instead of copying old rules, as when he replaced nearly 800 email filters with four instructions. Aparna would have a colleague try your AI helper without instructions and watch where they hesitate. ### [What AI changes in a product manager's job, and what stays]() Execution is getting cheap, so a PM's value moves toward choosing the problem, judging output and staying close to customers, while supervising agents becomes a new kind of coordination. - **Less coordination, more building.** Sachin Rekhi sees AI speeding up or taking over much of the coordination the role drifted into, and PMs who can't keep up with their engineers. He now writes shorter specs and builds more prototypes. He also says AI is still poor at product strategy. - **Teams and ratios are changing.** Rachel Wolan's count of 1,858 open roles at AI-native companies found 1 PM per 13.5 engineering openings, or about 1 to 8.7 counting forward deployed engineers, and 89% of product roles asking for 5+ years of experience. At Webflow she had been moving toward a lifecycle where two people validate the idea, a small crew hardens it for production, and one owner plus agents keeps it running. - **Working through agents puts PMs in manager mode.** Aparna Chennapragada argues that working through agents pulls makers onto a manager's schedule, and that PMs have to bet against the model curve instead of patching limits that get fixed upstream. Shishir Mehrotra adds that agents don't argue or bring new ideas, so the PM has to supply both. - **Judgment needs deliberate practice.** Ami Vora would hire a utility player who prototypes, roughs out the market and mocks up the go-to-market. Ami and Shishir both say junior PMs need deliberate reps if agents absorb the grunt work. Shishir's test of taste is picking the one good output out of 10 and explaining why. - **Customer contact stays with the PM.** Ami worries that PMs lose touch with customer problems if forward deployed engineers own the conversations. Sachin says feedback tools and engineers' notes can't replace a PM's own interviews. He recommends sending call notes into the same feedback tool as reviews and tickets, reading the raw quotes and calling 20 affected customers when a theme matters. ### [The operations work AI agents can take on now, and what still needs a person]() Agents handle high-volume work today when the answer is written down, the action can be undone and checking the output is quick. Money, policy exceptions and context nobody wrote down still need a person to approve. - **Bounded and reversible runs now.** Rachel Wolan's email agent clears notifications and drafts replies, then waits for her yes on a batch before anything runs. At Webflow, bounded agents such as the AI Site Builder worked where AIDA, an open-ended chatbot that could touch anything in the Designer, was pulled. Aparna Chennapragada adds read-only work nobody had the hours for, like reading every escalated ticket. - **Documented answers can go straight to customers.** At Chime, 70% of support interactions are AI-powered, and Dennis Yang says those documented, information-only answers reach members with no human in between. He would keep approval where money moves or a policy exception gets made. Rachel lists work that depends on context nobody wrote down as not ready yet. - **Checking has to cost less than doing.** Sam Stone draws the map around verification cost. At EvenUp, legal professionals edit AI-drafted demand packages in minutes, and the call on what a case is worth stays with them. Aparna suggests review screens that show what the agent checked, changed and was unsure about, so a person can approve a batch in seconds. - **Put review on a schedule.** Shishir Mehrotra would have people review agent work at set times, like a weekly one-on-one, inside the inbox, doc or ticket where the work already lives. He would ask of every agent when a human reviews its work, and where. - **Learn from what reviewers change.** Dennis would have reviewers tag why they took over a case, so each failure points to a missing article, a missing action or a judgment call. Sam would track which draft sections reviewers rewrite to see where spot checks are enough, and warns that years of tickets also teach an agent a team's bad habits. ### [What AI changes for services firms: the work, the review and the price]() The group expects AI to take on more of the preparation, drafting and follow-through for each client, moving a services firm's value toward checking the work, owning the outcome and pricing a result the client can use. - **One workflow covers more of the job.** Sam Stone points to EvenUp's Express Demands, where legal professionals draft and edit demand packages in minutes, as a firm handling more of a case in one connected workflow while its people check facts and use judgment. Dennis Yang cites Chime's 70% of AI-powered support interactions and says cost and customer experience have to be measured together. - **Firms sell judgment and delivery.** Rachel Wolan expects agencies to sell ongoing judgment and accountable delivery, with AI doing bounded pieces of the work. If she ran one, she would start with a repeatable task the client can inspect before it goes live, and measure revision time and errors as carefully as time saved. - **Quality means the client's result.** Ami Vora would ask customers which result they value and check whether the faster process produces more of it, such as a recommendation the client can act on. Her test, from Faire, is whether it saves the customer time without handing them a confusing new process. - **Watch setup per client.** Shishir Mehrotra saw Centrata grow like a consulting firm, even with multimillion-dollar software deals, because each deployment took so much hands-on work. He would watch how much setup each new client needs: if it stays flat, the firm still scales one specialist at a time. - **Price the accepted deliverable.** Shishir would quote a defined scope and acceptance standard, and track the full cost per accepted deliverable: AI use, review, revisions and setup. For an AI-assisted demand package, Sam would define acceptance as records accounted for, facts matching the sources, checkable calculations, a professional's sign-off and a correction window. The standard cannot promise a settlement amount. ### [Why high AI usage can leave business results unchanged]() Usage counts the input. Results show up when a team changes one workflow and measures what comes out of it, including the time spent checking and fixing the AI's work. - **Measure one workflow, end to end.** Sachin Rekhi would pick one workflow, set a baseline for its time and quality, then check whether the output changed a decision and whether that decision moved the team's goal. Dennis Yang treats Chime's 90% weekly AI use as an input. The measures he points to are in support, where cost to serve is down 60% and member satisfaction has more than doubled. - **Faster output can hide review debt.** At Exceeds, Mark Hull's internal measures showed 99% AI usage and a 1.89x average productivity lift, while review depth fell to 44.8 against a healthy threshold of 60. He would track review, rework, defects, customer outcome and tool cost alongside output, and compare against a baseline. - **Count the checking time.** Sachin's competitor-pricing research became useful only once a browser agent read each current pricing page and he hand-checked the comparison. Dennis would sample the support conversations behind a rising automation rate to find members who had to ask again. In a Chime case study he shared, domain experts owned ground truth and evals while engineering built the pipelines. - **Bring the AI into the work.** Shishir Mehrotra says every detour to a separate chat cuts how often the tool helps, which is why Superhuman has been building Superhuman Go to work inside the doc, inbox or tab. For an AI-drafted customer memo, he would check that its inputs were current and that it changed the customer conversation. - **Test ideas before building them out.** Sachin warns that AI makes it easy to build an idea before anyone checks demand. He added a survey, usage events and session replays to his Notejoy Ask AI prototype before committing engineering time, and would count ideas tested against real use rather than prototypes made. Aparna Chennapragada would widen what an agent does only as evidence supports it. ### [Running an engineering team when agents write most of the code]() When agents write most of the code, the slow parts become deciding what to build and judging the output. Teams need review sized for agent volume, tests that catch failures and engineers who stay hands-on. - **Engineers become managers sooner.** Shishir Mehrotra says token counts only prove people use AI. He thinks AI is turning engineers into managers earlier in their careers, spending most of their time reviewing and steering agent work, so delegating, setting the bar and critique matter most. He warns that a team adding agents to its old code review process will get buried in pull requests. - **Reviews get thinner.** At Exceeds, Mark Hull's team standardized on one tool and model, and agents now write about 80% of its code. Review depth then fell to 44.8 where 60+ is healthy, with people rubber-stamping 323-line PRs, and he says the team hasn't fully solved it. He recommends keeping PRs small enough to read, tracking review depth next to output and treating agent-suggested packages like links from a stranger. - **Set trust by area and task.** Mark describes an engineer who gives agents full autonomy on bug fixes and UI polish and stays in the loop on complex infrastructure. Rachel Wolan decides by area of the codebase: hosting, pricing logic, old customer contracts, regulated workflows and migrations get a human steward, while docs, internal tools, copy and reports can take lighter review. Ami Vora would turn reviews into rules that let some work skip review. - **Prove the tests can fail.** When Rachel's team swapped the model under its app generation agent, the agent kept dying and the evals didn't catch it. She now pushes teams to write evals they expect to fail as well as pass, and to rerun the full set on every model swap. She would track days from greenlit idea to revenue next to output. - **Give newer engineers reps.** Ami would have a newer engineer write the plan before the agent writes the code, debug high-risk incidents by hand and loosen review one task category at a time. She counts them ready when their reviews produce rules others reuse. She worries agents are too agreeable, so she suggests having one argue against the plan first. ### [Where AI saves time in design and content work, and where it doesn't]() AI makes exploration and first drafts fast: design variations in minutes, throwaway prototypes, a site from one prompt. The slow parts are now judging which draft is good and making that draft match your product. - **Explore more directions.** Sachin Rekhi asks prototyping tools like Bolt and Magic Patterns for several layouts at once and combines the best parts. Cheap prototypes let a team build several, test them and keep the winner. He suggests one baseline prototype with the product's functionality and styles, so new work matches the product from day one. - **First drafts come fast.** Rachel Wolan has Claude interview her about her week, research the last two weeks and draft a 1,000-word post that used to take her three or four hours. By early 2026, 22%+ of new Webflow sites started in its AI Site Builder. Sharmeen Browarek Chapp and a co-builder shipped Bukkit, a social bucket list app, in under 48 hours at a Lovable hackathon. - **Keep AI where the work happens.** Shishir Mehrotra says the "AI detour" of switching tools, prompting and pasting results back eats the time the model saved. Rachel's team put inline copy generation inside the Webflow editor for the same reason. - **Judging and scope stay slow.** With 10 AI drafts, someone still has to pick the good one and explain why, which Shishir calls taste. Sharmeen adds scope: when every fix feels like one prompt, hours go to polish. She would plan for ramp-up time, since her team's gains came later in the build as prompts got sharper. - **Question the old defaults.** Jack Brody, drawing on his Snap days, suggests teams question old conventions before using AI's speed to try many approaches. He says capable models often let a team build the breakthrough first and then work out which problem it solves, so designers spend more time productizing new capabilities than executing a set spec. ## Participants - [Ami Vora]() — CPO of Anthropic. Former CPO of Faire and VP of Product and Design at WhatsApp - [Sachin Rekhi]() — Founder and CEO of Notejoy. Former Head of Product for Sales Solutions at LinkedIn - [Dennis Yang]() — Principal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot - [Sam Stone]() — SVP of Product at EvenUp. Former Head of Product at Tome - [Rachel Wolan]() — Former CPO of Webflow and VP and GM of Dropbox Core and AI - [Aparna Chennapragada]() — VP of Wearables at Meta. Former CPO at Microsoft - [Shishir Mehrotra]() — CEO of Superhuman (formerly Grammarly). Co-founder of Coda and former Chief Product Officer of YouTube - [Hemal Shah]() — Senior Director, Enterprise AI at Google. Former Product Lead for ChatGPT Business and Enterprise at OpenAI - [Mark Hull]() — Co-founder and CEO of Exceeds. Former CPO of GoodRx - [Jack Brody]() — CPO of Suno. Former Head of Product and Head of Design at Snap - [Sharmeen Browarek Chapp]() — Former CPO of Midjourney and Head of Product, Revenue Suite at Stripe --- Copyright © 2026 Super Me Network, Inc. All rights reserved. | [Terms of Service](https://www.superme.ai/terms) | [Privacy Policy](https://www.superme.ai/privacy)