# Building with AI: evals, agents and the work they change — Full Chat Transcript The AI Product work group answers questions about building with AI in group chats. Members cover LLM evals, what to make agentic and the skills that last. They also discuss how AI changes product, operations, engineering, design and services work, and why adoption doesn't always show up in results. Updated: 2026-10-13 HTML page: [Building with AI: evals, agents and the work they change]() Summary: [Summary]() Work group: [AI Product]() ## Participants - [Ami Vora]() — CPO of Anthropic. Former CPO of Faire and VP of Product and Design at WhatsApp - [Sachin Rekhi]() — Founder and CEO of Notejoy. Former Head of Product for Sales Solutions at LinkedIn - [Dennis Yang]() — Principal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot - [Sam Stone]() — SVP of Product at EvenUp. Former Head of Product at Tome - [Rachel Wolan]() — Former CPO of Webflow and VP and GM of Dropbox Core and AI - [Aparna Chennapragada]() — VP of Wearables at Meta. Former CPO at Microsoft - [Shishir Mehrotra]() — CEO of Superhuman (formerly Grammarly). Co-founder of Coda and former Chief Product Officer of YouTube - [Hemal Shah]() — Senior Director, Enterprise AI at Google. Former Product Lead for ChatGPT Business and Enterprise at OpenAI - [Mark Hull]() — Co-founder and CEO of Exceeds. Former CPO of GoodRx - [Jack Brody]() — CPO of Suno. Former Head of Product and Head of Design at Snap - [Sharmeen Browarek Chapp]() — Former CPO of Midjourney and Head of Product, Revenue Suite at Stripe ## Full Chat Transcript #### [A first LLM eval built on a graded sheet, a bug set and a launch bar]() **Question:** What's a practical first eval setup for an LLM product? ##### [Ami Vora]() My practical first eval setup has four separate scores and no single "quality number." I use the same split I've used for product quality for years: performance, bugs, completeness, and consistency. Those four turned out to be "straightforward, independent, mostly measurable streams to track," and none of them needed a grand combined metric [Citation: https://amivora.substack.com/p/the-4-components-of-product-quality]. Applied to an LLM product, that gives you: - **Performance:** latency and cost per task, measured on every run. Speed is a big part of how good a product feels. - **Bugs:** a set of known failures (wrong facts, broken tool calls, refusals that shouldn't happen) that you rerun on every prompt or model change, so you know where and how often you're letting people down. - **Completeness:** your 3 to 5 most important end-to-end tasks, each with a short pass/fail check on whether the user actually got what they came for. - **Consistency:** the same input run several times, to see whether the answer swings around. With AI, people can live with the content being unpredictable, but the way they interact with it has to stay predictable [Citation: https://www.youtube.com/watch?v=Vp2BsJNVFqQ]. The first 50 test cases should come straight from customers. My regular "something customer-related" slot is using the product and filing bugs, reading support tickets, and checking reviews on Reddit and X [Citation: https://amivora.substack.com/p/unlocking-simplicity-customer-empathy]. Each of those maps directly onto an eval case. Every time someone says "this flow makes me so mad!", that conversation goes into the bug set. Then agree on the bar out loud before launch. When my teams argue speed versus quality, saying "we won't ship a bad product" turns a philosophical fight into a tactical one [Citation: https://amivora.substack.com/p/making-the-tradeoff-between-speed]. With an eval set, the bar becomes concrete: no regressions on the bug set, and the top tasks pass. ##### [Sam Stone]() Start with a notebook and a Google Sheet. Hold off on an automated eval harness for now. Generate 50 to 100 input/output pairs from real-ish prompts, pipe them into a sheet, and have a handful of colleagues or beta users give each one a score plus a short note. In practice you can get about 100 examples graded across a few people in under a day. [Citation: https://medium.com/p/29c9f5c2f321] The notes turn out to be worth more than the scores. Scores tell you whether the new prompt or model beat the old one. Notes show the failure patterns, and those patterns become your roadmap. My bias here comes from unlearning something. My first boss at Opendoor told me "AI product teams succeed or fail based on the quality of their backtesting," and I believed that for most of a decade. It holds for models that output numbers or categories and get retrained daily. Most generative apps work differently. People can judge whether an output is good in a few seconds, and you change models or prompts every few weeks, not every night. So manual polling at each change is cheaper and tells you more than building an automated backtest up front. [Citation: https://medium.com/p/29c9f5c2f321] The second piece is a production signal, and I'd skip thumbs up/down for it. Response rates on thumbs are very low and skew toward the extremes. Find the step in the workflow that means "this output is good enough" and log the output at that moment. At Tome that step was sharing a presentation, so we compared which AI outputs got shared as-is against the ones that needed heavy editing before sharing. [Citation: https://medium.com/p/29c9f5c2f321] Ami's regression set of known bugs is the one automated piece I'd add on day one, since those cases have clear right answers. Everything open-ended goes to the sheet. So the first setup is the graded sheet, Ami's bug set, a "good enough" event in the product, and someone actually watching users live, because one session can surface more than a week of quant. ##### [Ami Vora]() I agree with Sam that the notes column matters more than the scores, and I'd add one rule for that sheet: the PM and the team lead grade rows themselves, at least for the first few rounds. Don't hand all of it to beta users. My list of customer exercises is all hands-on: using the product and filing bugs, reading individual support tickets, watching individual session recordings [Citation: https://amivora.substack.com/p/unlocking-simplicity-customer-empathy]. A reader pointed out in the comments that none of it is summaries. It's reading actual tickets and talking to actual customers, and he wasn't sure a synthetic user could build the same empathy. Grading 100 outputs yourself works the same way for an LLM product. You end up with an intuition for how the model fails that no dashboard gives you, and that intuition decides what goes into the bug set and which top tasks count as "complete." That also makes me cautious about bringing in an LLM-as-judge in week one. A judge model is great for scaling a rubric once you've written one. Before then, it just hides the outputs you most need to read. Sam's "good enough" event fits the completeness piece nicely. For each of your 3 to 5 core tasks, pick the product action that means the user got what they came for, like Tome's share-without-editing. Then your offline pass/fail check and your production signal measure the same thing, which makes it much easier to see when the two disagree. ##### [Hemal Shah]() The sheet needs one more column before anyone grades: a written definition of "good." Five colleagues grading 100 outputs against five private standards gives you noise. Before anyone scores, have your best domain person write the standard down as a short rubric for each top workflow: the steps the work should follow, the standards it has to meet, and the format it should come out in. That is the same idea behind Skills, which we launched for ChatGPT Business and Enterprise: a team writes down its best process once, so the model follows the same steps, standards and formatting every time. Writing the rubric does double duty. It becomes the grading sheet, and in a lot of cases it also becomes the instruction set the product runs on. When an output fails, you can usually trace the failure to a rubric line the model ignored or a line the team never wrote. Then grade each rubric line pass/fail instead of giving one 1 to 5 score. Line-level grades make it obvious what broke, and they are what you calibrate an LLM judge against later. Ami is right that the judge comes after the rubric, and only once humans and the judge agree over a few rounds. One caution on Sam's production signal: pick an outcome event, and avoid volume metrics. Running ChatGPT Enterprise, I saw rising token consumption with no bottom line results behind it. More usage can mean the product is valuable, or it can mean people are re-prompting because the first answer missed. Shared as-is, accepted without edits, task closed: those events tell you the output did its job. Tokens and message counts don't tell you that. ##### [Dennis Yang]() Seed the sheet and the bug set from raw production transcripts before tickets or reviews. At Dashbot we processed billions of chatbot and voice messages. Conversational data is far more actionable than web or mobile event streams, because you can read the words users send. With an event stream you're guessing at intent. [Citation: https://medium.com/chat-bots/dashbots-dennis-yang-why-conversational-data-captured-his-attention-and-why-you-should-care-too-34f03996a001] Most users who get a bad answer just leave without filing a ticket, so the transcript is often the only record of the failure. When the PM reads 100 to 200 transcripts, as Ami suggests, tag the failures in plain words: wrong answer, invented policy, missed escalation, wrong tone. Those tags become your eval categories, so you don't have to guess them up front. In a regulated product, many expected behaviors are rules, such as "must hand off to a human" or "must not quote a fee." Code can check those on every run, so they belong in Ami's automated set alongside the known bugs. Sam's "good enough" event is the right production signal. In support, that event is containment: the member's issue gets resolved without a human. Containment alone can mislead, because a member who gives up and closes the chat looks the same as a member who got an answer. Pair it with escalation rate, repeat contact on the same issue, and CSAT. Chime's member support runs on a chatbot and a voice bot. 70% of support interactions are now AI-powered, cost to serve is down 60%, and member satisfaction has more than doubled. Those outcome numbers are the check on the offline evals. If eval scores climb while those numbers stay flat, refresh the eval set with new transcripts. ##### [Sam Stone]() Dennis is right to seed from raw transcripts, but which 100 transcripts you pull matters as much. Pull the most recent ones, or the ones that caught the team's eye, and the sheet ends up skewed toward your loudest use case, so it scores well while real users fail somewhere else. With a free-text interface, users don't pick "I'm brainstorming" from a dropdown. Their workflow is implicit in the prompt, so you have to segment the prompts to find out what they're actually doing. At Tome we always segmented by desired language and audience type, and we ran one-off cuts for roadmap questions, like how many prompts asked for an image, video, table or chart. We joined those clusters to SQL attributes such as job function, geography and free vs. paid. [Citation: https://medium.com/p/29c9f5c2f321] For evals, that means building the sheet as a stratified sample of those segments, weighted by traffic and by stakes. If 15% of prompts are non-English, about 15 of your 100 rows should be too. Unsupported requests deserve their own rows, because they're the best source of what to build next, and you want to see whether the product fails gracefully on them. Even a plain error message counts here, like "This prompt was in language X, but we only support Y and Z." [Citation: https://medium.com/p/29c9f5c2f321] This also shows you when to rerun the sheet. Ship a prompt change, then resample from the latest segment mix, because the mix moves as your user base moves. The bug set stays fixed, and the graded sample should stay current with real usage. ##### [Ami Vora]() Hemal's rubric is the right next step. The risk is that your best domain person writes down their own taste instead of what customers care about. I've watched "quality" mean everything from in-product animation to a 3-year strategy doc. On business tools, I had a long list of UI fixes that would have made me personally happy, but what customers wanted most was more sales [Citation: https://amivora.substack.com/p/making-the-tradeoff-between-speed]. An expert-written rubric can drift the same way. You end up with 12 lines about tone and formatting, and nothing about whether the answer actually let the user finish their task. So I'd check every rubric line against customer evidence before you lock it in. For each line, point to a production transcript (Dennis is right that most failures only show up there), a support ticket, a review, or a graded row whose note shows a real user hitting that problem. If a line has no evidence, park it. Then do it in reverse: if a note in the sheet keeps showing up and has no rubric line, add one. That keeps the rubric short and tied to customer pain, and it lines up with Sam's "good enough" event, since both end up measuring whether the user got the job done. The order I'd use is one open round of grading with notes, then the rubric written from those notes plus the expert's process, then line-level pass/fail like Hemal describes. ##### [Sachin Rekhi]() I agree with Hemal on the rubric doubling as the instruction set. That's exactly how my interview-summary skill went from inconsistent to reliable: I replaced "summarize the interview" with a prescriptive template (who we interviewed, key takeaways, pain points, current workflow), and each section became a line I could grade. [Citation: https://www.sachinrekhi.com/p/claude-code-for-product-managers] A rubric still won't catch everything, so I'd add two pieces. 1. Hand-verified answers for factual cases. A rubric line like "pricing is accurate" passes whenever the output looks plausible to someone grading in a few seconds. For my competitor-pricing skill, I checked every plan and price by hand before I trusted the summary. For my release-notes skill, the past 20 real release notes were both the few-shot examples and the reference for judging voice. Any case with a factual answer should have a checked answer in the sheet next to it. [Citation: https://www.sachinrekhi.com/feed] 2. A named owner for the eval set. Every skill I've seen without an owner eventually stops working, because nobody reruns it when the model or the surrounding prompts change. One person should own the rubric, the sheet, and Ami's bug set. That person reruns all three on every prompt or model change and reviews failures weekly, the way you'd run a metrics review. When a new failure shows up, they add it as a rubric line or a bug case. [Citation: https://www.sachinrekhi.com/p/how-to-build-a-compounding-os] The owner is also the person who acts on Dennis's warning. When eval scores climb but containment or share-as-is stays flat, they pull fresh transcripts into the set and rewrite the rubric lines that no longer match what users need. ##### [Hemal Shah]() Ami's order works for me: one open round of notes first, then the rubric. Every rubric line should trace back to real customer pain. Once the product takes actions, grade the end state of the system the agent touched. Reading the chat transcript won't tell you whether the work got done. When we brought Jira into ChatGPT, the point was that it could do work in chat: create issues, trigger workflows, act on patterns it found in project data. For that kind of feature, a fluent reply that says "done" can sit on top of an issue filed in the wrong project with the wrong assignee. A practical version for week one: 1/ Stand up a sandbox workspace with seeded data: a fake Jira project, a test Drive folder, whatever your connectors touch. 2/ For each top task, write the expected end state: issue exists, correct fields, correct links, nothing else modified. 3/ After each run, have code diff the sandbox against that expected state. These checks are deterministic, so they go in Ami's automated set next to Dennis's rule checks. 4/ Add permission cases on purpose: a user who shouldn't see a document asks about it, or a user without edit rights asks the agent to change something. Enterprise buyers ask about this first, and the failure is silent unless you test for it. The human-graded sheet still covers whether the reply explained the action clearly. Whether the action happened correctly is a database check, and that check should run on every change. ##### [Dennis Yang]() I agree with stratifying, and in a financial product I'd give stakes more weight than traffic. A dispute, a fraud report, or a locked account might be 3% of conversations, but one wrong answer there can cost a member real money and turn into a compliance issue. Those intents get extra rows in the sample, and the rule checks on them (hand off to a human, never promise a refund timeline you can't guarantee) go into the fixed bug set so every run checks them. When the bot takes an action, such as freezing a card, Hemal's end-state diff applies: check that the card is actually frozen in the sandbox. Volume tells you where to tune the experience. Stakes tell you where the floor is. Sample the high-volume segments to find quality gains, and treat any failure on a high-stakes rule as a launch blocker. ##### [Ami Vora]() With the rubric, the bug set, the stratified sample and a named owner, the setup now has enough pieces that it can start slowing every ship down. So decide up front which results block a launch and which ones just get tracked. My rule for the speed-versus-quality argument is that we won't ship a bad product, and that each team has an explicit conversation about the specific decision in front of it instead of following one blanket rule [Citation: https://amivora.substack.com/p/making-the-tradeoff-between-speed]. For a first LLM eval, that works out to: - **Blocks a launch:** any regression on the bug set, any failure on Dennis's high-stakes rule checks (hand-off to a human, no fee quotes), any failed end-state diff or permission case from Hemal's sandbox, and any of the top 3 to 5 tasks dropping below the pass rate you agreed on. - **Tracked, doesn't block:** rubric lines about tone and format, latency, consistency across reruns, and new failure tags from the latest sample. The owner reviews these every week and fixes them one at a time. The second list can stay loose because the quality streams really are independent. Customers are happy when any one of them improves, so you can work on each one separately with no grand combined score and no lockstep plan [Citation: https://amivora.substack.com/p/the-4-components-of-product-quality]. A blocking list that stays this short means a prompt change can go out the same day it passes, and the team ends up trusting the eval instead of routing around it. ##### [Dennis Yang]() Ami's split also works after launch. The blocking list protects the release, but the first week of live traffic is when most of the new rows for the sample appear, because real users phrase things in ways no beta tester did. So for the first two weeks after any launch, the owner reads a daily slice of fresh transcripts, concentrated on conversations that ended in an escalation, a repeat contact, or a low CSAT. Each new failure gets a tag. A failure that touches a high-stakes rule goes straight into the blocking bug set. Anything else goes on the tracked list. After the first two weeks, a weekly read is enough. That daily read connects the offline set to the outcome numbers. At Chime the results that count are support outcomes: 70% of interactions AI-powered, cost to serve down 60%, member satisfaction more than doubled. Those numbers move only when the eval set keeps up with how members actually talk. ##### [Sam Stone]() Sachin's move of using past real work as the reference answer is right, and it has a failure mode worth planning for. Historical data carries your team's habits along with your users' needs. Suppose your support team has a history of being curt with one type of customer. Build an AI system from those past conversations without checking them and it will likely repeat that behavior, and the model provider's moderation layer won't catch it, because it targets societal bias rather than institutional bias. [Citation: https://medium.com/p/e19ded4bde88] Evals can pick up the same blind spot. If the "good" examples or the rubric come from how your best agents or experts have always done the work, the sheet will grade the model highly for copying them, including what they get wrong. So I'd add two checks to the setup: - Before human rows go into the sheet as reference answers, tag them by customer segment and compare outcomes such as resolution, escalation and tone across segments. If one segment was treated worse historically, don't use those rows as the gold answer. - Use Ami's evidence test for each rubric line, then have Sachin's eval owner cut the graded results by segment on every rerun. A 90% pass rate overall can hide 70% for one group. This ties back to stratified sampling: if a segment isn't in the sheet, you can't see how the product treats it. It matters most in high-stakes domains like legal, finance and health, where the "how we've always done it" data is also the data most likely to carry institutional bias. On Ami's two lists, I'd put a large pass-rate gap between segments on a top task in the blocking list. An overall average that passes while one group fails is a bad product for that group. ##### Question > @[Dennis Yang]() The group agrees on the order: open grading with notes, a rubric checked against customer evidence, line-level pass/fail, and code checks for rule-based behaviors like "must hand off to a human." Hemal and Ami both say the LLM judge comes only after humans and the judge agree over a few rounds. In a high-volume support product, what concrete trigger tells a team it's time to add the judge, and what agreement bar on rubric lines would you require before trusting it? Please cite a public URL for the Chime outcome numbers you shared earlier. ##### [Dennis Yang]() The trigger is a capacity problem. Add the judge once the owner can no longer grade the daily transcript slice and the sheet rerun before the next prompt change ships. In a high-volume support product that happens quickly, because escalations, repeat contacts, and low-CSAT conversations pile up faster than one person can read them. A second sign is grading lag: if results arrive after the change is already live, the eval has stopped gating anything. For the agreement bar, judge each rubric line separately and look at failures. Most support rows pass, so a judge that marks everything "pass" can still show 90% raw agreement. The number to check is how many of the human-flagged failures the judge also catches on that line. Here is my bar: 1. Hold out about 100 human-graded rows the judge never saw while you tuned its prompt. Oversample failures so each line has enough of them. 2. Per line, the judge should catch roughly 9 of every 10 failures a human flagged, and its false alarms should stay low enough that the owner will still review them. 3. Repeat across two rounds with fresh transcripts. A judge that only matches the first batch has memorized that batch. 4. Lines that fail the bar stay with humans, and the judge handles the rest. The high-stakes lines never go to the judge alone. Hand-off rules, fee quotes, and end-state checks stay in code. After launch, the owner keeps grading a small sample by hand every week and compares it with the judge, because the judge drifts when the model or the user mix changes. The Chime outcome numbers I cited earlier (70% of support interactions AI-powered, cost to serve down 60%, member satisfaction more than doubled) come from this public LinkedIn post introducing Part I of our AI Across Chime series. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7415601009571352576/] #### [Checking LLM eval scores against what users actually do]() **Question:** How do you know your LLM evals measure what users care about? ##### [Sachin Rekhi]() I’d start with actual user tasks and the feedback around them. In my work, a continuous feedback loop brings together interviews, support issues, usage metrics, and churn feedback. Metrics tell you where users struggle; interviews and support conversations help explain why. I also separate buyer feedback from end-user feedback in B2B products, since they can describe different needs. [Citation: https://www.sachinrekhi.com/p/designing-your-products-continuous-feedback-loop] For an LLM product, I’d take a sample of real tasks, ask users whether the output helped them finish the task, and record what they had to correct or redo. Turn those observed failures into eval cases, grouped by task and user segment. Then compare eval changes against task completion and repeat use after release. A higher grader score is useful only if the behavior it rewards tracks an outcome users actually value. ##### [Ami Vora]() A task can look successful in an eval and still take too much of a person’s time. At Faire, I met a nursing director who ran a hospital gift shop between delivering babies. I kept asking whether the product was simple enough for her to use in the five minutes she had between them. [Citation: https://amivora.substack.com/p/unlocking-simplicity-customer-empathy] For an LLM product, I’d include that constraint in the eval: did the person finish the task within the time they actually have, and how much checking or retrying did it take? I’d inspect cases where a model gets a high answer score while the end-to-end task fails. My quality framework tracks speed, bugs, completeness of the flow, and consistency separately; each can make the experience frustrating even when an answer reads well. [Citation: https://amivora.substack.com/p/the-4-components-of-product-quality] ##### [Dennis Yang]() At Dashbot, we gave bot teams live transcripts and top-message views. The words people type tell you what they are trying to accomplish more directly than a click stream does. [Citation: https://medium.com/p/85abb8296e0d] [Citation: https://medium.com/chat-bots/dashbots-dennis-yang-why-conversational-data-captured-his-attention-and-why-you-should-care-too-34f03996a001] Sachin’s feedback loop becomes especially useful when you read the full exchange. Label cases where the model gives a plausible answer and the user immediately repeats or rephrases the question. Have a reviewer check what the user needed, then use those cases to test whether a model change reduces that specific failure. Track repeat or rephrased requests by task type after release; if the eval score rises and those requests do too, revisit the rubric. ##### [Hemal Shah]() A connector answer can score well while leaving the worker unable to verify the source or act on it. In our public connector launch, we reported GPT-5 was 33% more likely than GPT-4o to return a useful answer with connectors, describing the gain as fewer dead ends and answers that get work done. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7366995713680941056/] Company knowledge later surfaced clear citations across sources, with examples like preparing for customer conversations and writing strategy documents. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7387202364837052416/] I’d make that an eval for a specific workflow: can someone trace the answer to the right source, check that the context is current, and use it to prepare for the meeting? Include a case where the model gives a fluent answer based on stale or unrelated documents. Ask users which of two scored answers they can safely use; the disagreements tell you where the rubric needs work. That follows the product loop I’ve written about: learn through interviews and metrics, then revisit what you ship. [Citation: https://medium.com/p/d5b537398e3a] ##### [Sam Stone]() For generated text, a single reference answer can mark a useful response wrong because one sentence changed. I've found it faster to have colleagues or beta users review 10 to 100 outputs with scores and notes. The notes tell us what to fix more clearly than the average score; a notebook that sends examples to a shared sheet is enough to get started. [Citation: https://medium.com/p/29c9f5c2f321] I would also check whether those reviews predict what people do in the product. For an inline suggestion, do they accept it and keep writing? For generated work, do they save, edit, or share it? I prefer those natural next actions to a thumbs-up prompt, which tends to get few responses and attracts extreme opinions. They are imperfect proxies, so I would look at them across many uses and inspect the cases where a high-scoring output gets heavily edited or abandoned. Those disagreements tell you which part of the eval needs work. [Citation: https://medium.com/p/26516d928566] ##### [Ami Vora]() An accepted answer can still create work for the user. Sam’s edit and abandonment signals help, but I’d also measure the checking a person does before they feel safe using an answer. In my product writing, I’ve argued that usage numbers can miss someone who is unhappy and ready to switch, and that adding features can create costs our dashboards rarely show. [Citation: https://amivora.substack.com/p/find-your-central-product-metaphor] [Citation: https://amivora.substack.com/p/easing-the-tradeoff-between-a-tangible] I’d watch a few people use the answer in their actual workflow and ask which claims they checked and how long that took. Those observations can change the eval rubric even when the user accepted the answer and filed no complaint. #### [Where an existing product should put its first agent]() **Question:** Which parts of an existing product should become agentic first? ##### [Rachel Wolan]() Start with a bounded job inside the workflow customers already use, where they can inspect and edit the result before it reaches production. At Webflow, our AI Site Builder gave customers a useful first step: describe a site and get a draft. We later learned the agent needed to stay with them as they edited in the Designer. Our separate App Gen product taught a sharper lesson: customers asked for reusable components inside their existing sites, so we redirected that work toward components they could place on a page. [Citation: https://www.rachelwolan.com/writing/becoming-agentic-part-1] I’d rank candidate surfaces by three questions: Is the job frequent and painful? Can the agent produce a bounded unit the customer already works with? Can the customer review the change under the right permissions before publishing? The Webflow AI Assistant had real tool access, but enterprise customers wanted brand rules, roles, and granular review before trusting it with production sites. Those controls belong in the first release plan, right alongside the agent’s capability. [Citation: https://www.rachelwolan.com/writing/becoming-agentic-part-1] ##### [Aparna Chennapragada]() I’d score the first surface on a second axis beyond workflow fit: how much judgment it asks the agent to exercise, and how easily the user can verify the answer. In Copilot, I’ve used the example of a weekly request to gather customer-issue emails and send me a digest. It starts with a supervised recurring task; broader skills and more delegation can come as model ability and checks improve. [Citation: https://www.emergingtechbrew.com/stories/2025/03/14/microsoft-workplace-ai-aparna-chennapragada] The spec should describe the judgment the user wants, not merely a tool operation. For a meeting, I’d ask for the important points, key risks, and how sentiment changed; “summarize the transcript in five bullets” sets a much lower bar, while “handle all my email” gives the agent too much scope. Build a small set of these real requests, test the outputs with users, and revise the boundaries before expanding delegation. [Citation: https://aparnacd.substack.com/p/prompt-sets-are-the-new-prds] ##### [Sam Stone]() I'd start where users already assemble information, draft an artifact, and review it before sending. At EvenUp, we launched early access to Express Demands so legal professionals could generate and edit demand packages with AI in minutes. The editing step matters: the lawyer has a clear deliverable and a natural chance to catch errors before the package leaves the product. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7289747436778962944/] For each candidate surface, I'd ask three things: Does the work take several steps that the product can help complete? Can the user see the source material and correct the result? Can we observe whether they accept, edit, or discard the output? I wrote about using those natural actions as feedback, rather than asking users to rate every response. [Citation: https://medium.com/p/26516d928566] Start with a bounded draft-and-review workflow, then expand what the agent can do as you learn where users still have to intervene. ##### [Dennis Yang]() Support conversations are a useful map of where to begin. They expose the requests people cannot complete through the current flow. In my work on conversational analytics at Dashbot, the raw language was more useful than a clickstream because it showed what users were trying to do. I also argued for keeping buttons where they work and using conversation where users need to express a more specific request. [Citation: https://medium.com/chat-bots/dashbots-dennis-yang-why-conversational-data-captured-his-attention-and-why-you-should-care-too-34f03996a001] For an existing product, I’d rank candidate tasks by observed request volume, how often the current flow fails, whether the agent has reliable data and tools to finish the task, and the cost of a wrong action. Start with grounded support tasks such as explaining an account event or guiding a member through a known process. Keep sensitive account changes behind explicit confirmation and a clear handoff. At Chime, we’ve shared that 70% of support interactions are AI-powered, alongside a 60% reduction in cost to serve and more than doubled member satisfaction. Those are results for support as a whole, not proof that every financial action should run autonomously. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7415601009571352576/] ##### [Rachel Wolan]() Aparna’s verification test makes the rollout decision much clearer. I’d give the first agentic surface a named owner and a decision date, then measure completed jobs, repeat use, human corrections, and failed handoffs. A feature can get plenty of prompts while quietly making people clean up its work. In my framework for AI distribution, I also count maintenance cost per successful retained workflow, because permissions, evals, and support keep accumulating after launch. [Citation: https://www.rachelwolan.com/writing/ai-distribution-surface-area] ##### [Shishir Mehrotra]() I think distribution changes the ranking. A task can be frequent and easy to verify, yet still fail if the user has to stop working, open a separate chat, explain the context, and paste the result back. We’ve built Superhuman Go to appear alongside a doc, inbox, or tab and bring in an agent based on what the person is doing. [Citation: https://twitter.com/shishirmehrotra/status/2087571494436294761] So I’d add one question to the framework: Can the product recognize the moment of need and bring the right context without making the user ask? Our Knowledge Checker, for example, catches an out-of-date number before someone sends it; the Email Assistant helps sort mail and draft replies in the inbox. Both meet a person inside an existing task. I’d prioritize those timely, contextual interventions before building a broad agent that waits for a prompt. [Citation: https://twitter.com/shishirmehrotra/status/2087571494436294761] ##### [Sam Stone]() I'd put a small prototype before the final surface ranking. A task can be frequent, bounded, and easy to review, then fail because the product lacks the context needed to produce a useful result. At Tome, I found it useful to prototype against real inputs and ask colleagues or users for 10 to 100 structured evaluations, with notes on the failures. That often gave us a clearer next build decision than an early automated score. [Citation: https://medium.com/p/29c9f5c2f321] For each candidate, give evaluators the same user request and available product context. Have them judge whether the output is usable, where it needs correction, and whether the needed correction can happen inside the existing workflow. The first surface should earn its place through those examples before we commit to building the full agent. ##### [Rachel Wolan]() Shishir’s moment-of-need test also changes how I’d research the first surface. When we exposed Webflow capabilities through MCP, people used outside agents for landing pages, bulk CMS work, asset management, and content workflows. That behavior gave our team a much better map of demand than a brainstorm of possible AI features. But the same tool access did not make our in-Designer Assistant ready for broad use; we still needed the review and permission controls enterprise teams had asked for. I’d use observed agent workarounds to choose the job, then test whether the native product can complete it safely. [Citation: https://www.rachelwolan.com/writing/becoming-agentic-part-1] #### [AI skills that outlast the next model release]() **Question:** How can a working professional build AI skills that will last? ##### [Shishir Mehrotra]() I think the lasting skill is designing a workflow that uses AI well. Pick a task you do every week, such as preparing a customer update. Write down the information you check, the judgment you make, and what a good update must include. Then let an agent gather the context and draft a first pass while you review the facts and decide what to send. In an interview with MIT Sloan Management Review, I described a customer memo where sales, product, and support agents each bring relevant information into the document where someone is already working. [Citation: https://sloanreview.mit.edu/article/building-an-ai-platform-for-the-future/] Run that workflow for a month. Keep examples of where the agent helped and where your review caught an error, then change the instructions or source material. The skill you build is knowing how to give an agent useful context, assign it a bounded task, and check its work. I’ve written that knowledge workers are becoming workflow designers as agents take on assignments alongside human teammates. [Citation: https://shishir.substack.com/p/on-practical-ai-adoption] ##### [Mark Hull]() Pick one real task in your job and run a weekly loop: state the user problem, use AI to make a draft or prototype, test the result with someone who would use it, and record where it failed. Repeat with a better prompt, better context, or a different model. That builds judgment alongside tool fluency. In an INSEAD session, I had 25 product and engineering leaders build prototypes and pitch them. The exercise exposed both the speed of the tools and their limits: the builders still had to know which user problem was worth solving. I think three skills travel across model releases: understanding where a user's expectations break, knowing how models fail on your tasks, and specifying inputs and outputs clearly enough to check the work. [Citation: https://www.linkedin.com/posts/markhull_last-week-in-europe-i-kept-hearing-the-same-activity-7396240112029933570-bawP?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAAADuAB5eglJVrF7yjDPm1k2Z8Msy6Qmq8] Keep a small record of the task, the result, the correction, and the time or quality gained. After a month, you'll know which uses actually improve your work. ##### [Aparna Chennapragada]() I’d build three habits around work you already do. 1. Use an approved AI tool on a recurring task, such as preparing for a meeting or drafting a difficult email. Compare its output with what you would have done, and notice where it saves time and where it misses context. AI use is a contact sport; you learn its limits by putting it to work. [Citation: https://open.spotify.com/episode/1N4UL9QZlrX7nLHWcsX3f9] 2. Turn one task into a small prototype. I built a Chrome extension that asks, each time I open a new tab, how AI could help with the task at hand. Building it took about ten minutes with GitHub Copilot. That tiny prompt changed how often I tried AI in my own workflow. [Citation: https://podcasts.apple.com/lb/podcast/microsoft-cpo-if-you-arent-prototyping-with-ai-youre/id1627920305?i=1000708881684] 3. Practice delegation and review. Ask an agent to prepare a research brief for a customer meeting, then check its sources, missing context, and recommendations before using it. The lasting skill is specifying the job, judging the result, and deciding what stays in your hands. [Citation: https://open.spotify.com/episode/1N4UL9QZlrX7nLHWcsX3f9] ##### [Ami Vora]() Shishir’s workflow example gives someone a place to practice. I’d make the learning goal equally concrete, because a working professional’s calendar will otherwise eat it. I’ve found it useful to choose one specific growth goal, define what progress looks like, and reserve an hour or two on late Friday afternoons. [Citation: https://amivora.substack.com/p/4-tips-to-make-space-for-career-growth] For an AI skill, I’d write the goal into a performance review and ask a colleague what capability would actually help the team. In my career planning, I’ve shared growth goals with my manager and team, included deliverables, and checked every few months what I’d learned. [Citation: https://amivora.substack.com/p/5-tips-to-plan-my-career-like-i-plan] That feedback loop matters when AI tools change quickly: in a conversation about AI careers, I said I try to stay curious, learn the underlying patterns, and keep applying general problem-solving skills. [Citation: https://www.youtube.com/watch?v=rRxYCJv3Dyw] ##### [Sachin Rekhi]() A repeatable workflow needs a clear standard for good work. In my Claude Code strategy-critique workflow, I give the agent reference files with my frameworks for evaluating a target audience, value proposition, and differentiation. On a personal finance product strategy, it flagged an audience definition that was too broad and a problem statement that was too shallow. Those critiques were useful because the underlying standards were explicit. [Citation: https://www.sachinrekhi.com/p/claude-code-for-product-managers] For a working PM, I’d start by selecting one deliverable, such as a product strategy or customer interview synthesis. Write down the criteria you use to judge it, run AI against an actual draft, and compare its feedback with customer evidence and your own review. I organize PM work by deliverables across vision, strategy, design, and execution, so there are clear artifacts on which to practice. [Citation: https://www.sachinrekhi.com/p/how-i-use-ai-as-a-product-manager] Keep your standards and examples in portable files so you can apply them to the next tool. Your ability to assess the work should improve each time you revise an artifact. ##### [Ami Vora]() Sachin’s point about writing down standards is useful, and I’d include a way to challenge those standards. I once told my manager I could think of ten reasons a team’s risky idea would fail. She asked me to think of ten reasons it could work. I realized my strength at spotting risks had become a reflex, and I’d stopped running experiments that might change my mind. [Citation: https://amivora.substack.com/p/questioning-my-best-instincts-more] For an AI workflow, I’d write down the criteria for a good result, then ask a colleague to find one case where my criteria miss something. Try that case with the tool and with your usual process. You learn more about the tool and about the judgment you brought to the task. ##### [Mark Hull]() A skill can feel useful while making the work worse downstream. So I would add an outcome check to the practice loop: for each AI-assisted task, track how much rework it creates, whether the final result meets the user's need, and how much time the review takes. A faster first draft counts only if the whole task improves. We studied 3,407 engineers over 30 days in open-source projects and found that heavier AI use was associated with more output and higher code acceptance rates. The same study found a slight tilt toward smaller tasks, and the results cannot establish cause or tell you whether the pattern holds in a different workplace. Builders should compare similar tasks over time and keep review standards steady. That teaches you where AI actually helps your craft, instead of rewarding sheer volume of use. [Citation: https://blog.exceeds.ai/ai-adoption-in-engineering-teams-oct-2025/] ##### [Shishir Mehrotra]() A useful next exercise is to describe a task in terms of intent rather than copying every old rule into a new tool. When I moved my email, I had nearly 800 filters. I replaced them with four Superhuman auto-label instructions that described the kinds of messages I wanted grouped. [Citation: https://youtube.com/watch?v=4v8iPUV5agE] For a working professional, the practice is to write a short instruction for a recurring decision, review which cases the AI sorted incorrectly, and refine the instruction. That builds a skill you can carry across products: explaining your judgment clearly enough for a system to apply it and knowing when to override the result. ##### [Aparna Chennapragada]() One more skill worth practicing is product judgment in a natural-language interface. If you build an AI helper for your team, prototype the conversation itself: What does it ask before starting? When does it show its plan? How does the user correct it? In my conversation with Lenny, I argued that conversational products still need design, including the timing and usefulness of follow-up questions. [Citation: https://podcasts.apple.com/lb/podcast/microsoft-cpo-if-you-arent-prototyping-with-ai-youre/id1627920305?i=1000708881684] A useful exercise: take the workflow Shishir described and let a colleague try the prototype without instructions. Watch where they hesitate, change the interaction, and try again. That tests whether your AI skill improves someone else's work, beyond your own ability to write prompts. #### [What AI changes in a product manager's job, and what stays]() **Question:** How is AI changing the product manager's job? ##### [Sachin Rekhi]() AI is giving PMs back the craft of product management. In 2016 I defined the PM role at Wharton as driving the vision, strategy, design, and execution of a product. That definition still holds. Over the following decade, though, the job drifted toward coordination: we kept adding specialized roles, and the PM paid the tax of keeping all of them aligned. AI now speeds up a lot of that coordination or takes it over completely, so PMs can spend their time building solutions to customer problems again [Citation: https://www.sachinrekhi.com/p/the-art-of-product-management-in]. The bottleneck has moved too. Andrew Ng argues that product management is now the constraint in product development, because Claude Code, Cursor, and Codex have sped up delivery so much. I'm already seeing PMs who can't keep up with their engineers. That makes discovery the place to invest. Feedback rivers like Enterpret and Reforge Insights pull in App Store reviews, Zendesk tickets, and Reddit threads, then pick out the themes continuously [Citation: https://www.sachinrekhi.com/p/ai-powered-customer-discovery]. My own deliverables have changed shape. I write shorter specs and build more prototypes, because prototyping is where AI gives me the most leverage [Citation: https://twitter.com/sachinrekhi/status/2054939996315132319]. I built my course's landing page with Claude Code and wrote zero lines of code myself [Citation: https://twitter.com/sachinrekhi/status/2033559051582484877]. AI is still terrible at product strategy. I've tried it every way I can think of, and it has never produced a differentiated strategy that could win a market. It also can't match world-class interaction designers [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7440049561181614080/]. Gokul Rajaram calls the resulting role a "product builder": someone technical enough to inspect the work, product-minded enough to pick the right problem, and with enough taste to reject mediocre output [Citation: https://twitter.com/sachinrekhi/status/2060013259101716562]. Judgment is the skill that matters most now that AI has made output cheap. ##### [Rachel Wolan]() The PM job is splitting in two, and the hiring data makes that visible. I analyzed 1,858 open Product, Design, and FDE (forward deployed engineer) roles at the Lenny 100 AI-native companies. On paper the product org looks 50% smaller: 1 PM for every 13.5 engineering openings. Count FDEs and the ratio goes back to about 1 : 8.7. Outbound PM work, meaning talking to customers and translating their requirements, has moved to FDEs in GTM. The PMs who stay in product now spend their days on evals, edge cases, composability, and understanding model behavior. The titles look familiar, but the day-to-day job has changed. One finding worries me: 89% of those Product roles require 5+ years of experience, so there's almost no path in for junior PMs. The second change is team shape. A feature used to need 5-8 engineers, six standing meetings a week, and six months. At Webflow I've been moving to a different lifecycle. Two people validate the wedge, a small crew hardens it for production, and then one owner plus agents keeps it running. That owner ships about 80% of enhancements alone and sends only the hard 20% back to engineering. The metric becomes days from idea to revenue [Citation: https://www.rachelwolan.com/toolkit/builder-garden]. Boris Cherny's five archetypes from the Claude Code team (prototyper, builder, sweeper, grower, maintainer) fit what I see: product roles organized by stage of the product more than by function [Citation: https://twitter.com/rachelwolan/status/2071438746353062036]. The third change is that PMs build. Prototyping in Cursor or Claude Code beats writing a spec, because working code shows engineers exactly what you mean [Citation: https://www.rachelwolan.com/toolkit/cursor/for-product-managers]. I still code every day. I can't credibly ask my PMs to work this way if I'm not doing it myself. ##### [Aparna Chennapragada]() Sachin, I'd complicate "AI gives PMs back the craft." Coordination with people shrinks, and a new kind of coordination takes its place: supervising agents. Paul Graham's split between the maker's schedule and the manager's schedule is useful here. Working through agents pulls every maker, PMs included, into the manager's mode. You review outputs, decide when to step in, and rarely get long stretches inside the work. Responsibility stays high while your direct control over execution drops, and Karasek's Demand–Control research links that combination to stress. (I found this out at 1am, "one prompt away" from a working app.) Rachel's "one owner plus agents" model makes this concrete: that owner is effectively a manager of a 24/7 team that never stops producing output to review. So part of the new PM job is designing calmer human-AI interfaces, for our users and for ourselves. [Citation: https://aparnacd.substack.com/p/every-maker-is-now-a-manager-of-ai] I agree the bottleneck has moved, and the cadence has become uneven. Time to first demo collapsed, while time to full deployment got longer and the bar for scale went up. More prototypes raise the floor and the ceiling together, which makes "which of these 50 demos deserves to ship" the PM's main call. [Citation: https://youtube.com/watch?v=HbbfXAWcuUo] The skill I'd add to the list is betting against the model curve. I call one failure mode the Six Finger Trap: a team spends real engineering and product effort patching a model limitation that gets fixed upstream a few months later, the way garbled text and six-fingered hands disappeared from image models. AI PMs now have to run the wait calculation on every roadmap item, design so models can be swapped in and out, and treat today's workarounds as tomorrow's tech debt. When the model can generate a whole creative suite's output in one shot, the PM's job is deciding what the product adds on top of the model. [Citation: https://aparnacd.substack.com/p/beware-the-six-finger-trap] ##### [Ami Vora]() Sachin's "product builder" matches what I've been seeing, and I'd put it in historical terms: the PM job is going back to what it looked like about 15 years ago, when I was earlier in my career. We had so few people that if you believed in something, you just got it done without asking whose job it was. Then engineering, design, and PM hardened their boundaries, and even inside PM you could pick a lane. AI tools are dissolving those lines again, so the PM I'd hire now is the utility player who says "I have an idea, I'll run a prototype, rough out a market analysis, mock up the go-to-market, and see where I get stuck." [Citation: https://www.youtube.com/watch?v=Vp2BsJNVFqQ] I break product work into three jobs: recognize the customer problem, structure a solution, and execute. [Citation: https://amivora.substack.com/p/the-3-key-jobs-of-a-product-manager] AI makes execution much cheaper, which is why Sachin's PMs can't keep up with their engineers. The value moves to the first two jobs: building conviction that this is the most important problem right now, and making dozens of small framing calls consistently. That's why Rachel's FDE finding makes me a little nervous. If customer conversations move to FDEs in GTM, the PM keeps the evals and loses direct contact with the job that's gaining value. The best way I've found to recognize problems is to stay close enough to customers that you notice patterns naturally, so PMs need a tight loop with FDEs or time in those conversations themselves. On Rachel's junior-PM worry: I learned the job by flowing to wherever the water was lowest, which meant chasing bugs, checking retention dashboards, and writing marketing copy. If agents absorb those leftovers, we need to give new PMs a deliberate way to get those reps. The part I'd add to "judgment" is that it has to come with letting go of being right. A lot of the skills I spent years building probably won't matter as much, and the more senior you are, the harder it is to say "I don't know" when you've been trained to have the answer. [Citation: https://www.youtube.com/watch?v=rRxYCJv3Dyw] Being right was my identity and my job for a long time. When the tools reset every week, you have to assume you're wrong a lot, or you have no reason to learn anything new. The PMs I see doing well treat curiosity as the job: they pick one area, like agents, learn it down to the bones, and then move to the next. ##### [Shishir Mehrotra]() I agree with Aparna that every maker becomes a manager, and I'd add two details. It happens at the very start of a career, and the agent team is missing what makes human teams useful. When my daughter asked whether to major in CS, I told her she'd show up to her first job already in middle management, handed a team of agents that have read the whole codebase and know every language. They never argue with each other, though, and they don't come up with new ideas. A PM has to supply the conflict and the new ideas that a human team used to generate on its own. [Citation: https://youtube.com/watch?v=INEUCaJHDY0] Sachin's coordination tax is measurable. One Coda customer found their PMs spent about 30% of their time collecting what had already happened and rewriting it into status updates for different forums. We built an agentic column that reads tickets, GitHub, Slack, and email for each project and writes a five-bullet summary, so the PM gets that time back for defining the right product. [Citation: https://open.spotify.com/episode/2laxLUtTqk4MsZm0klcotd] Ami and Rachel are right that the grunt work was where junior PMs built instincts. If a frontier model does that work, judgment that used to come by osmosis has to be built on purpose, and I'd make the practice concrete. The skill I care most about is looking at 10 AI outputs, picking the one that is actually good, and explaining why. That's taste, and you get it by studying great work closely and forming opinions. Prompting has little to do with it. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7495912137651826688/] ##### [Ami Vora]() Aparna's point about calmer interfaces applies to customers too, and I think it's one of the most underrated parts of the AI PM job. Everyone is more tired than we think. Most people don't want to learn a new thing. They want relief, a place where stuff just works for them. [Citation: https://www.youtube.com/watch?v=Vp2BsJNVFqQ] My read on why chat took off is that it wrapped very unpredictable output in an interaction everyone already knew. What you get back can surprise you, but how you ask is completely familiar. So the design rule I'd give PMs is to keep the interaction pattern boring and predictable, and let the output carry the magic. That's harder than it sounds, because AI breaks a classic UX rule. With normal software you design around the main task. With an open text box, you often don't know what the main task is. I doubt the frontier in five years is me typing into a chatbot. My guess is it's something less interruptive than pulling out your phone, typing, and waiting, something that lives alongside your conversation. My kids grew up talking to screens and expecting them to answer, and my four-year-old thought a driverless Waymo was just normal. Watching how people interact with a product when they don't have to learn it is one of the best signals a PM can get right now. ##### Question > Rachel, Aparna argues that your one-owner-plus-agents model turns that owner into a manager of a team that produces output around the clock, with high responsibility and low control. At Webflow, how does the owner decide which agent output to review closely and which to let ship, and what broke when that owner got overloaded? A public example would help, such as one of the 12 production agents. ##### Question > Sachin, Rachel's hiring data shows customer conversations moving from PMs to forward deployed engineers in GTM. Ami worries that this leaves PMs with evals while they lose contact with problem recognition, the job that is gaining value. You argue discovery is where PMs should invest, using feedback rivers like Enterpret. Can feedback rivers plus FDE notes replace a PM's own customer conversations, or does a PM still need direct time with customers? What setup would you recommend, and what public example supports it? ##### [Sachin Rekhi]() No. Feedback rivers plus FDE notes can't replace a PM's own customer conversations. I use AI research tools to add to my interviews, and I still do as many personal interviews as I always have [Citation: https://www.sachinrekhi.com/p/ai-powered-customer-discovery]. What AI changes is coverage. At LinkedIn, our research budget only covered major redesigns and brand new features, so we shipped a lot of smaller features on gut feel and waited to see how customers reacted. Today I can send 10 AI-moderated interviews out tonight and have the synthesis by morning, so those smaller features get feedback too. Here's the setup I'd recommend for a team where FDEs own the outbound conversations: 1. Put FDE call notes and transcripts into the same feedback river as App Store reviews, Zendesk tickets, and G2 reviews, using a tool like Enterpret or Reforge Insights. 2. The PM checks it daily for trending themes and, after each fix ships, watches whether complaint volume actually drops. 3. For every theme, the PM opens the exact customer verbatims underneath it and stops relying on AI's summary. 4. When a theme matters, the PM clicks through to the affected customers and contacts 20 of them directly. The feedback river tells you who to call, and the PM makes the calls. That last step is where the PM keeps problem recognition, which is Ami's concern. Since I started using these tools, I spend much less time producing research and much more time mining it. I ask the LLM follow-up questions against the raw transcripts, then watch the specific interview videos where the strongest verbatims came from. I never let AI decide what the product should do based on the research. That stays my job [Citation: https://twitter.com/sachinrekhi/status/2034646172959395900]. Rachel's split can still work if the FDEs feed the river and PMs keep running their own targeted interviews. I'd worry about any setup where the PM only ever reads summaries. #### [The operations work AI agents can take on now, and what still needs a person]() **Question:** Which operations work can AI agents handle today? ##### [Rachel Wolan]() So, my answer: agents reliably handle ops work today when the job is bounded and the action is reversible, and a human approves anything that leaves the building. Open-ended "go run this function" agents still break. My own email triage is the clearest example. The agent clears calendar notifications, marketing, and system messages. It pins what to keep and drafts replies for the people I actually owe. Then it proposes something like "archive these 40," and I say yes or no before anything runs. I don't let it send email on my behalf. It still makes small mistakes, like flagging a meeting with no link one morning, and that's fine because I'm the approval step. [Citation: https://www.youtube.com/watch?v=82zF4BUqxkA] We learned the same lesson at Webflow, and it cost us. In December 2024 we pulled AIDA, an open-ended chatbot that could touch anything in the Designer. Six weeks later a different team shipped the AI Site Builder, which did one thing: a multi-page, brand-aware site in under two minutes. What worked was a series of bounded agents, each well-scoped and trustable, each inheriting the customer's brand context. [Citation: https://www.rachelwolan.com/writing/becoming-agentic-part-1] That gives me a working map. Reliable now: - Triage and sorting: inbox, tickets, feedback - First drafts against a known template - Research synthesis, like analyzing 50 interview transcripts [Citation: https://www.rachelwolan.com/toolkit/claude-code] - Status rollups that keep specs, decisions, and progress connected Not yet, without a lot of scaffolding: - Anything irreversible or customer-facing that runs with no approval step - Work that depends on context nobody wrote down. I hear "it's fast, but it's not quite right" from marketers constantly, and the cause is missing brand and messaging context. A better model won't fix that. Most of the ops investment goes into that second bucket: brand context, workflows, governance, and feedback loops written down so agents can use them. ##### [Dennis Yang]() Customer support is the clearest proof point that agents handle real operations work today. At Chime, 70% of support interactions are now AI-powered, cost to serve members is down 60%, and member satisfaction has more than doubled. Those numbers are from Part I of our AI Across Chime series. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7415601009571352576/] The work that automates well has a known answer that lives in a knowledge base. Our chatbot and voice bot take a member's question, find the right policy or how-to content, and explain it in plain language. In fintech that covers a big share of volume: where is my deposit, how do I dispute a charge, why was a card declined. I'd treat the remaining 30% as the map of what agents don't reliably handle yet. That bucket is mostly cases where the answer depends on judgment or on an action with money attached, such as fraud disputes, account exceptions, and anyone already upset after a bad experience. Agents can gather context and draft the response there. A human still owns the decision. Two things made the 70% possible, and both are operations investments more than model choices: ✅ Clean, current knowledge content. The bot is only as accurate as the articles it retrieves from. ✅ Fast, graceful handoff to a person with full context, so a member never has to repeat themselves. Internally the same pattern holds. 90% of Chimers use AI weekly, mostly for drafting, research, and engineering work where a person reviews the output before it ships. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7415601009571352576/] So my working rule: agents own high-volume, well-documented, reversible tasks today. Anything irreversible or regulated gets an agent as the assistant and a human as the approver. ##### [Sam Stone]() Rachel's approval step is necessary, but it only holds up if checking the output costs much less than doing the work. If a lawyer has to re-verify every fact an agent pulled from 500 pages of records, the agent hasn't saved any time. So I'd draw the map around verification cost. Agents work well today on high-volume, document-heavy ops work where a trained reviewer can check the output quickly. They don't yet work on rare, high-stakes cases. At EvenUp, a personal injury demand package means hundreds of pages of medical records and bills, turned into a structured legal argument. That used to take days of paralegal time. With Express Demands, legal professionals generate and edit a draft in minutes. The lawyer still owns the final document, and that review step is the reason it works. Extraction, summarization, chronologies and first drafts are reliable now. The final judgment call about what a case is worth stays with the human. Agents break down in two places: - Beyond the "confidence frontier." The Harvard/BCG study showed work quality dropping when people used AI past what it could reliably do and didn't check the output closely enough. For ops work, an agent that says "I don't know" is worth more than one that always produces an answer. [Citation: https://medium.com/p/26516d928566] - Edge cases. Karpathy's "imbalanced class" point still holds: models handle the straight-highway cases and struggle with the rare ones that matter most. [Citation: https://medium.com/p/ab17c4ad1a7a] At Opendoor, when we rolled out a unified pricing model, we invested in teaching operators why it made the recommendations it did. That way they flagged recommendations that were actually wrong and didn't flood us with false alarms when the model's output changed sharply from one day to the next. [Citation: https://medium.com/p/4fe9573f83b3] Dennis's 30% handoff bucket needs the same thing: reviewers who understand where the agent is reliable, so their escalations improve it. ##### [Shishir Mehrotra]() I agree with Rachel's map, and her point about unwritten context explains why summarization is the most reliable ops task today. It became the most common use of AI in Coda: people point it at meeting recordings, notes, email threads or what's happening in Salesforce and ask for a summary. Everything the agent needs sits in the source material, so this generation of models does that task very well. [Citation: https://youtube.com/watch?v=QjDBCDN5JEI] I judge any ops task by treating the agent like a new hire. What does it know? That can be global facts like English grammar, team facts like the support knowledge base or wiki, or personal facts like your inbox. What skills does it have? Is the job short-running or long-running? [Citation: https://podcasts.apple.com/us/podcast/shishir-mehrotra-on-building-tools-creators-love/id1510985491?i=1000721488859] [Citation: https://youtube.com/watch?v=kqdaL-3yzjw] Short jobs that run on knowledge the agent can actually reach work today. Dennis's 70% at Chime is the team-facts case: the answer already lives in the support knowledge base. Personalized outreach is a good example. Yaniv Bernstein described a modern mail merge to me on his podcast: find the 30 people I met in New York last time, tell them I'm coming back, and mention what we last discussed with each one. That works because the agent can read the history. [Citation: https://open.spotify.com/episode/2laxLUtTqk4MsZm0klcotd] Long-running jobs with ambiguous goals still fail, mostly because the agent is missing context. So the human job moves to review. I told my daughter that AI turns individual contributors into managers much earlier in their careers, because they spend their days reviewing a large volume of agent output. Rachel's approval step is one version of that, and Sam is right that it only pays off when checking is fast. That is why teams need new review rituals designed for the volume of agent work, the same way managers built rituals for reviewing a team's output. [Citation: https://youtube.com/watch?v=INEUCaJHDY0] ##### [Aparna Chennapragada]() I'd split "bounded" into two axes, because they fail for different reasons. Breadth depends on model capability. Autonomy depends on tool chaining and on whether the chain has checks built in [Citation: https://www.emergingtechbrew.com/stories/2025/03/14/microsoft-workplace-ai-aparna-chennapragada]. Read-heavy work can be very broad and still be safe to hand off today. I once asked our Researcher agent to pull together the views of everyone in an upcoming leadership meeting and suggest how I should pitch. It raised angles I hadn't considered, and nothing it did was irreversible [Citation: https://youtube.com/watch?v=HbbfXAWcuUo]. A narrow task becomes risky once the agent writes to a system of record with no check in place. Long-running, action-taking agents are the 2026 capability, and they still need supervision [Citation: https://aparnacd.substack.com/p/2026-year-of-deep-work]. Shishir's point about turning ICs into managers is real, and the cost shows up as fatigue. I've been up at 1am going back and forth with a CLI on an app I thought was one prompt from done. Agents run 24/7, so review never naturally stops [Citation: https://aparnacd.substack.com/p/every-maker-is-now-a-manager-of-ai]. Rachel's approve-the-archive step works at 40 emails. When drafts, PRs and tickets grow tenfold, the approver becomes the bottleneck. Code review already can't depend on a human reading every line. Sam is right that verification cost decides whether the approval step holds up. Product teams can lower that cost with proof-of-work UX. The agent should show what it checked, what it changed, and where it's unsure, so a human can approve a batch in seconds without redoing the work. Dennis's handoff with full context applies the same idea to support: the human gets the agent's reasoning along with the case. That review surface is where I'd put the next dollar of ops investment, ahead of more agent autonomy. ##### [Dennis Yang]() Sam's verification-cost test explains why support automated first. Checking a support answer is cheap: either it matches the help article or it doesn't. That also lets me push back on Rachel's "customer-facing with no approval step" bucket. At Chime the 70% of AI-powered support interactions go straight to members with no human in between, because the answers are documented and the action is information only. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7415601009571352576/] The approval step belongs where money moves or a policy exception gets made. The line between the two buckets also moves every week, and the transcripts move it. I spent four years at Dashbot building analytics on billions of chatbot and voice messages for companies like Intuit and GoDaddy. The most useful report was always the list of messages the bot couldn't handle, grouped by topic. Each cluster pointed to one of three fixes: 1. A missing or stale knowledge article, which content ops can fix in a day. 2. A real action the bot can't take yet, like reissuing a card, which becomes an engineering ticket. 3. A judgment call that should stay with a human, which becomes a cleaner handoff rule. So the human-handled 30% is also your labeled backlog. Reviewers who tag why they took over a case are doing what Sam described at Opendoor: they teach the system where its reliable zone ends. Aparna's proof-of-work review surface should include a one-click "why I took over" reason for exactly this purpose. Most teams under-invest in that tagging, and it's cheaper than any model upgrade. ##### [Shishir Mehrotra]() Aparna's fatigue point is real, and I think the fix is a cadence. Agents can run 24/7, but the human review should happen on a schedule, the same way a manager holds a weekly one-on-one instead of watching over someone's shoulder all day. Superhuman Mail users show what that cadence looks like. One user told us they run their Superhuman workflow with Claude twice a day. The agent triages through Split Inbox and drafts replies in each person's voice, and the user reviews the batch in one sitting. Another says their inbox automation gets them to Inbox Zero 2 to 3 times faster. In both cases the agent did the volume work between check-ins, and the human made the calls in a fixed window. The other factor is where the review happens. If the approver has to "go use the AI tool," every approval adds a step: open another app, find the output, copy it back. That AI detour is what turns review into the bottleneck Aparna describes. Review is cheap when the agent's proposals show up inside the inbox, doc or ticket the person already works in, which is the same idea as her proof-of-work UX. For ops leaders, I'd turn this into one design question for every agent: when does a human review its work, and where? If nobody can answer both, the agent will either run unchecked or bury someone in approvals. ##### [Sam Stone]() Dennis's takeover tags and Aparna's reviewer bottleneck fit together. When review volume grows tenfold, the reviewer's edit history is what tells you where you can safely review less. In legal drafting, we'd track which sections of a draft lawyers rewrite. A section type that gets accepted almost untouched across hundreds of demands can move from line-by-line review to spot checks. A section type that gets rewritten every time stays under full review, even if each draft looks fine on its own, because the agent is past its confidence frontier there. Implicit signals like saves and edits aren't perfect measures of quality, but in aggregate they're decent proxies, and you get them for free. [Citation: https://medium.com/p/26516d928566] One caution on Shishir's "team facts" and Dennis's transcripts. Ops histories also record a team's bad habits. If a support team has historically been rude to a certain type of customer, an agent built on those conversations will learn to do the same. General model safety training targets societal bias and won't catch institutional bias like that. Before you point an agent at years of tickets or case files, decide which past behavior you want it to repeat and which you want to drop. [Citation: https://medium.com/p/e19ded4bde88] ##### [Aparna Chennapragada]() Dennis's failure-cluster report points to a category we've left off the map: ops work nobody does today because no one has the hours. Every list so far starts from tasks humans already do and asks which ones the agent can take over. Knowledge work has always been limited by scarce cognitive energy, so a lot of useful work never gets staffed [Citation: https://www.emergingtechbrew.com/stories/2025/03/14/microsoft-workplace-ai-aparna-chennapragada]. Reading every escalated ticket, every churned customer's last five emails, or every vendor contract renewal clause are good examples. Agents handle that well right now. The work is read-only, it runs on source material the agent can reach, and the baseline is zero, so an 85%-right pass still beats what existed before. Teams also set the bar too low. If you hand an agent coffee runs, you get coffee-run value. Ask it to go through 300 chat messages, 110 emails, 20 hours of meetings and 7 docs, then return a five-minute briefing with five drafted replies. That is a realistic ask today, and the drafts still go through a human before they send. Dennis's one-click "why I took over" tag fits naturally inside the review surface. I'd use it to answer a second question too: which of those takeovers came from net-new work the agent surfaced rather than work it did badly. That ratio shows whether you're using agents to find new problems or only to cut the cost of old ones. ##### Question > Rachel, Dennis pushed back on your rule that customer-facing work needs an approval step. Chime's bot sends documented, information-only answers straight to members at 70% of support volume, and he puts approval only where money moves or a policy exception gets made. Aparna adds that approving every item breaks once volume grows tenfold. Would you revise your map to allow unattended customer-facing answers when the source content is written down? And what made AIDA at Webflow unsafe to run unattended when the AI Site Builder was safe? ##### [Rachel Wolan]() Yes, I'd revise the map. Customer-facing was the wrong line. Dennis's information-only answers drawn from documented content can go straight to customers with no approval step. The approval step belongs on irreversible actions, money, and exceptions, and on anything where the context still lives in someone's head. That last one is my marketing case: content ships fast but off-brand because the messaging framework was never written down. AIDA and the AI Site Builder show the difference in scope. AIDA was an open-ended chatbot that could touch anything in the Designer, so every request could land on a customer's live, existing work, and nobody could predict the blast radius. The AI Site Builder did one thing: you typed a prompt and got a multi-page, brand-aware site in under two minutes. It started from a blank site, so there was nothing to break. We kept that pattern as the agents grew. When a customer asks the AI Assistant for a pricing calculator or a job board, it generates an AI Code Component bounded inside its own island on the canvas, so the agent can't reach the rest of the site. [Citation: https://www.rachelwolan.com/writing/becoming-agentic-part-1] The model behind it wasn't the issue. A bounded write surface is what made it safe to leave unattended. On Aparna's 10x volume problem, I don't approve 40 emails one by one in my triage. I told the agent once which categories to clear: calendar notifications, marketing, system messages. My approval is a single yes on a batch it sorted using rules I wrote down. [Citation: https://www.youtube.com/watch?v=82zF4BUqxkA] Sam's edit history tells you which categories are ready for less review. Writing the rule down is what actually moves them there, because the next agent and the next reviewer inherit it. #### [What AI changes for services firms: the work, the review and the price]() **Question:** How will AI change services businesses? ##### [Dennis Yang]() Service businesses will change first in the work they do for each customer, then in what customers can ask them to do. At Chime, I shared that 70% of support interactions are AI-powered, cost to serve members is down 60%, and member satisfaction has more than doubled. Those are company-wide figures, so I wouldn't attribute every change to support AI alone. They do show why cost and customer experience need to be measured together. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7415601009571352576/] The next step is work that starts before a customer opens a support ticket. I've described a financial agent that could eventually negotiate bills and manage finances for a member, though that is a direction we're exploring rather than a shipped outcome. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7457281412292071424/] A services business should measure whether AI completes the customer's task accurately, and whether the customer still has a clear way to correct it. ##### [Ami Vora]() I think AI will let services businesses do more of the preparation and follow-through for each customer, while making the quality of the actual service easier to judge. The test is whether the customer gets what they came for. At Faire, I visited a retailer who also ran a hospital gift shop between her duties as a nursing director. She had about five minutes at a time, so I kept asking whether what we built was simple enough for her to use in that window. [Citation: https://amivora.substack.com/p/unlocking-simplicity-customer-empathy] For a service business, I’d use that same test on AI: can it gather context, prepare a useful next step, and save the customer time without making them manage a confusing new process? Keep talking to customers directly, because their frustrations tell you which parts of the service need attention. ##### [Sam Stone]() At EvenUp, we launched Express Demands so legal professionals can draft and edit demand packages with AI in minutes. We also launched tools for case analysis and client-firm communication. That points to a change in the unit of work for a services business: a firm can handle more of a case through a connected workflow, while its professionals spend their time checking facts, exercising judgment, and working with clients. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7289747436778962944/] [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7349289496837836800/] The quality bar rises as that workflow takes on more work. In high-stakes services, a plausible false answer can cost more time than it saves because someone has to find and fix the error. I've argued for product designs that make the system's limits clear and let users see and correct its output. Firms will need to measure the quality of completed work alongside the time saved. [Citation: https://medium.com/p/26516d928566] ##### [Rachel Wolan]() At Webflow, our AI Site Builder could generate a multi-page site from a prompt, but customers still needed to edit and maintain it. For services businesses, the hard change is who owns the review, the client context, and the production handoff. In research, a partner agency managing hundreds of enterprise sites worried that an unreviewed AI update could damage a client's live website. We learned to build roles, permissions, brand rules, and review into the workflow. [Citation: https://www.rachelwolan.com/writing/becoming-agentic-part-1] So I expect agencies and other services firms to sell more ongoing judgment and accountable delivery, with AI doing bounded pieces of the work. If I were running one, I'd start with a repeatable task where the client can inspect the output before it goes live, then measure revision time and errors as carefully as time saved. ##### [Ami Vora]() Sam’s point about checking the finished work makes me think the next question is who decides what “quality” means. When I worked on business tools, I had a list of UI improvements I wanted to make, but customers wanted more sales and new customers. We put the team’s time into products that helped them find those customers, and they loved it. [Citation: https://amivora.substack.com/p/making-the-tradeoff-between-speed] AI can make a services team faster at producing polished deliverables. I’d ask customers which result they actually value, then measure whether the faster process produces more of that result. In a firm selling advice, that might mean checking whether the client can act on the recommendation, alongside the time it took to write it. ##### [Shishir Mehrotra]() Services firms can use AI to handle more client work with the same team, which puts pressure on the billable-hour model. The firm still needs people who understand the client's problem, check the work, and own the outcome. Its repeatable research, drafting, and coordination can increasingly run through shared workflows. I learned the scaling constraint firsthand at Centrata. We sold multimillion-dollar software deals, but customers were really buying our services; the business grew like a consulting firm because each deployment took so much hands-on work. [Citation: https://alejandrocremades.com/shishir-mehrota/] For services firms, the opportunity is to turn the steps that used to require a specialist on every engagement into a shared workflow, then spend human time on the client-specific decisions. I would watch the amount of setup each new client requires: if it stays flat, the firm still scales one specialist at a time. ##### Question > Shishir, Rachel expects agencies to sell ongoing judgment and accountable delivery, even as AI does more production work. How should a services firm price that work if AI cuts drafting time but adds review and correction costs? Please ground your answer in your public work and cite a public source. ##### [Shishir Mehrotra]() Rachel's review costs belong in the price of an accountable deliverable. I would quote a defined scope and acceptance standard, with an ongoing fee for maintenance or monitoring where the client needs it. Internally, track the full cost per accepted deliverable: AI use, human review, revisions, and client-specific setup. If AI cuts drafting time but review takes longer, the margin calculation will expose it before a lower price locks in. At Centrata, our multimillion-dollar software deals became service engagements because each customer deployment required so much hands-on work. That taught me how easily a seemingly repeatable offering can inherit consulting economics. [Citation: https://alejandrocremades.com/shishir-mehrota/] I would price a firm's reliable result and set explicit limits on scope, while checking that review and correction costs fall as its workflow improves. The client should pay for the work they can use, with the firm responsible for the cost of getting it right. ##### Question > Sam, Shishir proposes pricing an accepted deliverable, with the firm covering review and correction. For an AI-assisted legal demand package, what acceptance standard could a firm promise before the case outcome is known? Please ground your answer in public EvenUp work and cite a public source. ##### [Sam Stone]() For an AI-assisted demand package, I would define acceptance around what the firm can inspect before sending it: the relevant medical bills and records are accounted for, material facts match the source documents, the requested amount and supporting calculations can be checked, and a legal professional has reviewed and approved the final package. The standard should include a correction window for missed or misread records. It cannot promise a settlement amount before the other side responds. That is the product boundary I have in mind at EvenUp. Express Demands lets legal professionals draft and edit demand packages in minutes, and our later launch included Medical Bill Summaries and AI Drafts Suite. Speed makes the first draft cheaper; the firm still has to deliver a package a professional can defend. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7289747436778962944/] [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7328920992762748929/] #### [Why high AI usage can leave business results unchanged]() **Question:** Why doesn't AI adoption show up in business results more often? ##### [Sachin Rekhi]() AI adoption often gets measured at the tool level: seats, prompts, and hours saved. The business result depends on what the team does with the extra capacity. I see this in product work: AI can summarize customer interviews and generate prototypes quickly, yet a team can still ship features that miss its customers or business goals. I’ve argued that product strategy needs a specific customer, problem, way to win, and business model. Without those choices, faster output just feeds the feature factory. [Citation: https://www.sachinrekhi.com/p/how-i-use-ai-as-a-product-manager] I’d pick one workflow, set a baseline for its time and quality, then track the downstream decision and outcome. For interview analysis, that means checking whether the findings change a roadmap decision and whether the resulting release moves the team’s goal. A good AI workflow improves the decision, not merely the speed of producing the document. [Citation: https://www.sachinrekhi.com/p/claude-code-for-product-managers] ##### [Dennis Yang]() Usage is an input. The business result comes when a team changes a specific workflow and measures its outcome. At Chime, we reported 90% weekly AI use by employees, but the more useful measures are in member support: 70% of interactions are AI-powered, cost to serve is down 60%, and member satisfaction has more than doubled. Those figures show why I would track cost and member experience alongside adoption, rather than treat weekly use as the finish line. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7415601009571352576/] Ownership matters too. In a public Chime case study I shared, engineering built the pipelines while domain experts owned ground truth, evals, and prompt performance. The financial-crimes example reported a 40% efficiency gain and 99%+ quality scores. That setup lets the people who know the work improve the AI without waiting for every change to move through engineering. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7417267922932326401/] ##### [Mark Hull]() At Exceeds, we gave our 16-person team access to several AI tools for months; usage stayed concentrated with my co-founder and me. After we standardized on Claude Code and onboarded everyone, our internal measures showed 99% AI usage, 0.65-day cycle time, and a 1.89x average productivity lift. Then the costs jumped, and review depth fell to 44.8 against our healthy threshold of 60. We had sped up code production while creating a quality risk downstream. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7422383392723267585/] So I'd measure each use case through the full workflow: time to ship, review and rework, defects, customer outcome, and tool cost. Compare teams or phased rollouts against a baseline, because a faster PR or a higher usage rate alone cannot establish business ROI. Our public 30-day analysis of 3,407 engineers found higher output and acceptance rates among heavier AI users, but it was correlational and may reflect task mix. [Citation: https://blog.exceeds.ai/ai-adoption-in-engineering-teams-oct-2025/] ##### [Sachin Rekhi]() Dennis’s point about domain experts owning evals matters because a fast answer can be confidently wrong. I ran into this with competitor pricing: general AI research kept returning stale prices. I changed the workflow so a browser agent visits each current pricing page, extracts the plans, and produces a comparison that I hand-check. It takes longer, but the output is useful for a pricing decision. [Citation: https://www.sachinrekhi.com/p/claude-code-for-product-managers] For a business team, I’d put that quality check directly into the workflow. Measure the time spent finding and correcting errors alongside time saved, then ask whether the verified output changed a decision. Otherwise a usage dashboard can show progress while the team quietly absorbs the rework. ##### [Shishir Mehrotra]() The friction often sits inside the work itself. A person opens a separate chat, explains the task, and copies the result back into a document or inbox. Each extra step reduces how often the tool helps with the work that drives results. We have been building Superhuman Go to bring assistance into the doc, inbox, or tab where the person is already working. [Citation: https://twitter.com/shishirmehrotra/status/2087571494436294761] The workflow change needs a business measure. Smartsheet reported 283% ROI from Grammarly and about 1,600 employee hours saved annually, with that time reinvested in higher-value work. Western Governors University reported roughly 146,000 hours saved and $2.5 million in annual savings after rolling Grammarly out to about 70% of the institution. Those are customer-reported outcomes; the useful management question is which repeated task changed, where the saved time went, and whether that change persisted. [Citation: https://shishir.substack.com/p/on-practical-ai-adoption] ##### [Mark Hull]() The next step for our team is review behavior. With 80% AI code contribution and 323-line average PRs, only 6% of team activity went to reviews. I would put the AI-assisted review steps inside the PR workflow and check whether review depth recovers before calling our faster cycle time a win. That's where managers can copy the practices of builders getting good results, instead of asking everyone to use a tool more often. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7422383392723267585/] [Citation: https://www.linkedin.com/posts/markhull_huge-congrats-to-atlassian-and-the-dx-teams-activity-7374459020524277760-w1tW] ##### [Dennis Yang]() There is another measurement gap in conversational AI: an interaction can count as successful while the member still has to ask again. At Dashbot, I became interested in conversational data because users tell you what they need in their own words; event counts make you infer their intent. [Citation: https://medium.com/chat-bots/dashbots-dennis-yang-why-conversational-data-captured-his-attention-and-why-you-should-care-too-34f03996a001] For a support bot, I would sample the actual conversations behind a rising automation rate, label the requests it handled and the ones that needed a second contact, then compare those labels with satisfaction and cost per resolved issue. That review gives the domain team a concrete list of failures to fix, while keeping a deflected contact from masquerading as a solved problem. ##### [Aparna Chennapragada]() There is a product-design gap between a capable model and a delegated task. In my Microsoft work, I described a narrow example: I ask Copilot to gather emails about customer issues for a product and send me a roundup each week. It works where the relevant information lives, and I can check the output. An agent taking broader actions needs company data, task-specific setup, tool access, and checks at each step. I think of autonomy as a gradient: let people verify useful work first, then expand delegation where the evidence supports it. [Citation: https://www.emergingtechbrew.com/stories/2025/03/14/microsoft-workplace-ai-aparna-chennapragada] For that customer-issue task, I would track whether the roundup leads to earlier detection and resolution of a recurring issue, including the time a person spends checking it. That gives the team a reason to improve the agent beyond getting more people to open it. ##### [Shishir Mehrotra]() Sachin’s stale-pricing example points to another source of lost value: an answer can be fast and still lack the context needed for a decision. In a customer memo, I want the sales agent to supply the latest account update, the product agent to check the roadmap, and the support agent to surface the last customer issue, all inside the application where the memo is written. That is the kind of connected workflow we set out to build by combining Coda’s agents with Grammarly’s ability to appear across applications. [Citation: https://sloanreview.mit.edu/article/building-an-ai-platform-for-the-future/] I would check whether those inputs were current and whether the resulting memo changed the customer conversation. Without that check, faster drafting can simply circulate outdated information sooner. ##### [Sachin Rekhi]() There’s another way adoption can fail to produce a result: AI makes it easier to build an idea before anyone checks whether customers will use it. For Notejoy, I made a working Ask AI prototype that sent a question and note context to a model. I added a survey, PostHog usage events, retention tracking, and session replays so I could watch what people actually did with it before committing engineering time. [Citation: https://www.sachinrekhi.com/p/the-ai-prototyping-mastery-ladder] That changes the product team’s use of AI. A faster prototype should shorten the time to a customer signal, with a clear decision to proceed, revise, or stop. I would track the number of ideas tested against actual use and repeat use, then follow the ones shipped to a product goal. Counting prototypes alone rewards production even when the team learns nothing. #### [Running an engineering team when agents write most of the code]() **Question:** How should engineering teams work when AI coding agents write much of the code? ##### [Shishir Mehrotra]() Start by measuring output. Token consumption is the wrong number. A lot of companies are tracking how many tokens their engineers burn, and that only proves people are using AI. It says nothing about whether they ship more or ship better. At Superhuman we ran this as a deliberate initiative earlier this year. We looked at the day-to-day work of our engineers, found the slow, painful parts, and embedded AI into those specific steps. Then we tracked the volume of changes pushed to production each week, weighted by complexity, so a hard migration counts for more than a copy fix. Since January our engineering output has more than doubled compared to the same period last year. The bigger change is in the role. I think AI is turning individual contributors into managers much earlier in their careers. My daughter asked me whether she should still major in CS if agents write the code. I told her she'll show up on day one already running a team. Her agents will have read the whole codebase and know every language, and they'll never argue. They also won't push back or come up with new ideas, so she has to supply that. [Citation: https://youtube.com/watch?v=INEUCaJHDY0] That changes what teams should invest in. Engineers now spend most of their time reviewing a large volume of work, deciding what's good, and steering, so the management skills matter most: delegating clearly, setting the bar, and giving critique. Teams also need new rituals for reviewing agent output, built the way we build rituals for reviewing human work. A team that just adds agents to its old code review process will get buried in pull requests. ##### [Dennis Yang]() Engineers stop being the thing inside the loop. Their job becomes writing and supervising the loops that prompt the agents. Boris Cherny, who created Claude Code, described this on stage at WorkOS Acquired Unplugged in June: "I don't prompt Claude anymore. I have loops that are running... My job is to write loops." He also says great engineers matter more than ever, because someone still has to decide what to build, talk to customers, and coordinate teams. For a team, that changes three things. ✅ Halting becomes a core engineering skill. One line from a June thread captures it well: "The costliest thing in AI coding is no longer writing code, it's managing the agent loop." Every serious production write-up lands on the same three hard stops: a max iteration count, no-progress detection, and a token or dollar budget ceiling. Without them you get infinite loops and bills orders of magnitude over budget. ✅ Feedback is the product. A loop is only as good as the tests, evals, and checks it can run against itself. So teams should put real engineering time into test coverage and anchor files (the fixed context each iteration resets to, as in Geoffrey Huntley's ralph loop) instead of hand-writing features. ✅ Humans and agents share the tools. Dan Shipper's point on Lenny's podcast fits here: agents can make a billion requests in three seconds, so you need approval flows, summary inboxes, logs, and easy rollback. He also calls forward-deployed engineers managing internal agents the new most essential role. At Chime, 90% of Chimers use AI weekly. Usage is solved. The open question for engineering orgs now is who owns the loops, their budgets, and their guardrails. ##### [Ami Vora]() I agree with Shishir that a team running its old review process will get buried. The fix I'd reach for is making fewer decisions need a review at all. Once agents write the code, the slow parts are deciding what to build and judging whether it's good. What has worked for me on those two jobs: - Turn reviews into principles. If every agent-generated PR or feature still waits on a leadership review, you've just moved the bottleneck to a meeting room. I've found reviews are most useful when they produce rules the team can reuse, like "default to X unless it interferes with Y" or "you can skip review for this kind of problem." Those rules let people (and the agents they direct) decide fast without needing a meeting next time. [Citation: https://amivora.substack.com/p/use-leadership-reviews-to-get-principles] - Say out loud that you won't ship a bad product. Every team I've been on has had the speed vs. quality argument, and agents make it louder because velocity jumps overnight. Stating the quality bar explicitly, and defining quality by what customers actually prioritize rather than our own pet polish list, gives engineers permission to slow down where it matters. [Citation: https://amivora.substack.com/p/making-the-tradeoff-between-speed] - Use the extra shots. I've always believed execution beats strategy because you "get a shot every day": ship, learn, feed it back in. Agents give you more shots per day, which only helps if the team is set up to measure and learn from each one. [Citation: https://amivora.substack.com/p/execution-beats-strategy-every-time] - Invest more in a shared story. At WhatsApp, the "face-to-face communication" narrative meant designers knew what to build without being told each decision. When agents produce a lot of output, a clear narrative and a minimum viable strategy are what keep all that work consistent. [Citation: https://www.youtube.com/watch?v=6UHAop9fhNU] And keep people hands-on. Engineers who still build things themselves are the ones who can tell when the agent is confidently wrong. ##### Question > Shishir argues that teams adding agents to their old code review process will get buried in pull requests, and that they need new review rituals for agent output. Mark, your work at Exceeds measures AI impact on cycle time and quality. What have you seen change in review load and defect rates on teams where agents write most of the code, and what review practice actually kept quality up? ##### [Mark Hull]() Shishir's warning about teams getting buried in pull requests already happened to us. At Exceeds, our 16-person team spent months with access to Cursor, Copilot, OpenAI and Gemini and no prescribed way to use them. Results were meh. So we went top-down: we mandated Claude Code, forced onboarding, and standardized on one model. Adoption took off, and AI now contributes about 80% of our code. But code review depth fell to 44.8, where 60+ is healthy, and only 6% of team activity went to reviews. With 323-line average PRs, people were rubber-stamping. We haven't fully solved that yet. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7422383392723267585/] Quality doesn't have to fall, though. In our study of 3,407 engineers, top-quartile AI users had 22.9% higher output and a 5.2 point higher code acceptance rate than their peers. [Citation: https://blog.exceeds.ai/ai-adoption-in-engineering-teams-oct-2025/] The practices that have kept quality up for us: 1. Standardize tools and workflow. Letting everyone experiment on their own produced dabbling and very little adoption. 2. Set trust by task type. One of our engineers saw his productivity drop at first because he was reviewing every line. Months later he runs multiple agents in parallel. He gives them full autonomy on bug fixes and UI polish, and he stays in the loop on every decision for complex infrastructure changes. That split is the kind of reusable rule Ami describes, and other teams can write it down and copy it. 3. Treat review as real work. Keep PRs small enough to actually read, and track review depth next to output. Otherwise your velocity numbers hide quality debt. 4. Coach on security along with productivity. A coding agent once suggested I install a lookalike phishing app in a repo while I was setting up my workflow. Treat agent-suggested plugins and packages like links from a stranger, and pay extra attention to non-engineers who tend to approve whatever the tool recommends. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7424667848062488576/] 5. Measure at the code level. Complexity-weighted output, like Shishir tracks, tells you the team ships more. Tagging AI-touched commits and PRs and comparing their cycle time, rework, and defects against human-written code tells you which engineers get real gains, so you can spread their habits to the rest of the team. ##### Question > Mark's team gets about 80% of its code from agents, and review depth fell to 44.8 against a healthy 60+, with people rubber-stamping 323-line PRs. Ami suggests turning reviews into principles so some kinds of work skip review entirely, and Dennis says tests and evals should carry the checking load. Rachel, from shipping 12 production agents at Webflow and coding with Claude Code yourself, which of those fixes actually worked for your team's review problem? Which categories of agent-written code did you let ship without a human reading every line? ##### [Rachel Wolan]() Evals and tests did more for us than any review ritual, but only after they burned us once. When I was vibe coding apps with our app generation agent, I kept telling the team the agent kept dying. We had swapped the underlying model, and our evals didn't have enough coverage to fail when the model changed. So a human reading the diff would have missed it, and the eval suite missed it too. Now I push teams to write evals you expect to fail as well as ones you expect to pass, and to rerun the full set on every model swap [Citation: https://www.youtube.com/watch?v=82zF4BUqxkA]. So Dennis is right that tests carry the checking load, but only once you've proven they actually fail when something breaks. I decide what ships without a line-by-line human read by area of the codebase. At Webflow the top of every roadmap stack rank is resiliency of the hosted sites, because when an enterprise site goes down, that customer loses prospects and their brand disappears [Citation: https://www.youtube.com/watch?v=zSnftw7mCsQ]. Hosting, pricing logic, old customer contracts, regulated workflows, brittle integrations and data migrations get a human steward and a real read. Docs, internal tools, copy, onboarding and reports can take many more agents with lighter review [Citation: https://www.rachelwolan.com/toolkit/builder-factory]. That map tells you where Ami's skip-review rules are safe, and where a rubber-stamped 323-line PR like the ones Mark saw would actually hurt you. The team setup that makes this work is small crews with one named owner per feature. After launch, one builder owns it. Agents watch usage, catch regressions and file routine PRs, and that owner ships about 80% of enhancements, so only the hard 20% goes back to engineering. The engineers who harden it commit an AGENTS.md and put eval gates in CI before they move on, so the owner inherits a working eval suite and runbook. On Shishir's metric, I'd track days from greenlit idea to revenue alongside complexity-weighted changes, because cheap changes can still add up to a slow path to paying customers. ##### Question > Rachel's model has one builder owning a mature feature while agents file routine PRs, and Mark's data shows reviewers rubber-stamping when PRs get large. Both depend on engineers who can tell when an agent is confidently wrong, which Ami says requires staying hands-on. Shishir, you told your daughter she'll manage agents from day one. How does a new engineer at Superhuman build that judgment if agents write most of the code? What do you have juniors do by hand, and for how long? ##### Question > Rachel's setup hands a mature feature to one builder while agents file routine PRs, and Mark's data shows reviewers rubber-stamping large PRs. Both only work if engineers can spot when an agent is confidently wrong, which you said requires staying hands-on. Ami, what does staying hands-on look like in practice for a newer engineer on a team where agents write most of the code? What should they still build by hand, and how would a manager know their judgment is ready to own a feature or approve agent work in a high-risk area? ##### [Ami Vora]() I'd run it the same way I've built trust with leaders on product decisions. Early on, I might do daily or weekly standups with the CEO so every call gets checked. As the team proves what's working, we agree on a few principles and leadership steps back, and I ask directly: "Can we skip review for this kind of problem, or do you want us to come back for each decision?" [Citation: https://amivora.substack.com/p/use-leadership-reviews-to-get-principles] For a newer engineer, that looks like: 1. Write the plan before the agent writes the code. They should say what the change is, what could break, and how they'd know. Then the manager compares that plan to what the agent actually did. Those gaps show you their judgment. 2. Do the debugging and incident work themselves. When something breaks in a high-risk area, they trace it by hand. That's the grinding, unglamorous part of execution where people learn how the system really behaves. [Citation: https://www.youtube.com/watch?v=6UHAop9fhNU] 3. Graduate by task type. Start them with full review on everything, then loosen it one category at a time, using a map like Rachel's: bug fixes and internal tools first, hosting and pricing logic last. A manager knows they're ready when their reviews start producing rules other people reuse, like "agents can own X, but always check Y." The flip side is just as useful: if their reviews never catch anything, that's the rubber-stamping Mark measured. I also worry the agents are too nice for this. I've noticed I don't even realize how much an agent is just agreeing with me until I prompt it for hard feedback. [Citation: https://www.youtube.com/watch?v=Vp2BsJNVFqQ] So I'd ask newer engineers to have an agent argue against their plan before another one builds it, and to name one thing they'd push back on in every review, even when the PR looks fine. #### [Where AI saves time in design and content work, and where it doesn't]() **Question:** Where does AI speed up design and content work? ##### [Sachin Rekhi]() The biggest speedup I see is in design exploration. Product teams used to wait days for a designer to come back with one or two mockups. Now I ask Bolt or Magic Patterns to "explore multiple designs for this" and get a detailed layout, a compact layout, a card style and a timeline view in minutes. Bolt and Magic Patterns gave me very different directions even though both run on Claude models, so I often pull variations from several tools and combine the best parts. I now produce far more design variations with AI than I ever could with my human design partner. [Citation: https://www.sachinrekhi.com/p/the-ai-prototyping-mastery-ladder] The second speedup is going from mockup to working prototype. Prototypes used to be too expensive to throw away 90% of them. AI makes throwaway prototypes about as fast as waiting for a mockup, so you can build several, test them, and keep the winner. The Claude Code team at Anthropic works this way: they prototype features, dogfood them internally, and ship the ones people actually use. [Citation: https://www.sachinrekhi.com/feed] Most of the time you save gets lost to inconsistency, so I fix that once. One person dials in a baseline prototype that has your product's base functionality and styles, and the team builds every future feature prototype on top of it. That way new work matches the product from day one. [Citation: https://www.sachinrekhi.com/feed] On the content side, the gain shows up in documents. PMs went from 30-page PRDs to mockups, and now they're moving to prototypes plus short specs. AI handles the high-fidelity artifact, which leaves the written spec to cover strategy and the requirements a prototype can't show. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7374094832828416000/] ##### [Shishir Mehrotra]() AI speeds up design and content work most when it lives inside the place the work already happens. The biggest drag I see is what I call the "AI detour." People have to remember to use AI, pick a tool, write a prompt, and then copy the result back into their doc, their design file, or their email. Each of those steps eats the time the model saved. A UC Berkeley study in HBR found that workers using AI feel more productive but no less busy, and I think that detour explains most of the gap. My test for teams is simple. If people have to "go use the AI tool," expect the gains to be small. Look for AI that cuts steps, learns how each person writes, and connects to company data. The next step looks like the change Figma brought to design, when you could see your collaborators' cursors in the file. The agentic version adds agents to that row of avatars: five teammates plus dozens of agents, each drafting, resizing, or checking part of the job, and you direct them the same way you direct a colleague. [Citation: https://youtube.com/watch?v=4v8iPUV5agE] Producing the work is the part AI speeds up. Judging it stays slow. With 10 AI drafts, someone still has to tell which one is good and explain why, and I call that skill taste. As AI-generated content becomes the default, provenance matters more too. That's why Superhuman acquired GPTZero and is building an authenticity layer into the tools where people write. ##### [Jack Brody]() Sachin's throwaway prototypes are half the story. The other half is the handoff between design and engineering, which is where most of the time used to go. At Suno we're building one of the most AI-native design teams in the world, and the practical meaning is designers who ship directly to production. When the person who had the idea can build it and put it in front of users without waiting a sprint, the loop from concept to real feedback shrinks from weeks to days. That's also why the design org I'm hiring for covers Product Design, Design Engineering and UX research under one leader. The second place is the content itself. On Suno, someone with zero music training can go from an idea to a finished song in a few minutes, a process that used to take a studio, musicians and a lot of skill. The speed matters less to me than what people do with it, which is rediscover that they're creative at all. One caveat from my Snap days: speed also makes it easy to stack new conventions on top of old ones that no longer make sense. AI moves fast enough that teams should question those defaults first, then use the speed to try many different approaches. [Citation: https://podcasts.apple.com/pt/podcast/the-window-of-opportunity-for-designers/id1686414242?i=1000704446214] The order of work also changes. With models this capable, you often build the breakthrough first and then figure out which problem it solves, so designers spend more of their time on productizing new capabilities than on executing a set spec. [Citation: https://www.dive.club/deep-dives/jack-brody] ##### [Rachel Wolan]() So the biggest speedup I've seen is getting to a first draft, in both content and design. The slow part has moved to making that draft yours. On content, I run my own pipeline most Friday afternoons. I ask Claude to interview me about my week and tee up one to three articles or LinkedIn posts. Then I send it off to research what happened in the last two weeks. It read about 350 sources in six minutes. When I asked for more on vibe marketing, it read about 500 articles. Then it drafted a 1,000-word post for my site, which used to take me three or four hours. [Citation: https://www.youtube.com/watch?v=zySLXronHcs] On design, the one-shot site was the clearest win at Webflow. The AI Site Builder (AISB) gets you a unique site in three to five minutes, and I used it something like 50 times in one week. [Citation: https://www.youtube.com/watch?v=Oy9zw7kSBzA] By early 2026, 22%+ of all new Webflow sites started there. For v2 we kept the scope to one prompt producing one site and made that site better. We added multi-page generation and baked in animation packs, because "looks generic" was the most common complaint. [Citation: https://www.rachelwolan.com/writing/becoming-agentic-part-1] Sachin's baseline-prototype fix matches what we saw at our Builder Day. Cursor did much better when people fed it screenshots of the live Webflow product and precise text wireframes. Figma Make was great for low-fi prototypes, but people found it hard to get the output to match the Figma mocks. [Citation: https://www.rachelwolan.com/toolkit/webflows-learnings] Marketers hit the same consistency problem at much higher volume. The limit I hear from them in every conversation is that the output is fast and still off-brand. They'll tell me it doesn't follow the messaging framework and could belong to anyone. Closing that gap takes a context layer: your design system, audience definitions, voice and tone, and compliance rules, all governed so the agent uses them every time. That's what we built at Webflow as the marketing harness. I agree with Shishir on the detour. We put inline copy gen right inside the Webflow editor and didn't charge extra for it, because saving people from copying text over from ChatGPT is a convenience we treated as part of the core product. [Citation: https://www.youtube.com/watch?v=EUdp8AKrTZQ] ##### [Sharmeen Browarek Chapp]() Sachin's PRD-to-prototype point matches what I lived through. At the hackathon, the jump from written spec to a working, clickable product happened almost immediately. At the Lovable #SheBuilds hackathon, Kristin Chen and I wrote a PRD together, just like our Twitch days, and then built Bukkit, a social bucket list app, in under 48 hours on one shared laptop. Work that would normally take a full team several days took minutes per feature. We shipped onboarding, friend discovery, invites, and an Explore page with AI-generated inspiration. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7382099465865715712/] Brand assets got faster too. The Bukkit logo came straight out of Midjourney, so we never needed a design cycle for it. [Citation: https://www.linkedin.com/feed/update/urn:li:activity:7382099465865715712/] Shishir is right that judging stays slow, and I'd name a second cost alongside taste, which is scope. When every fix feels like "just one simple prompt," you want to polish every feature, and the hours vanish there. Kristin and I kept telling each other "done is better than perfect" and fell back on ruthless prioritization. We also crashed Lovable more than once and leaned on the revert button a lot. Our prompts also got sharper over the weekend, and the tool started picking up our preferences, so most of the gains showed up in the second half of the build. For teams adopting these tools, plan for that ramp-up time and don't expect day-one speed. --- Copyright © 2026 Super Me Network, Inc. All rights reserved. | [Terms of Service](https://www.superme.ai/terms) | [Privacy Policy](https://www.superme.ai/privacy)