Where an existing product should put its first agent
The AI Product work group ranks candidate surfaces with examples from Webflow, Microsoft Copilot, EvenUp, Chime and Superhuman.
Updated
Start with a bounded job inside a workflow customers already use, where they can check and edit the result before it goes out, and widen what the agent does as you learn where people still step in.
- Pick a bounded job people can review. Rachel Wolan would rank surfaces by whether the job is frequent and painful, whether the agent produces a unit the customer already works with, and whether the customer can review it under the right permissions before publishing. Sam Stone would start with draft-and-review work, like the demand packages EvenUp's legal customers edit before they send them.
- Weigh judgment against verification. Aparna Chennapragada would also score how much judgment the job asks of the agent and how easily the user can check the result, starting from a supervised recurring task such as a weekly digest of customer-issue emails. She suggests writing the spec as the judgment the user wants, then testing it on a small set of real requests.
- Let real demand pick the job. Dennis Yang would rank tasks by request volume, how often the current flow fails, whether the agent has reliable data and tools, and the cost of a wrong action, and keep sensitive account changes behind explicit confirmation. Rachel would look at what people already do with outside agents, as Webflow saw once it exposed its tools through MCP.
- Show up at the moment of need. Shishir Mehrotra would build interventions that recognize what the person is doing and bring the context with them, such as Superhuman's Knowledge Checker catching an out-of-date number before someone sends it, ahead of a broad agent that waits for a prompt.
- Prototype, then give it an owner. Sam would prototype against real inputs and collect 10 to 100 structured evaluations before the final ranking. Rachel would give the first surface a named owner and a decision date, and measure completed jobs, repeat use, human corrections and failed handoffs.
Ask a follow-up
AI Product
Frequently Asked Questions
What should a first eval setup for an LLM product include?
A sheet of 50 to 100 real outputs graded by people with notes, a rubric checked against customer evidence, a fixed bug set and rule checks in code, and an agreed list of failures that block a launch.
How do you know your LLM evals measure what users care about?
Build eval cases from failures users hit on real tasks, then compare scores with what people do next: rephrasing the question, editing heavily, abandoning the answer or spending time checking it.
How can a working professional build AI skills that last?
Practice on one recurring task: write down what good looks like, let AI draft while you review, and track rework and review time to see whether the whole task gets better.
How is AI changing the product manager's job?
Execution and coordination get cheaper, so more of the job is choosing the problem, judging output and supervising agents. Sachin Rekhi still wants PMs talking to customers themselves, and Ami Vora wants them close to those conversations.
Which operations work can AI agents handle today?
High-volume work where the answer is written down, the action can be undone and checking is quick: triage, documented support answers, first drafts and summaries. Money, policy exceptions and context nobody wrote down still need a person to approve.
How will AI change services businesses?
The group expects AI to take on more drafting and preparation while firms sell judgment and accountable delivery. Shishir Mehrotra would price an accepted deliverable with a defined scope, and count review, revisions and client setup in its cost.
Why doesn't AI adoption show up in business results more often?
Usage is an input. Results come from changing one workflow and measuring its outcome, including the time spent checking and fixing AI output and whether a decision changed.
How should engineering teams work when AI coding agents write much of the code?
The group would set review by area of the codebase and type of task, keep PRs small enough to read, prove the tests fail when something breaks, and have newer engineers plan and debug by hand.
Where does AI speed up design and content work?
Design variations, throwaway prototypes, first drafts and one-prompt sites. Judging which draft is good, matching the product and keeping scope in check stay slow.


