Why high AI usage can leave business results unchanged
The AI Product work group on usage dashboards, review debt, workflow friction and prototypes nobody tested, with numbers from Chime and Exceeds.
Updated
Usage counts the input. Results show up when a team changes one workflow and measures what comes out of it, including the time spent checking and fixing the AI's work.
- Measure one workflow, end to end. Sachin Rekhi would pick one workflow, set a baseline for its time and quality, then check whether the output changed a decision and whether that decision moved the team's goal. Dennis Yang treats Chime's 90% weekly AI use as an input. The measures he points to are in support, where cost to serve is down 60% and member satisfaction has more than doubled.
- Faster output can hide review debt. At Exceeds, Mark Hull's internal measures showed 99% AI usage and a 1.89x average productivity lift, while review depth fell to 44.8 against a healthy threshold of 60. He would track review, rework, defects, customer outcome and tool cost alongside output, and compare against a baseline.
- Count the checking time. Sachin's competitor-pricing research became useful only once a browser agent read each current pricing page and he hand-checked the comparison. Dennis would sample the support conversations behind a rising automation rate to find members who had to ask again. In a Chime case study he shared, domain experts owned ground truth and evals while engineering built the pipelines.
- Bring the AI into the work. Shishir Mehrotra says every detour to a separate chat cuts how often the tool helps, which is why Superhuman has been building Superhuman Go to work inside the doc, inbox or tab. For an AI-drafted customer memo, he would check that its inputs were current and that it changed the customer conversation.
- Test ideas before building them out. Sachin warns that AI makes it easy to build an idea before anyone checks demand. He added a survey, usage events and session replays to his Notejoy Ask AI prototype before committing engineering time, and would count ideas tested against real use rather than prototypes made. Aparna Chennapragada would widen what an agent does only as evidence supports it.
Ask a follow-up
AI Product
Frequently Asked Questions
What should a first eval setup for an LLM product include?
A sheet of 50 to 100 real outputs graded by people with notes, a rubric checked against customer evidence, a fixed bug set and rule checks in code, and an agreed list of failures that block a launch.
How do you know your LLM evals measure what users care about?
Build eval cases from failures users hit on real tasks, then compare scores with what people do next: rephrasing the question, editing heavily, abandoning the answer or spending time checking it.
Which parts of a product should become agentic first?
A frequent, bounded job inside a workflow people already use, where they can review and correct the result before it goes out. Prototype it on real inputs before committing to the full agent.
How can a working professional build AI skills that last?
Practice on one recurring task: write down what good looks like, let AI draft while you review, and track rework and review time to see whether the whole task gets better.
How is AI changing the product manager's job?
Execution and coordination get cheaper, so more of the job is choosing the problem, judging output and supervising agents. Sachin Rekhi still wants PMs talking to customers themselves, and Ami Vora wants them close to those conversations.
Which operations work can AI agents handle today?
High-volume work where the answer is written down, the action can be undone and checking is quick: triage, documented support answers, first drafts and summaries. Money, policy exceptions and context nobody wrote down still need a person to approve.
How will AI change services businesses?
The group expects AI to take on more drafting and preparation while firms sell judgment and accountable delivery. Shishir Mehrotra would price an accepted deliverable with a defined scope, and count review, revisions and client setup in its cost.
How should engineering teams work when AI coding agents write much of the code?
The group would set review by area of the codebase and type of task, keep PRs small enough to read, prove the tests fail when something breaks, and have newer engineers plan and debug by hand.
Where does AI speed up design and content work?
Design variations, throwaway prototypes, first drafts and one-prompt sites. Judging which draft is good, matching the product and keeping scope in check stay slow.
