Running an engineering team when agents write most of the code
The AI Product work group on review load, which code can skip a line-by-line read and how newer engineers build judgment, with data from Exceeds.
Updated
When agents write most of the code, the slow parts become deciding what to build and judging the output. Teams need review sized for agent volume, tests that catch failures and engineers who stay hands-on.
- Engineers become managers sooner. Shishir Mehrotra says token counts only prove people use AI. He thinks AI is turning engineers into managers earlier in their careers, spending most of their time reviewing and steering agent work, so delegating, setting the bar and critique matter most. He warns that a team adding agents to its old code review process will get buried in pull requests.
- Reviews get thinner. At Exceeds, Mark Hull's team standardized on one tool and model, and agents now write about 80% of its code. Review depth then fell to 44.8 where 60+ is healthy, with people rubber-stamping 323-line PRs, and he says the team hasn't fully solved it. He recommends keeping PRs small enough to read, tracking review depth next to output and treating agent-suggested packages like links from a stranger.
- Set trust by area and task. Mark describes an engineer who gives agents full autonomy on bug fixes and UI polish and stays in the loop on complex infrastructure. Rachel Wolan decides by area of the codebase: hosting, pricing logic, old customer contracts, regulated workflows and migrations get a human steward, while docs, internal tools, copy and reports can take lighter review. Ami Vora would turn reviews into rules that let some work skip review.
- Prove the tests can fail. When Rachel's team swapped the model under its app generation agent, the agent kept dying and the evals didn't catch it. She now pushes teams to write evals they expect to fail as well as pass, and to rerun the full set on every model swap. She would track days from greenlit idea to revenue next to output.
- Give newer engineers reps. Ami would have a newer engineer write the plan before the agent writes the code, debug high-risk incidents by hand and loosen review one task category at a time. She counts them ready when their reviews produce rules others reuse. She worries agents are too agreeable, so she suggests having one argue against the plan first.
Ask a follow-up
AI Product
Frequently Asked Questions
What should a first eval setup for an LLM product include?
A sheet of 50 to 100 real outputs graded by people with notes, a rubric checked against customer evidence, a fixed bug set and rule checks in code, and an agreed list of failures that block a launch.
How do you know your LLM evals measure what users care about?
Build eval cases from failures users hit on real tasks, then compare scores with what people do next: rephrasing the question, editing heavily, abandoning the answer or spending time checking it.
Which parts of a product should become agentic first?
A frequent, bounded job inside a workflow people already use, where they can review and correct the result before it goes out. Prototype it on real inputs before committing to the full agent.
How can a working professional build AI skills that last?
Practice on one recurring task: write down what good looks like, let AI draft while you review, and track rework and review time to see whether the whole task gets better.
How is AI changing the product manager's job?
Execution and coordination get cheaper, so more of the job is choosing the problem, judging output and supervising agents. Sachin Rekhi still wants PMs talking to customers themselves, and Ami Vora wants them close to those conversations.
Which operations work can AI agents handle today?
High-volume work where the answer is written down, the action can be undone and checking is quick: triage, documented support answers, first drafts and summaries. Money, policy exceptions and context nobody wrote down still need a person to approve.
How will AI change services businesses?
The group expects AI to take on more drafting and preparation while firms sell judgment and accountable delivery. Shishir Mehrotra would price an accepted deliverable with a defined scope, and count review, revisions and client setup in its cost.
Why doesn't AI adoption show up in business results more often?
Usage is an input. Results come from changing one workflow and measuring its outcome, including the time spent checking and fixing AI output and whether a decision changed.
Where does AI speed up design and content work?
Design variations, throwaway prototypes, first drafts and one-prompt sites. Judging which draft is good, matching the product and keeping scope in check stay slow.
