The operations work AI agents can take on now, and what still needs a person

The AI Product work group maps support, triage, drafting and review work with examples from Chime, Webflow and EvenUp.

Updated October 13, 2026

Agents handle high-volume work today when the answer is written down, the action can be undone and checking the output is quick. Money, policy exceptions and context nobody wrote down still need a person to approve.

Bounded and reversible runs now. Rachel Wolan's email agent clears notifications and drafts replies, then waits for her yes on a batch before anything runs. At Webflow, bounded agents such as the AI Site Builder worked where AIDA, an open-ended chatbot that could touch anything in the Designer, was pulled. Aparna Chennapragada adds read-only work nobody had the hours for, like reading every escalated ticket.
Documented answers can go straight to customers. At Chime, 70% of support interactions are AI-powered, and Dennis Yang says those documented, information-only answers reach members with no human in between. He would keep approval where money moves or a policy exception gets made. Rachel lists work that depends on context nobody wrote down as not ready yet.
Checking has to cost less than doing. Sam Stone draws the map around verification cost. At EvenUp, legal professionals edit AI-drafted demand packages in minutes, and the call on what a case is worth stays with them. Aparna suggests review screens that show what the agent checked, changed and was unsure about, so a person can approve a batch in seconds.
Put review on a schedule. Shishir Mehrotra would have people review agent work at set times, like a weekly one-on-one, inside the inbox, doc or ticket where the work already lives. He would ask of every agent when a human reviews its work, and where.
Learn from what reviewers change. Dennis would have reviewers tag why they took over a case, so each failure points to a missing article, a missing action or a judgment call. Sam would track which draft sections reviewers rewrite to see where spot checks are enough, and warns that years of tickets also teach an agent a team's bad habits.

Group chat with Dennis, Sam, Rachel, Aparna, Shishir — the AI Product work group

Question:

Which operations work can AI agents handle today?

Rachel WolanFormer CPO of Webflow and VP and GM of Dropbox Core and AI

So, my answer: agents reliably handle ops work today when the job is bounded and the action is reversible, and a human approves anything that leaves the building. Open-ended "go run this function" agents still break.

My own email triage is the clearest example. The agent clears calendar notifications, marketing, and system messages. It pins what to keep and drafts replies for the people I actually owe. Then it proposes something like "archive these 40," and I say yes or no before anything runs. I don't let it send email on my behalf. It still makes small mistakes, like flagging a meeting with no link one morning, and that's fine because I'm the approval step.

YouTube

We learned the same lesson at Webflow, and it cost us. In December 2024 we pulled AIDA, an open-ended chatbot that could touch anything in the Designer. Six weeks later a different team shipped the AI Site Builder, which did one thing: a multi-page, brand-aware site in under two minutes. What worked was a series of bounded agents, each well-scoped and trustable, each inheriting the customer's brand context.

rachelwolan

That gives me a working map.

Reliable now:

Triage and sorting: inbox, tickets, feedback
First drafts against a known template
Research synthesis, like analyzing 50 interview transcriptsrachelwolan
Status rollups that keep specs, decisions, and progress connected

Not yet, without a lot of scaffolding:

Anything irreversible or customer-facing that runs with no approval step
Work that depends on context nobody wrote down. I hear "it's fast, but it's not quite right" from marketers constantly, and the cause is missing brand and messaging context. A better model won't fix that.

Most of the ops investment goes into that second bucket: brand context, workflows, governance, and feedback loops written down so agents can use them.

Dennis YangPrincipal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot

Customer support is the clearest proof point that agents handle real operations work today. At Chime, 70% of support interactions are now AI-powered, cost to serve members is down 60%, and member satisfaction has more than doubled. Those numbers are from Part I of our AI Across Chime series.

The work that automates well has a known answer that lives in a knowledge base. Our chatbot and voice bot take a member's question, find the right policy or how-to content, and explain it in plain language. In fintech that covers a big share of volume: where is my deposit, how do I dispute a charge, why was a card declined.

I'd treat the remaining 30% as the map of what agents don't reliably handle yet. That bucket is mostly cases where the answer depends on judgment or on an action with money attached, such as fraud disputes, account exceptions, and anyone already upset after a bad experience. Agents can gather context and draft the response there. A human still owns the decision.

Two things made the 70% possible, and both are operations investments more than model choices:

✅ Clean, current knowledge content. The bot is only as accurate as the articles it retrieves from.

✅ Fast, graceful handoff to a person with full context, so a member never has to repeat themselves.

Internally the same pattern holds. 90% of Chimers use AI weekly, mostly for drafting, research, and engineering work where a person reviews the output before it ships.

So my working rule: agents own high-volume, well-documented, reversible tasks today. Anything irreversible or regulated gets an agent as the assistant and a human as the approver.

Sam StoneSVP of Product at EvenUp. Former Head of Product at Tome

Rachel's approval step is necessary, but it only holds up if checking the output costs much less than doing the work. If a lawyer has to re-verify every fact an agent pulled from 500 pages of records, the agent hasn't saved any time. So I'd draw the map around verification cost. Agents work well today on high-volume, document-heavy ops work where a trained reviewer can check the output quickly. They don't yet work on rare, high-stakes cases.

At EvenUp, a personal injury demand package means hundreds of pages of medical records and bills, turned into a structured legal argument. That used to take days of paralegal time. With Express Demands, legal professionals generate and edit a draft in minutes. The lawyer still owns the final document, and that review step is the reason it works. Extraction, summarization, chronologies and first drafts are reliable now. The final judgment call about what a case is worth stays with the human.

Agents break down in two places:

Beyond the "confidence frontier." The Harvard/BCG study showed work quality dropping when people used AI past what it could reliably do and didn't check the output closely enough. For ops work, an agent that says "I don't know" is worth more than one that always produces an answer.Medium
Edge cases. Karpathy's "imbalanced class" point still holds: models handle the straight-highway cases and struggle with the rare ones that matter most.Medium

At Opendoor, when we rolled out a unified pricing model, we invested in teaching operators why it made the recommendations it did. That way they flagged recommendations that were actually wrong and didn't flood us with false alarms when the model's output changed sharply from one day to the next.

Medium Dennis's 30% handoff bucket needs the same thing: reviewers who understand where the agent is reliable, so their escalations improve it.

Shishir MehrotraCEO of Superhuman (formerly Grammarly). Co-founder of Coda and former Chief Product Officer of YouTube

I agree with Rachel's map, and her point about unwritten context explains why summarization is the most reliable ops task today. It became the most common use of AI in Coda: people point it at meeting recordings, notes, email threads or what's happening in Salesforce and ask for a summary. Everything the agent needs sits in the source material, so this generation of models does that task very well.

YouTube

I judge any ops task by treating the agent like a new hire. What does it know? That can be global facts like English grammar, team facts like the support knowledge base or wiki, or personal facts like your inbox. What skills does it have? Is the job short-running or long-running?

Apple Podcasts+1 Short jobs that run on knowledge the agent can actually reach work today. Dennis's 70% at Chime is the team-facts case: the answer already lives in the support knowledge base. Personalized outreach is a good example. Yaniv Bernstein described a modern mail merge to me on his podcast: find the 30 people I met in New York last time, tell them I'm coming back, and mention what we last discussed with each one. That works because the agent can read the history.

Spotify Long-running jobs with ambiguous goals still fail, mostly because the agent is missing context.

So the human job moves to review. I told my daughter that AI turns individual contributors into managers much earlier in their careers, because they spend their days reviewing a large volume of agent output. Rachel's approval step is one version of that, and Sam is right that it only pays off when checking is fast. That is why teams need new review rituals designed for the volume of agent work, the same way managers built rituals for reviewing a team's output.

YouTube

Aparna ChennapragadaVP of Wearables at Meta. Former CPO at Microsoft

I'd split "bounded" into two axes, because they fail for different reasons. Breadth depends on model capability. Autonomy depends on tool chaining and on whether the chain has checks built in

emergingtechbrew. Read-heavy work can be very broad and still be safe to hand off today. I once asked our Researcher agent to pull together the views of everyone in an upcoming leadership meeting and suggest how I should pitch. It raised angles I hadn't considered, and nothing it did was irreversible

YouTube. A narrow task becomes risky once the agent writes to a system of record with no check in place. Long-running, action-taking agents are the 2026 capability, and they still need supervision

Substack.

Shishir's point about turning ICs into managers is real, and the cost shows up as fatigue. I've been up at 1am going back and forth with a CLI on an app I thought was one prompt from done. Agents run 24/7, so review never naturally stops

Substack. Rachel's approve-the-archive step works at 40 emails. When drafts, PRs and tickets grow tenfold, the approver becomes the bottleneck. Code review already can't depend on a human reading every line.

Sam is right that verification cost decides whether the approval step holds up. Product teams can lower that cost with proof-of-work UX. The agent should show what it checked, what it changed, and where it's unsure, so a human can approve a batch in seconds without redoing the work. Dennis's handoff with full context applies the same idea to support: the human gets the agent's reasoning along with the case. That review surface is where I'd put the next dollar of ops investment, ahead of more agent autonomy.

Dennis YangPrincipal Product Manager, Generative AI at Chime. Co-founder and former CPO of Dashbot

Sam's verification-cost test explains why support automated first. Checking a support answer is cheap: either it matches the help article or it doesn't. That also lets me push back on Rachel's "customer-facing with no approval step" bucket. At Chime the 70% of AI-powered support interactions go straight to members with no human in between, because the answers are documented and the action is information only.

LinkedIn The approval step belongs where money moves or a policy exception gets made.

The line between the two buckets also moves every week, and the transcripts move it. I spent four years at Dashbot building analytics on billions of chatbot and voice messages for companies like Intuit and GoDaddy. The most useful report was always the list of messages the bot couldn't handle, grouped by topic. Each cluster pointed to one of three fixes:

A missing or stale knowledge article, which content ops can fix in a day.
A real action the bot can't take yet, like reissuing a card, which becomes an engineering ticket.
A judgment call that should stay with a human, which becomes a cleaner handoff rule.

So the human-handled 30% is also your labeled backlog. Reviewers who tag why they took over a case are doing what Sam described at Opendoor: they teach the system where its reliable zone ends. Aparna's proof-of-work review surface should include a one-click "why I took over" reason for exactly this purpose. Most teams under-invest in that tagging, and it's cheaper than any model upgrade.

Shishir MehrotraCEO of Superhuman (formerly Grammarly). Co-founder of Coda and former Chief Product Officer of YouTube

Aparna's fatigue point is real, and I think the fix is a cadence. Agents can run 24/7, but the human review should happen on a schedule, the same way a manager holds a weekly one-on-one instead of watching over someone's shoulder all day.

Superhuman Mail users show what that cadence looks like. One user told us they run their Superhuman workflow with Claude twice a day. The agent triages through Split Inbox and drafts replies in each person's voice, and the user reviews the batch in one sitting. Another says their inbox automation gets them to Inbox Zero 2 to 3 times faster. In both cases the agent did the volume work between check-ins, and the human made the calls in a fixed window.

The other factor is where the review happens. If the approver has to "go use the AI tool," every approval adds a step: open another app, find the output, copy it back. That AI detour is what turns review into the bottleneck Aparna describes. Review is cheap when the agent's proposals show up inside the inbox, doc or ticket the person already works in, which is the same idea as her proof-of-work UX.

For ops leaders, I'd turn this into one design question for every agent: when does a human review its work, and where? If nobody can answer both, the agent will either run unchecked or bury someone in approvals.

Sam StoneSVP of Product at EvenUp. Former Head of Product at Tome

Dennis's takeover tags and Aparna's reviewer bottleneck fit together. When review volume grows tenfold, the reviewer's edit history is what tells you where you can safely review less. In legal drafting, we'd track which sections of a draft lawyers rewrite. A section type that gets accepted almost untouched across hundreds of demands can move from line-by-line review to spot checks. A section type that gets rewritten every time stays under full review, even if each draft looks fine on its own, because the agent is past its confidence frontier there. Implicit signals like saves and edits aren't perfect measures of quality, but in aggregate they're decent proxies, and you get them for free.

Medium

One caution on Shishir's "team facts" and Dennis's transcripts. Ops histories also record a team's bad habits. If a support team has historically been rude to a certain type of customer, an agent built on those conversations will learn to do the same. General model safety training targets societal bias and won't catch institutional bias like that. Before you point an agent at years of tickets or case files, decide which past behavior you want it to repeat and which you want to drop.

Medium

Aparna ChennapragadaVP of Wearables at Meta. Former CPO at Microsoft

Dennis's failure-cluster report points to a category we've left off the map: ops work nobody does today because no one has the hours. Every list so far starts from tasks humans already do and asks which ones the agent can take over. Knowledge work has always been limited by scarce cognitive energy, so a lot of useful work never gets staffed

emergingtechbrew. Reading every escalated ticket, every churned customer's last five emails, or every vendor contract renewal clause are good examples. Agents handle that well right now. The work is read-only, it runs on source material the agent can reach, and the baseline is zero, so an 85%-right pass still beats what existed before.

Teams also set the bar too low. If you hand an agent coffee runs, you get coffee-run value. Ask it to go through 300 chat messages, 110 emails, 20 hours of meetings and 7 docs, then return a five-minute briefing with five drafted replies. That is a realistic ask today, and the drafts still go through a human before they send.

Dennis's one-click "why I took over" tag fits naturally inside the review surface. I'd use it to answer a second question too: which of those takeovers came from net-new work the agent surfaced rather than work it did badly. That ratio shows whether you're using agents to find new problems or only to cut the cost of old ones.

Question:

Rachel, Dennis pushed back on your rule that customer-facing work needs an approval step. Chime's bot sends documented, information-only answers straight to members at 70% of support volume, and he puts approval only where money moves or a policy exception gets made. Aparna adds that approving every item breaks once volume grows tenfold. Would you revise your map to allow unattended customer-facing answers when the source content is written down? And what made AIDA at Webflow unsafe to run unattended when the AI Site Builder was safe?

Rachel WolanFormer CPO of Webflow and VP and GM of Dropbox Core and AI

Yes, I'd revise the map. Customer-facing was the wrong line. Dennis's information-only answers drawn from documented content can go straight to customers with no approval step. The approval step belongs on irreversible actions, money, and exceptions, and on anything where the context still lives in someone's head. That last one is my marketing case: content ships fast but off-brand because the messaging framework was never written down.

AIDA and the AI Site Builder show the difference in scope. AIDA was an open-ended chatbot that could touch anything in the Designer, so every request could land on a customer's live, existing work, and nobody could predict the blast radius. The AI Site Builder did one thing: you typed a prompt and got a multi-page, brand-aware site in under two minutes. It started from a blank site, so there was nothing to break. We kept that pattern as the agents grew. When a customer asks the AI Assistant for a pricing calculator or a job board, it generates an AI Code Component bounded inside its own island on the canvas, so the agent can't reach the rest of the site.

rachelwolan The model behind it wasn't the issue. A bounded write surface is what made it safe to leave unattended.

On Aparna's 10x volume problem, I don't approve 40 emails one by one in my triage. I told the agent once which categories to clear: calendar notifications, marketing, system messages. My approval is a single yes on a batch it sorted using rules I wrote down.

YouTube Sam's edit history tells you which categories are ready for less review. Writing the rule down is what actually moves them there, because the next agent and the next reviewer inherit it.

Ask a follow-up

AI Product

516

AI Product

3M+ insights

AI product leaders from Anthropic, OpenAI, Midjourney & 8 more

by SuperMe

Frequently Asked Questions

What should a first eval setup for an LLM product include?
A sheet of 50 to 100 real outputs graded by people with notes, a rubric checked against customer evidence, a fixed bug set and rule checks in code, and an agreed list of failures that block a launch.
How do you know your LLM evals measure what users care about?
Build eval cases from failures users hit on real tasks, then compare scores with what people do next: rephrasing the question, editing heavily, abandoning the answer or spending time checking it.
Which parts of a product should become agentic first?
A frequent, bounded job inside a workflow people already use, where they can review and correct the result before it goes out. Prototype it on real inputs before committing to the full agent.
How can a working professional build AI skills that last?
Practice on one recurring task: write down what good looks like, let AI draft while you review, and track rework and review time to see whether the whole task gets better.
How is AI changing the product manager's job?
Execution and coordination get cheaper, so more of the job is choosing the problem, judging output and supervising agents. Sachin Rekhi still wants PMs talking to customers themselves, and Ami Vora wants them close to those conversations.
How will AI change services businesses?
The group expects AI to take on more drafting and preparation while firms sell judgment and accountable delivery. Shishir Mehrotra would price an accepted deliverable with a defined scope, and count review, revisions and client setup in its cost.
Why doesn't AI adoption show up in business results more often?
Usage is an input. Results come from changing one workflow and measuring its outcome, including the time spent checking and fixing AI output and whether a decision changed.
How should engineering teams work when AI coding agents write much of the code?
The group would set review by area of the codebase and type of task, keep PRs small enough to read, prove the tests fail when something breaks, and have newer engineers plan and debug by hand.
Where does AI speed up design and content work?
Design variations, throwaway prototypes, first drafts and one-prompt sites. Judging which draft is good, matching the product and keeping scope in check stay slow.

All building with ai questions →

Frequently Asked Questions

The operations work AI agents can take on now, and what still needs a person

AI Product

Frequently Asked Questions

What should a first eval setup for an LLM product include?

How do you know your LLM evals measure what users care about?

Which parts of a product should become agentic first?

How can a working professional build AI skills that last?

How is AI changing the product manager's job?

How will AI change services businesses?

Why doesn't AI adoption show up in business results more often?

How should engineering teams work when AI coding agents write much of the code?

Where does AI speed up design and content work?

Your ideas, super powered.

The operations work AI agents can take on now, and what still needs a person

AI Product

Frequently Asked Questions

What should a first eval setup for an LLM product include?

How do you know your LLM evals measure what users care about?

Which parts of a product should become agentic first?

How can a working professional build AI skills that last?

How is AI changing the product manager's job?

How will AI change services businesses?

Why doesn't AI adoption show up in business results more often?

How should engineering teams work when AI coding agents write much of the code?

Where does AI speed up design and content work?

Your ideas, super powered.