Building with AI: evals, agents and the work they change
The AI Product work group answers questions about building with AI in group chats. Members cover LLM evals, what to make agentic and the skills that last. They also discuss how AI changes product, operations, engineering, design and services work, and why adoption doesn't always show up in results.
Updated
Questions Answered
A first LLM eval built on a graded sheet, a bug set and a launch bar
Start with people grading real outputs and writing notes, automate only the checks that have clear answers, and agree before launch on which failures stop a release.
Checking LLM eval scores against what users actually do
Build eval cases from failures users actually hit, then check whether a rising score comes with better outcomes in the product. Where the two disagree, the rubric needs work.
Where an existing product should put its first agent
Start with a bounded job inside a workflow customers already use, where they can check and edit the result before it goes out, and widen what the agent does as you learn where people still step in.
AI skills that outlast the next model release
Practice on a task you already do every week. The skill that carries over is giving AI the right context, a bounded job and a clear standard, then judging what comes back.
What AI changes in a product manager's job, and what stays
Execution is getting cheap, so a PM's value moves toward choosing the problem, judging output and staying close to customers, while supervising agents becomes a new kind of coordination.
The operations work AI agents can take on now, and what still needs a person
Agents handle high-volume work today when the answer is written down, the action can be undone and checking the output is quick. Money, policy exceptions and context nobody wrote down still need a person to approve.
What AI changes for services firms: the work, the review and the price
The group expects AI to take on more of the preparation, drafting and follow-through for each client, moving a services firm's value toward checking the work, owning the outcome and pricing a result the client can use.
Why high AI usage can leave business results unchanged
Usage counts the input. Results show up when a team changes one workflow and measures what comes out of it, including the time spent checking and fixing the AI's work.
Running an engineering team when agents write most of the code
When agents write most of the code, the slow parts become deciding what to build and judging the output. Teams need review sized for agent volume, tests that catch failures and engineers who stay hands-on.
Where AI saves time in design and content work, and where it doesn't
AI makes exploration and first drafts fast: design variations in minutes, throwaway prototypes, a site from one prompt. The slow parts are now judging which draft is good and making that draft match your product.
Ask a Work Group
Don't see your question? Ask the group whose members answered these.
More Work Groups
Not the right group? Ask another one instead.
AI Product
