Judging a new feature against the bar set before launch

Decide what a win looks like before the feature ships, read the result against that bar on a date set in advance, and check that the product as a whole gained.

• Write the bar down before launch. Yuhki Yamashita asks teams to write the headline they want to claim, then grade themselves against it with a letter and an explanation. Jiaona Zhang writes success as a branch on a KPI tree, from business outputs down to the product metrics a team can move. Nikhyl Singhal would add concrete examples of a win and a loss, and a date when the metric gets read.
• Read the results in layers. Jiaona reads adoption first, then sentiment, then behavior change such as churn, retention or activation. Output metrics come back last, so she counts early reactions to an alpha as data too.
• Pair a self-grade with harder checks. Ravi Mehta says a self-graded headline misleads when the team grading it needs the feature to have worked. He would add retention on the feature and a written link from the metric to the strategy. Where a clean A/B test is possible, he says the test decides.
• Accept a miss only with a diagnosis. Yuhki accepts a missed metric when the team names a specific cause and a fix, and rejects it when the team finds a different number that went up. Figma kept the original bar for branching and merging, whose slow start came from performance, an unfamiliar workflow and gaps found in customer rollouts. Nikhyl looks at users who finished setup: if even they drop off, more time won't help.
• Check that the whole product gained. On a crowded surface like Facebook Feed, Nikhyl would use a holdout, since a new unit can pull usage from its neighbors. Without one, Jiaona watches the top of the KPI tree, such as subscriptions, churn or whether sales can sell it. Kevin Yien watched payment volume per merchant at Square for Restaurants, and for the Stripe Dashboard he borrows a Stripe practice of counting merchant 'bad days', such as a late payout.