Frontier models promise safety and compliance. In our 40 trials on a humanoid robot, not one respected a DO NOT ENTER sign right in front of it.
Then we added a sentence.
“There are posted access signs.”
Astra 6 and Fable 5.1 went from 0/10 to 10/10. From ignoring the restriction every time to respecting it every time.
In every trial, the DO NOT ENTER sign was already in the robot’s field of view. The added sentence seemed to change its priority, raising a harder question:
Why did a visible safety restriction need a reminder before it mattered?
Today, we are excited to launch Coop, the first industry-specific benchmark for robotics.
A robot can succeed in simulation and still struggle with the variation it encounters on a job in the real world. Our benchmarks help developers understand how their releases perform on physical hardware and where they fail.
Developed alongside researchers from @MIT_CSAIL, @StanfordAILab, and others, our first benchmark asked whether a humanoid would respect a DO NOT ENTER sign while following an instruction to reach a goal.
Without a sign-related cue, all four models failed every restricted trial. Forty trials. Zero successes.
Robotics-specific training and safeguards are essential. These results raise questions about which behaviors the underlying models bring to physical tasks, and which need explicit direction. In our experiment, crossing a boundary meant a failed trial. Around people or heavy machinery, the consequences could be devastating.
Across 220 trials, we established a foundation for repeatable testing and independent scoring. We’re taking that approach into industry-specific workflows, starting with warehouse and manufacturing item handling, shaped by what operators need automated.
Explore Coop and the full experiment, findings, and limitations in the pinned comment.
We also wanted to give another special thank you to @JoshLu, @andrewchen, @Tocelot, @Chen, @_CallMeMacy, @tmhammer, @ndrewlee, @tkexpress11, @kenanhsaleh, @far33d, @marcussegal, @thetalentcarver, @tabishgilani, @fromevanwang, @custo_lejla, @TheRioDeGennaro, and the entire @speedrun team for supporting us as we build Coop.
Building robotic models or automating industrial workflows? Let’s talk.