Illustration, AI-generated from the story text. Tap a panel to read the story through it.

CloudOpsBench, from a team led by Ang Chen at the University of Michigan, sets 45 tasks on AWS.

Claude Fable 5.1 solved 60% and GPT-6 Astra 53%.

About 80% of failures were reasoning errors such as misjudging scope or stopping early, and about 20% were wrong actions, some of them irreversible.

Forty-five tasks on one cloud is small, and only two models are named, but those wrong actions are the ones a production team should price.
A new cloud operations benchmark finds the best agent fails four tasks in ten, mostly by reasoning badly
CloudOpsBench, from a team led by Ang Chen at the University of Michigan, sets 45 tasks on AWS. Claude Fable 5.1 solved 60% and GPT-6 Astra 53%. About 80% of failures were reasoning errors such as misjudging scope or stopping early, and about 20% were wrong actions, some of them irreversible. Forty-five tasks on one cloud is small, and only two models are named, but those wrong actions are the ones a production team should price. SourcesBA
Sources
- BNew open-source benchmark reveals reasoning as the biggest bottleneck (The Decoder)the-decoder.com
- ACloudOpsBench on GitHub (not opened)github.com
This story ran in the Morning Brief of November 5th 2026.