Illustration, AI-generated from the story text. Tap a panel to read the story through it.

Co-Design Bench, built by researchers including Shuai Bai and Dayiheng Liu of Alibaba's Qwen team, takes 273 tasks from 26,314 real coding sessions and grades the results against 2,331 hidden acceptance criteria written by experts. The paper was submitted October 26th and written up on Sunday.

OpenAI's GPT-6 Astra did best, meeting 69.9% of a task's criteria on average but fully satisfying only 26.4% of tasks. It fully solved 41% on the track and just under 18% on the repository track.

Claude Opus 5 and Qwen3.8-Max came next and Grok 4.6 last.

The gap between 70% of criteria and 26% of tasks is the number that matters for anyone deploying these agents: most of what they ship is nearly right, and somebody still has to find the remaining 30%.
The best coding agent fully completes 26.4% of 273 real change requests
Co-Design Bench, built by researchers including Shuai Bai and Dayiheng Liu of Alibaba's Qwen team, takes 273 tasks from 26,314 real coding sessions and grades the results against 2,331 hidden acceptance criteria written by experts. OpenAI's GPT-6 Astra did best, meeting 69.9% of a task's criteria on average but fully satisfying only 26.4% of tasks. It fully solved 41% on the multimodal track and just under 18% on the repository track. Claude Opus 5 and Qwen3.8-Max came next and Grok 4.6 last.
The gap between 70% of criteria and 26% of tasks is the number that matters for anyone deploying these agents: most of what they ship is nearly right, and somebody still has to find the remaining 30%. The paper was submitted October 26th and written up on Sunday. SourcesAB
Sources
- AarXiv: Co-Design Bencharxiv.org
- BThe Decoder: AI coding agents still need human cleanupthe-decoder.com
This story ran in the Morning Brief of November 9th 2026.