Quick Take
- The finding: Codex with GPT-5.5, which scores 82% on Terminal-Bench, reaches only 23.3% overall on ALE-CLI and 0.0% on the hardest Last-Exam tier.
- Still unsaturated: Across mainstream harness and backbone setups, average full pass rate is below 1% on the hardest tier; most agents including Claude Code are near zero there.
- What it means: Short coding-agent benchmarks are saturating. Long expert workflows are not.
Coding agents look strong on Terminal-Bench and similar suites. Agents’ Last Exam (ALE) asks a harder question: can they complete long, expert-authored professional workflows that take hours? A large consortium spanning universities and industry released ALE with 1,490 task instances across domains, keeping most of them private to limit overfitting.
What the study shows
Only about 150 of 1,490 instances (~10%) are public; the rest stay in a private pool. Tasks are split into Near-Term (67 workflows where top pass rates approach 40%), Full-Spectrum and Last-Exam (38 hardest workflows). Each run is capped at five hours.
| Configuration / tier | Pass rate |
|---|---|
| Codex + GPT-5.5 on Terminal-Bench (reported) | 82% |
| Same setup, ALE-CLI overall | 23.3% |
| Same, Near-Term | 37.2% |
| Same, Full-Spectrum | 19.0% |
| Same, Last-Exam | 0.0% |
| Average full pass, hardest tier (mainstream) | below 1% |
- Costly to run: A single frontier-agent run on one ALE task costs about $3–10 and takes tens of minutes to hours.
- Timeouts are real: Overall timeout rate is 3.8%, from ~1% for lightweight harnesses to 5.7% for OpenClaw.
- Failure modes vary: Analyses show incomplete implementations, domain-knowledge gaps, hallucinations and tool-use errors across configs.
Why it matters: headroom returns
When agents score 70–80% on popular coding benches, it is easy to declare the problem solved. ALE shows that economically valuable, multi-hour workflows remain almost untouched.
The private holdout is an explicit response to contamination of public agent tests.
Method in brief
Expert-contributed workflows with a public/private split, multi-tier difficulty, five-hour caps and a living leaderboard. Project site and Hugging Face assets accompany the paper.
What remains open
- Expensive evaluation: $3–10 per task limits who can iterate quickly.
- Moving target: Harnesses and backbones change; leaderboard numbers will drift.
Context
ALE sits above short CLI and GUI agent benches. More agent evaluation is in Benchmarks & Evals and Agents.
What it means for agent builders
- Stop optimizing only Terminal-Bench: Add long workflow evaluations.
- Budget for hours, not minutes: Timeouts and cost are part of the product.
- Use the private split: Treat public ALE tasks as practice, not the score that matters.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “Agents’ Last Exam” (arXiv:2606.05405v3, submitted June 3, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from ALE-CLI results vs Terminal-Bench comparison of the paper, which is published under a CC BY 4.0 license.
i10x Perspective
ALE’s zero on the hardest tier for a model that dominates Terminal-Bench is the kind of result the field needs: proof that agent capability is still narrowly concentrated. The private pool is almost as important as the tasks. Without it, tomorrow’s agents would memorize today’s expert workflows.
