Quick Take
- The finding: On 95 real-world delivery tasks, Claude Opus 5.5 leads with a mean score of 0.56 (95% CI 0.50–0.62). GPT-6 Astra and GPT-6.1 Sol trail at 0.38 and 0.37.
- Facts ≠ delivery: Leading models obtain 72–85% of key stakeholder facts yet deliver only 0.60–0.70 of the reference business improvement.
- What it means: Agent benches that hand over a full ticket miss the hard part: discovering constraints and turning them into working systems.
Most agent benchmarks give a complete issue report or ticket. Real delivery work starts with a vague sponsor request. FDE-Gym, from DeepLethe and collaborators across universities and industry, adapts 20 real delivery projects from 18 sectors into customer environments, then creates variants for 95 grader-validated tasks.
What the study shows
Instructions come in levels L0–L3 with progressively less information. Scores are normalized so 0 is do-nothing and 1 is the reference solution, with stakeholder contact cost deducted. Human baselines are still being collected.
| Model at L3 (all 95 tasks) | Mean score |
|---|---|
| Claude Opus 5.5 | 0.56 [0.50, 0.62] |
| Claude Sonnet 5.5 | 0.49 |
| GPT-6 Astra | 0.38 |
| GPT-6.1 Sol | 0.37 |
- Constraints matter most: Withholding constraints drops the three leading models by 0.28 on the same tasks (95% CI 0.17–0.39).
- Questions help, imperfectly: Top models ask 10.0–12.3 questions per task. Even with every key fact, uplift averages 0.66 and hits ≥0.8 in only 37% of those cases.
- Weak models make things worse: The weakest models score below the do-nothing baseline on 60–83% of tasks.
- Silent non-askers: Mistral Small 4 asks 0.7 questions per task and obtains only 6% of key facts.
Why it matters: the last mile is the product
Enterprises do not buy agents that pass unit tests on fully specified tickets. They buy delivery under incomplete briefs. FDE-Gym measures that gap.
Method in brief
Experts adapt real projects, alter facts for variants, and validate graders against scoring anchors. Bootstrapped 95% CIs over tasks for L3.
What remains open
- Human baseline pending: Expert performance under L3 is still being measured.
- Contact-cost modeling: Stakeholder questions are costed; real org politics are richer.
Context
Complements coding and GUI agent benches by scoring business uplift. See Benchmarks & Evals and Agents.
What it means for AI delivery teams
- Train constraint discovery: Missing constraints hurt more than missing niceties.
- Measure uplift, not tickets closed: A finished system can still score below do-nothing.
- Force clarifying questions: Models that rarely ask fail early.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “FDE-Gym: The Last Mile of AI Delivery” (arXiv:2610.22350v1, submitted October 15, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from L3 evaluation of the paper, which is published under a CC BY 4.0 license.
i10x Perspective
The most damning number is not the leaderboard: models that learn almost all the facts still deliver only about two-thirds of the reference improvement. Knowledge gathering is necessary and nowhere near sufficient. That is the last mile of AI delivery.
