Quick Take
- The finding: Across Creation, Editing and Optimization of agents, no coding-agent builder comes close to delivering what a requirement asks for.
- The demo–delivery gap: In Optimization, candidates gain 13.1 points on development cases but only 1.8 on held-out cases.
- Hard failures: In Creation, 58.1% of traced held-out executions fail a validity gate — usually by erroring or timing out.
Coding agents are increasingly asked to build other agents for a user’s workflow. Beijing Academy of Artificial Intelligence and collaborators introduce AgentSWE to test whether the delivered agent actually meets the requirement — not whether it demos well on a few examples.
What the study shows
25 tasks across Creation (agent from a natural-language requirement), Editing (production agent codebases) and Optimization (toward a measurable target). Builders iterate on illustrative cases; acceptance runs the frozen submission on held-out cases with requirement-derived rubrics.
- Low averages: Strongest builders average 19.1 on Creation and 31.6 on Editing.
- Optimization barely moves the needle: No builder closes more than 5% of the remaining gap to the target on average; only 13 of 25 runs beat their starter.
- Overfitting playbooks: On PinchBench, DeepSeek-V4-Flash and Pro gained 8.1 and 59.4 on development cases via keyword playbooks, then lost 15.2 and 15.1 on held-out cases.
- Cases matter: Going from zero to two development cases raises mean Result by 34.3 points on PDF translation and 29.3 on PPTX generation; without any case, all three PDF deliveries score zero.
Why it matters: commissions are not exams
Passing a few demo cases is how agents get greenlit. AgentSWE shows that pattern systematically overstates delivery.
Method in brief
Held-out acceptance with hard pass/fail gates or source-benchmark objectives for Optimization.
What remains open
- 25 tasks: Early benchmark; coverage will grow.
- Builder set: Results reflect current coding agents and harnesses.
Context
Related coding-agent evals in Agents.
What it means for teams commissioning agents
- Hold out acceptance cases: Never grade only on what the builder saw.
- Require validity gates: Many submissions never even run.
- Watch keyword playbooks: Huge train gains can reverse on held-out work.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “AgentSWE: Can Coding Agents Build the Agent You Actually Want?” (arXiv:2610.13361v1, submitted October 9, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from main comparison of the paper, which is published under a CC BY 4.0 license.
i10x Perspective
AgentSWE’s core claim is almost moral: a commission is a promise about unseen work. Today’s coding agents still deliver demos. Until Creation clears validity gates and Optimization generalizes, “build me an agent” remains an aspirational prompt.
