Quick Take
- The finding: With identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 in Terminus 2 but 49.6% in Codex CLI — the harness is half the story.
- Creation results: LLM-built harnesses still lag mature human references on code and search/research, while matching or exceeding them on writing and ML experimentation.
- Evolution helps modestly: Self-runtime revisions improve held-out scores by +1.43 to +4.44 points (mean +3.11), but feedback scores predict held-out gains only 53.1% of the time.
Agent benchmarks usually fix a harness and score the model. ByteDance Seed and collaborators flip the question: can the model create and evolve the harness itself? HarnessDev grades runnable infrastructure, not just task answers.
What the study shows
Creation starts from a minimal seed and a few cases; Evolution revises the agent’s own harness using downstream feedback. Results cover six creator LLMs, four domains and five downstream benchmarks totaling 2,207 unique instances, with hidden evaluation tasks withheld from development.
- Harness defects dominate some failures: 77.8% of failed Data tasks were attributed to harness defects rather than executor capability alone.
- Executor–harness coupling: Switching executors can break specialized rules: in one Opus Search harness, duplicate-query rate jumped from 10.1% to 88.2% after the executor changed.
- Weak executors bottleneck strong harnesses: Several Qwen and DeepSeek harnesses improved under Gemini, including +17.6 on BrowseComp and +12.9 on MLE-bench.
- Diagnosis is rare: Trajectory inspection covered only 0.5%–40.2% of feedback tasks depending on lineage.
Why it matters: the missing eval axis
If changing the harness moves scores by double digits, reporting only model–task pairs is incomplete. HarnessDev makes infrastructure a first-class deliverable.
Method in brief
Capability (held-out task success) and efficiency (execution-token cost) scored for each constructed harness. Evolution uses feedback pairs from SWE-Pro-100 and Terminal-Bench-89.
What remains open
- Reference ceilings: Exceeding 100% means beating the selected human reference, not humans in general.
- Feedback ≠ held-out: Only 2 of 9 declared versions were held-out optimal.
Context
Related harness work includes HarnessAudit. See Agents.
What it means for agent platforms
- Version harnesses like models: They move the score.
- Re-test after executor swaps: Rules are model-specific.
- Inspect failures: Diagnosis is underused.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?” (arXiv:2609.01437v1, submitted September 1, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from introductory harness comparison of the paper, which is published under a CC BY-NC-ND 4.0 license.
i10x Perspective
HarnessDev’s quiet provocation is that agents should be graded on the systems they build to run themselves. The 14-point Terminus-to-Codex gap for the same GPT-5 weights is the punchline: we have been scoring the wrong object.
