Quick Take
- The finding: FrontierCS 2 asks coding agents to rebuild a paper’s deleted contribution so that it beats the authors’ own code. Across 48 tasks in 13 computer-science domains, Claude Opus 5.5 in Claude Code passed 58% of runs. Every other agent passed under half, and no agent passed any of 16 tasks.
- Why the bar is hard: A run passes only when the final patch beats the human reference on every metric and every held-out workload. Most runs that score higher than the reference on a combined score still miss at least one threshold.
- What it means: Agents can contribute to frontier CS R&D, but the median task is about as hard as what the strongest agent can handle. Which agent leads also depends on the cost budget per run.
Most open-ended coding benchmarks hand an agent a problem statement and grade one program after the run. Real research looks different: you inherit a working repository, change one component, test on the paper’s workloads and are judged against the authors’ own implementation. FrontierCS 2, a large multi-institution effort, builds that setting into a benchmark of 48 tasks and asks whether today’s LLM agents can clear it.
What the study shows
Each task is built by subtraction. The authors take a paper’s released repository, delete the code that implements the paper’s contribution and keep that deleted code as a private human reference. The agent may edit only the files that held the contribution, submit patches for five hours and receive development-suite scores. The evaluator, the held-out final suite and the reference stay with the judge. At the end the agent names one patch as final, and that patch alone is graded on held-out workloads the agent never saw.
| Agent (harness) | Pass rate |
|---|---|
| Claude Opus 5.5 (Claude Code) | 58% |
| GPT-6 Astra (Codex CLI) | 44% |
| Gemini 4 Argon (Pi) | 40% |
| GPT-6.1 Sol (Codex CLI) | 35% |
| GLM 5.3 (ZCode) | 25% |
| Kimi K3 (Kimi Code) | 21% |
| Muse Wonder 1.0 (Muse Code) | 21% |
| DeepSeek V4.1 Flash | 15% |
| Muse Spark 1.3 / Qwen 3.8 Max / GLM 5.3 Flash | 12% |
| Step 5 Preview (Step Code) | 8% |
- Pass ≠ higher score: Most runs whose combined final score exceeds the human reference still fall short on some metric or workload. Conversely, some passing runs score at or below the reference’s final score, because a pass needs every metric to hold at once.
- Sixteen tasks stay unsolved: No agent passed any of 16 tasks. On SVG-EAR, for example, GPT-6 Astra was faster than the reference and closer to full attention on average, yet still failed image-quality bands on two held-out clips.
- Shared capability scale: The FrontierCS Capabilities Index (FCI) places agents and tasks on one scale. Claude Opus 5.5 leads FCI; the median task sits at about the strongest agent’s FCI.
- Budget changes the leader: Under a small cost cap per run, DeepSeek V4.1 Flash leads (with a caveat that its token costs are lower bounds). Claude Opus 5.5 leads at the largest budgets.
Three case studies show what a pass captures and what it misses. On QuarterPIR (private information retrieval), Muse Wonder 1.0 replaced heavy reply payloads with two RAID-6-style checksums and shrank lookup size far more than other passing runs, yet a pass counts the same for a small or large saving. On BBQ (FPGA packet scheduler), Claude Opus 5.5 and Gemini 4 Argon both beat the reference on throughput, but Claude’s circuit used more lookup tables; the final score rewarded speed and ignored size. On SVG-EAR, speed was easy; staying inside every image-quality band was not.
Why it matters: graded against the authors’ own code
- Oracle is the paper’s code: Unlike SWE-bench tests or Kaggle leaderboards, the bar is the authors’ released implementation measured on the paper’s own workloads.
- Held-out final suite: Development scores guide the agent but do not count. Only the agent-chosen final patch is graded on held-out seeds, queries or graphs.
- 13 domains, not only ML: Tasks cover databases, HPC, cryptography, networks, architecture, security, robotics, programming languages and more, spanning 48 public tasks plus a private set of more than 100.
- Isolated sandboxes: The agent has no network except a driver that forwards model calls and patches, so it cannot download the authors’ code. GPU tasks keep agent and verifier GPUs separate.
Method in brief
Agents: 12 model–harness pairs, each run once per task (48 runs each). Harnesses include Claude Code, Codex CLI, Pi, ZCode, Muse Code, DeepSeek Harness, Kimi Code, Qwen Code and Step Code. Time: five hours per task. Pass: beat the human reference on every metric across every held-out workload. FCI: an item-response model fitted to per-metric outcomes and score margins over the reference (624 runs including an anchor agent), with 90% bootstrap intervals. Two Gemini 4 Argon runs refused by the provider count as failures.
What remains open
- One run per cell: With a single attempt per agent–task pair, pass rates have no per-cell confidence intervals; FCI intervals help, but individual task difficulty estimates are uncertain.
- Side costs ignored: Pass and final score can miss circuit size, bytes saved or other trade-offs that matter in practice. The authors say benchmarks like this will need to report those next to the pass.
- Harness is part of the agent: Results attribute performance to a model–harness pair. Separating model strength from harness design would need a factorial design.
- Private holdout: More than 100 tasks remain private for future evaluation, so today’s leaderboard can overfit the public 48.
Context: from SWE-bench to research repositories
FrontierCS 2 sits between SWE-bench-style environment grading and ML-only research agents such as PaperBench, ResearchGym and MLS-Bench. The closest design is MLS-Bench, which also asks agents to improve a named component; FrontierCS 2 extends that idea beyond machine learning and sets the bar with the authors’ own code. It also connects to earlier i10X coverage of scientific agents that finish only a fraction of end-to-end workflows.
What it means for teams evaluating coding agents
- Measure against real baselines: Beating a weak starter or a test suite is easier than beating the paper’s own code. Vendor claims should say which bar they cleared.
- Budget matters: The cheapest agent is not the same as the best agent at unlimited spend. Pick a cost cap before ranking.
- Inspect side costs: A pass that blows up circuit size or ignores memory can be the wrong win for production systems.
For models in the leaderboard, see our Anthropic, OpenAI, Google and DeepSeek hubs, or more benchmarks and evals.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “FrontierCS 2: A Benchmark for Computer Science R&D” (arXiv:2610.29932v1, submitted October 21, 2026), including appendices on task sources and the FCI model. All numbers come from the paper; i10X did not rerun the experiments. The paper uses arXiv’s non-exclusive license, so charts were redrawn by i10X from Figure 2. Code is at github.com/FrontierCS/FrontierCS-2.
i10x Perspective
FrontierCS 2 is a useful correction to leaderboard hype. Agents can rebuild real research contributions, but only the top systems clear a majority of tasks, and many “better scores” still fail a pass. The interesting next step is not another 48 tasks, but reporting the side costs that labs actually care about — size, latency, memory — next to the pass. Until then, treat a FrontierCS 2 pass as evidence of research-level coding under a hard bar, not as a general claim that agents can replace research engineers.
