Quick Take
- The finding: Agentic explorers form a clear tier above classical retrieval on SWE-Explore, but file-level localization is already strong — line-level coverage and ranking are what still separate systems.
- Why lines matter: Exploration metrics strongly track downstream repair; random non-core code can cut resolve rate by 7–9 points when core evidence is scarce.
- What it means: End-to-end resolve rate hides whether an agent found the decisive span or just the right file.
SWE-bench asks whether a patch resolves an issue. Shanghai Jiao Tong University and collaborators ask the prior question: did the agent explore the right parts of the repository under a fixed line budget?
What the study shows
SWE-Explore covers 848 issues across 10 languages and 203 open-source repositories (Python alone is 547 instances). Ground truth comes from line regions consulted by independent agent trajectories that successfully solved the same issue.
- Three metric families: Coverage, ranking and context-efficiency; Context Efficiency shows the highest Pearson correlation with downstream resolve rate.
- Low F1 is a recall problem: Non-oracle explorers often hit the right files but miss decisive lines.
- Noise is costly: Injecting random non-core code at α=0 lowers resolve rate by 7–9 pp.
Why it matters: localization is half the battle
If exploration is weak, better patch generators still fail. SWE-Explore lets teams grade explorers without waiting for a full repair run.
Method in brief
Explorer returns ranked code regions under a line budget; metrics validated against a controlled downstream repair protocol.
What remains open
- Oracle trajectories: Ground truth inherits biases of successful agent runs.
- Budget choice: Line budgets affect rankings.
Context
Complements SWE-bench. See Agents and Benchmarks & Evals.
What it means for coding-agent teams
- Report line metrics: File hit rates flatter explorers.
- Penalize redundant context: Noise hurts repair.
- Test explorers separately: Do not wait for end-to-end resolves.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “SWE-Explore: Benchmarking How Coding Agents Explore Repositories” (arXiv:2606.07297v1, submitted June 5, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from dataset stats and noise study of the paper, which is published under a CC BY-NC-ND 4.0 license.
i10x Perspective
SWE-Explore formalizes something practitioners already feel: agents often “find the file” and still edit the wrong function. Measuring exploration at line level is how that intuition becomes an eval.
