Quick Take
- The finding: On Taste-Bench, the best frontier model answers only 59.7% of decision-fork questions correctly — against a 25% random baseline for the bidirectional protocol.
- Harder later: Forks whose deciding evidence appears later in the trajectory are much harder for every model, and a larger reasoning budget does not improve accuracy.
- Trainable: Distilling a teacher’s outcome-aware judgment into a student improves decisions on unseen tasks and end-to-end success on held-out SWE-bench Pro tasks.
Long-horizon agents constantly choose which hypothesis to test or which implementation to build on. Researchers from City University of Hong Kong, Microsoft and an independent collaborator call that judgment “taste” — and note that end-to-end success benchmarks never measure it directly.
What the study shows
Taste-Bench mines 502 questions automatically from agent trajectories: 390 engineering (from 2,677 rollouts on 517 SWE-bench Pro tasks) and 112 research (from 1,132 runs on 47 AI R&D tasks). Each question is a fork where one direction later proves better; the model must choose without seeing the aftermath. Human reviewers agree with mined labels 98.8% of the time.
- Wide model gap: Fourteen models differ widely; none is close to solving the bench.
- Evidence horizon matters: Accuracy falls as the deciding evidence moves later in the trajectory.
- More thinking ≠ better taste: Varying reasoning budgets for GPT-5.6 variants did not raise accuracy.
- Construction quality: Of 4,657 candidate forks, 10.8% passed all filters to become questions.
Why it matters: success hides bad forks
An agent can finish a task after several lucky recoveries from bad choices. Taste-Bench isolates the choices themselves.
Method in brief
Forks mined from parallel attempts and detours; bidirectional evaluation so position bias scores 0%. Code and Hugging Face dataset released.
What remains open
- Derived from agent runs: Question quality depends on the teacher trajectories.
- Two domains: Engineering and research; other work may differ.
Context
Complements SWE-bench-style outcome benches. See Benchmarks & Evals and Agents.
What it means for agent trainers
- Train taste explicitly: Outcome distillation of forks helps.
- Do not rely on longer CoT alone: Budgets did not fix late-evidence forks.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks” (arXiv:2609.25804v2, submitted September 22, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The paper is distributed under arXiv’s non-exclusive license, so i10X does not reproduce its figures; the chart was drawn by i10X from numbers reported in Sections 3–4.
i10x Perspective
59.7% for the best model on a 25% baseline is not catastrophic — but it means agents still misread nearly half of the forks that decide a run. Teaching taste, not only tools, looks like the next leverage point.
