Quick Take
- The finding: Under a unified harness, even the strongest model successfully completes only about 30% of StartupBench tasks.
- High partial, low finish: Kimi-K3 and GPT-5.6-sol average 73.67% and 73.61% overall scores, yet none completes even one-third of tasks at the strict ≥90 acceptance criterion.
- What fails: Complex instruction following and domain expertise are major failure sources; Finance is hardest (54.48% average).
Most agent benches invent tasks. ByteDance Seed, Nanjing University and collaborators start from AI startup products with demonstrated adoption, study their workflows and users, then turn those into deliverable-oriented tasks with fine-grained rubrics.
What the study shows
Models run under one harness with three independent runs and 95% bootstrap CIs (10,000 resamples). An automatic evaluator agrees with experts on 92.78% of rubric judgments and 92.84% of success decisions.
- Core rubrics are hardest: Average satisfaction falls from 68.67% (Auxiliary) to 65.89% (Important) to 63.45% (Core).
- Format still slips: On 56 tasks with explicit output-format requirements, success ranges from 97.6% down to 86.3% — no model is perfect.
- Partial progress is common: Agents make substantial progress without producing professionally usable deliverables.
Why it matters: market demand ≠ solved
If paying customers already buy a workflow, that is a strong signal of value. StartupBench shows agents still cannot reliably finish those workflows end-to-end.
Method in brief
Market-validated product workflows translated into rubric-scored deliverables; automatic judge validated against experts.
What remains open
- Product sample: Grounded in selected AI startups; not all domains.
- Harness fixed: Unified harness may favor some models.
Context
Related E2E agent benches in Agents and Benchmarks & Evals.
What it means for agent product teams
- Optimize for Core rubrics: That is where scores drop.
- Ship partial carefully: High averages can hide unfinished deliverables.
- Invest in domain packs: Finance lags badly.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows” (arXiv:2608.17800v1, submitted August 18, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from main results of the paper, which is published under a CC BY-NC-ND 4.0 license.
i10x Perspective
StartupBench’s methodological bet pays off: start from what the market already wants, then ask if agents can finish it. Approximately 30% says not yet — and the ~74% averages explain why demos look better than deployments.
