Christopher Ort
-
RL Post-Training Makes Agents More Accurate — and Less Able to Explore
Across 14 base/post-trained pairs on 3 agentic benches, base models often beat RL successors at pass@K. On WebShop, gemma-4-31B’s “pass given compute” shrinks from 87.6% to…
-
Turn Off Thinking Mode and Models Still Reason 99.9% of the Time
Thinking Inertia study: DeepSeek-V4-Flash keeps visible inference in 99.9% of open-ended answers with thinking disabled. Strict answer-only prompts raise Empty-Thinking Rate to ~40% but cut accuracy…
-
Even Top Agents Finish Only ~30% of Real Startup Product Workflows
StartupBench, built from market-validated AI products: Kimi-K3 and GPT-5.6-sol average ~73.6% partial scores, but no model completes one-third of tasks under the ≥90 success bar.
-
No Single Agent Memory Architecture Wins Everywhere, Systems Study Finds
SJTU and Tsinghua evaluate 12 memory systems across 5 workloads and 11 datasets: hybrids lead conversational QA, graphs win single-hop facts but struggle temporally, and update…
-
Coding Agents Can’t Build the Agent You Actually Want — Yet
AgentSWE’s 25 commission-style tasks: strongest builders average 19.1 on Creation and 31.6 on Editing. Optimization closes under 5% of the remaining gap; 58.1% of held-out Creation…
-
Coding Agents Reach the Right File but Still Miss the Lines That Matter
SWE-Explore grades line-level exploration on 848 issues across 203 repos and 10 languages. Agentic explorers beat classical retrieval, but line coverage — not file hits —…
-
AI Agents Almost Never Stop in Time When a Task Is Impossible
Across 28,000+ tasks, timely abstention recall stays under 40% for all models. Best web AbsRec@1 is 26.7%; a scaffold can lift Llama-3.3-70B from 26.7% to 57.4%.
-
Agent “Skills” Mostly Anchor Procedures, Not Inject Knowledge — and Retrieval Breaks Them
Skills beat workflow memory by 6.06 points, but 65.7% of their benefit is procedural anchoring. As skill pools grow from 5 to 100, actual-use precision falls…
-
LLMs Can Build Their Own Agent Harnesses — But Still Trail Human Ones on Code
HarnessDev: six creator models build execution infrastructure from a seed. Generated harnesses match or beat references on writing and ML experiments, but lag on code and…
-
Hybrid Computer-Use Agents Top Out at 41% When GUI and CLI Must Work Together
WeaveBench’s 114 real-world tasks: best PassRate 41.2%. GUI-only or CLI-only collapse below 3.5%. Outcome-only grading overestimates GPT-5.5 from 53.5% to an audited 33.3%.