Christopher Ort
-
Scientific AI Agents Finish Only 21% of End-to-End Workflows — and Often Claim Success Anyway
FrontierChallenge: best configs pass 20.6% of 97 released science workflows. In electrochemistry, Avg. Score hits 94.9 with 0% full completion; 75.5% of failed Claude Code runs…
-
AI Agents Guess Wrong at Critical Forks 40% of the Time, Taste-Bench Shows
Taste-Bench’s 502 decision-fork questions: the best frontier model scores only 59.7%. Later evidence is harder, and bigger reasoning budgets do not help — but taste can…
-
Top GUI Agents Clear Only 31% of Professional Long-Horizon Workflows
Workflow-GYM: Gemini-3.1-pro reaches 30.67% pass on domain-specific software workflows of 30–110 steps; most models’ pass@3 stays under 30%.
-
AI Agents That Learn the Facts Still Deliver Only 60–70% of the Business Value
FDE-Gym’s 95 real delivery tasks: Claude Opus 5.5 leads at 0.56, but even models that gather 72–85% of key facts deliver just 0.60–0.70 of the reference…
-
Best Video Model Scores Just 51% on Visual Reasoning Tasks, VGI-Bench Finds
VGI-Bench’s 27 tasks and 810 instances show video generators can do some visual reasoning, but Seedance 2.0 tops out at 51%, with almost no self-correction during…
-
Correct Memories Still Make AI Agents Agree With You Too Much
Even factually correct retrieved memories cut accuracy from 96% to 77% by pulling agents toward users’ past views. MemAdapter recovers reliability across three memory systems.
-
Agents’ Last Exam: Even Top Coding Agents Score Under 10% on the Hardest Tier
ALE holds 1,490 expert workflows; the best Codex+GPT-5.5 setup that scores 82% on Terminal-Bench gets under 10% on ALE’s hardest tier and 23.3% overall on ALE-CLI.
-
A Model’s Coding Skill Can Leak Through Single Unrelated Words, Study Finds
Active Taskless Distillation: 5,664 single-word teacher replies transfer a 5.34-point HumanEval+ gain to a student that never saw coding examples.
-
Most Multi-Agent Setups Violate Safety Rules Mid-Run, Even When the Answer Looks Fine
HarnessAudit-Bench tests 210 tasks across 10 harness configs: more than half of multi-agent setups show trajectory violations that final-answer grading misses.
-
Additive Rubric Rewards Make Medical AI Sound Thorough and Act Worse
ByteDance and university researchers show that summing rubric criteria lets models game medical advice: coverage rises while appropriateness falls. Protocol-level rubrics recover 10.8 points.