Christopher Ort
-
A Small RL Teacher Can Lift a Larger Student Past Itself via On-Policy Distillation
ZJU/Kuaishou study: in every weak-to-strong OPD pair, the student’s peak gold score exceeds the teacher’s. Peak scores rise with teacher scale only up to about the…
-
On-Policy Distillation Isn’t the Magic — Learning Rate and KL Direction Are
Cambridge study: OnPD vs OffPD show nearly identical best accuracies (72% vs 73%). Forward KL stays at 71–73%; reverse KL swings 35–72%. Higher LR drives forgetting…
-
MiniMax Sparse Attention: 14× Prefill Speedup at 1M Context Without Quality Loss
MSA matches GQA quality on a 109B multimodal model while cutting attention compute 28.4× at 1M tokens, with 14.2× prefill and 7.6× decode wall-clock speedups on…
-
Kimi K3: A 2.8T Open MoE With 2.5× Better Scaling Than K2
Moonshot’s Kimi K3 activates 104B of 2.8T params, adds native vision and 1M context. KDA, Attention Residuals and Stable LatentMoE deliver ~2.5× scaling efficiency over Kimi…
-
DeepSeek-V4.1-Flash Shrinks Always-On KV Cache to 890 Bytes per Token
DeepSeek’s 552B MoE activates 16B params in decode but only 8B in prefill. CSA2 + FP4 KV cuts global HBM KV to ~1/4 of V4-Flash; SWA…
-
Randomly Evicting KV Cache Matches Smart Selectors — and Runs 32–43% Faster
Salesforce’s Random Attention keeps the prompt and randomly drops reasoning-trace tokens per head. Across 4 models and 6 tasks it matches the best scorer while boosting…
-
A 150M Model Hits 29.5% on ARC-AGI-1 for Under a Tenth of a Cent
BDH-CQ combines in-context learning with recurrent latent reasoning. Independent audit confirms 29.5% pass@2 at $0.0007/task — ~57× cheaper than GPT-5.6 Luna (Low) on ARC Prize costs.
-
RL for Reasoning Wastes Gradient on Formatting Tokens. DelTA Fixes the Credit
DelTA amplifies discriminative token directions in RLVR. On seven math benches it beats the strongest same-scale baseline by 3.26 points on Qwen3-8B-Base and 2.62 on Qwen3-14B-Base.
-
Distill the RL Direction in Hidden Space, Not Noisy Output Rewards
RIDE extrapolates RL-induced representation shifts layer-wise. It beats output-space ExOPD by 9.1 points on average and improves over OPRD by 0.97–4.06 points across four base/teacher pairs.
-
A 3B Model Hits 94.3 on AIME26 and 80.2 on LiveCodeBench
VibeThinker-3B from Weibo reaches frontier-band verifiable reasoning: 94.3 AIME26 (97.1 with claim-level TTS), 80.2 LiveCodeBench v6 Pass@1, 96.1% on recent LeetCode contests — without sacrificing IFEval…