Tuesday, October 27, 2026i10X News · Section desk7 stories
Studies
Efficiency & Training
Training methods, scaling, distillation, inference efficiency
FilterMeta
Filtered: Meta in Efficiency & Training · 1 storyReset filter ×
-
On-Policy Distillation Isn’t the Magic — Learning Rate and KL Direction Are
Cambridge study: OnPD vs OffPD show nearly identical best accuracies (72% vs 73%). Forward KL stays at 71–73%; reverse KL swings 35–72%. Higher LR drives forgetting and denser updates.
By
Editor's picks
- Chunked KV Compression Creates Periodic Blind Spots Up to 40 Points Wide
- Diffusion-Augmented LLMs Deliver Up to 3× Lossless Speedups Over Autoregressive Decoding
- A Small RL Teacher Can Lift a Larger Student Past Itself via On-Policy Distillation
- On-Policy Distillation Isn’t the Magic — Learning Rate and KL Direction Are
- MiniMax Sparse Attention: 14× Prefill Speedup at 1M Context Without Quality Loss
More stories
· RSS