Quick Take
- The finding: In every observed weak-to-strong OPD pair, the student’s peak held-out gold score exceeds its teacher’s — so a compact RL expert can transfer capability upward.
- Useful-transfer regime: Early training raises gold score roughly linearly in √KL from the student initialization.
- Scale laws: Peak gold improves with teacher scale only up to roughly the student’s scale; same-base Vanilla-OPD peaks within 0.5 pp of direct-RL references.
RL builds strong reasoners, but how that capability transfers across sizes is unclear. Zhejiang University and Kuaishou researchers map on-policy distillation scaling across weak-to-strong, same-base and strong-to-weak setups.
What the study shows
They fit power laws for G_peak and the useful-transfer slope versus student/teacher sizes and teacher gold score. Example: 7B student peaks rise from 72.7% to 81.9% with larger teachers; 14B from 77.7% to 87.3%.
- Off-policy cold starts hurt more as gaps grow: Costs of 6.6–15.7+ points appear in weak-to-strong gaps.
- Transfer endpoint: Defined when trajectories leave a 95% predictive band for three checkpoints; 9/25 Vanilla-OPD runs depart within support.
- Predictive check: Joint law predicts a held-out peak within 0.3 points (74.1 vs 73.8).
Why it matters: distill up, not only down
Conventional wisdom emphasizes strong-to-weak distillation. This work shows weak-to-strong OPD can overshoot the teacher — useful when RL is affordable only at small scale.
Method in brief
Token-level reverse KL diagnostics; power-law fits; Vanilla-OPD vs OffPD comparisons.
What remains open
- Same-family setups: Cross-family transfer untested here.
- Gold score definition: Held-out accuracy metric as defined by authors.
Context
Related OPD work in Efficiency & Training.
What it means for training stacks
- Try small RL teachers on larger students: They can overshoot.
- Watch √KL curves: Useful-transfer ends predictably.
- Avoid off-policy cold starts at large gaps: They grow costly.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “Scaling Properties of Same-Family On-Policy Distillation” (arXiv:2609.32722v1, submitted September 26, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The paper is distributed under arXiv’s non-exclusive license, so i10X does not reproduce its figures; the chart was drawn by i10X from numbers reported in weak-to-strong peak results.
i10x Perspective
The punchline — every weak-to-strong student beats its teacher — reframes OPD as an upward capability pump, not just compression. Scaling laws that cap teacher usefulness at student size are the practical constraint to design around.
