Quick Take
- The finding: Standard sequence-level RLVR builds its update from advantage-weighted token-gradient centroids that can be dominated by shared formatting tokens, diluting sparse discriminative signals.
- The fix: DelTA estimates token coefficients that amplify side-specific directions. Training DAPO on only the top 50% λ tokens beats full-token DAPO and random 50% selection.
- The payoff: On seven math benchmarks, DelTA surpasses the strongest same-scale baseline by 3.26 average points on Qwen3-8B-Base and 2.62 on Qwen3-14B-Base.
Reinforcement learning from verifiable rewards (RLVR) has become central to reasoning models, but how response-level rewards become token-level probability changes is poorly understood. Researchers from Renmin University of China and Ant International introduce a discriminator view of those updates — and a method to stop formatting tokens from stealing the credit.
What the study shows
They show the policy-gradient direction acts as a linear discriminator over token-gradient vectors. Centroid construction under standard RLVR averages advantage-weighted gradients, so high-frequency shared patterns dominate. DelTA reweights tokens to amplify what actually separates high- from low-reward responses.
- Top half > full tokens: Optimizing only top-λ tokens outperforms full-token DAPO despite half the gradient terms.
- Bottom half collapses: Low-λ tokens are not merely uninformative — their directions can harm learning.
- Not just sparsity: Random 50% selection does not match top-λ gains.
- Vs forking tokens: Compared against DAPO with top-20% high-entropy “forking” tokens among other baselines.
Why it matters: credit assignment is the bottleneck
If gradients reinforce “\\boxed{}” instead of the key algebraic step, RLVR spends capacity on style. DelTA is a way to measure and filter that.
Method in brief
Built on DAPO; experiments on 8× NVIDIA B200 GPUs; code at github.com/RUCBM/DelTA.
What remains open
- Math focus: Seven mathematical benchmarks; other domains untested here.
- Coefficient estimation cost: Extra compute to score λ each step.
Context
Related RLVR work in Reasoning.
What it means for RL trainers
- Inspect which tokens get credit: Formatting may dominate.
- Try discriminative filtering: Top-λ can beat full-token updates.
- Do not equate entropy with value: Forking-token filters are a different axis.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards” (arXiv:2605.21467v1, submitted May 20, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from main results and token-selection ablation of the paper, which is published under a CC BY-NC-ND 4.0 license.
i10x Perspective
DelTA’s discriminator lens is the conceptual win: RLVR was always classifying good vs bad trajectories in gradient space, and shared tokens were drowning the signal. Amplifying the rare discriminative directions is an overdue correction.
