Quick Take
- The finding: On a 109B multimodal model, MiniMax Sparse Attention matches GQA downstream quality while reducing per-token attention compute by 28.4× at 1M context.
- Wall-clock: A co-designed GPU kernel delivers 14.2× prefill and 7.6× decoding speedups on H800 at 1M tokens.
- How: An Index Branch picks Top-k KV blocks per GQA group; the Main Branch runs exact block-sparse attention only on those blocks.
Agentic workflows and repo-scale code need hundred-thousand to million-token contexts, but softmax attention’s quadratic cost blocks deployment. MiniMax and university collaborators propose MSA: deliberately simple blockwise sparse attention on top of GQA.
What the study shows
Each query still attends to only k×B_k = 16×128 = 2,048 KV tokens under the reported tight budget, yet long-context capability holds. Kernels use exp-free Top-k selection for broad GPU portability.
- Group-specific retrieval: Each GQA group can select different blocks.
- Kernel tables: Top-k latency measurements across shapes show large gains vs baselines (Table 1).
Why it matters: sparsity that ships
Many sparse attentions are hard to productionize. MSA optimizes for deployability across GPUs while keeping quality.
Method in brief
Index + Main branches; block-sparse exact attention; H800 wall-clock measurements.
What remains open
- Author kernels/hardware: Speedups are H800-specific as reported.
- Top-k budget: Results assume the stated block budget.
Context
Related long-context sparsity in Efficiency & Training.
What it means for long-context serving
- Prefer simple block sparsity: Easier kernels.
- Co-design Top-k: Selection cost can erase gains.
- Validate at 1M: Quality claims need your workload.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “MiniMax Sparse Attention” (arXiv:2606.13392v2, submitted June 11, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The paper is distributed under arXiv’s non-exclusive license, so i10X does not reproduce its figures; the chart was drawn by i10X from numbers reported in abstract timing and FLOPs claims.
i10x Perspective
28× fewer attention FLOPs with matched quality is the dream number for million-token agents. MSA’s bet is that simplicity — not exotic patterns — is what actually gets into production.
