Quick Take
- The finding: Across four models and six reasoning tasks, randomly evicting within each head — while always keeping the prompt — matches the strongest scored selector on accuracy.
- Why random works: The prompt is the fragile part; the reasoning trace protects itself via textual restatement and cross-head redundancy.
- The speedup: With vLLM, Random Attention delivers 32–43% higher tokens/sec at 32k-token generations because it never runs a scoring pass.
Long chains of thought blow up the KV cache. Almost every compression method scores tokens and keeps the top ones. Salesforce AI Research and UIUC collaborators show the scoring signal contributes almost nothing — if you simply refuse to evict the prompt.
What the study shows
Main grid at ~4× compression (~3× for LiveCodeBench) on Qwen3-4B/14B/32B and Phi-4-reasoning. Significance gated by paired problem-clustered bootstrap CIs and sign tests.
- No selector significantly beats random: Apparent leads (e.g., TriAttention +0.3–0.6 on some GPQA cells) sit inside noise.
- Scored methods can lose: SnapKV trails significantly on every model; R-KV and VaSE also lose on some models.
- Prompt preservation explains most gaps: Once the prompt is protected, selection heuristics add little.
Why it matters: complexity that does not pay
Production reasoning stacks pay latency for elaborate eviction heuristics. Random Attention suggests most of that complexity is optional.
Method in brief
Uniform random eviction within heads after protecting prompt tokens; vLLM integration; clustered bootstrap testing.
What remains open
- Reasoning traces: Designed for CoT redundancy; other workloads may differ.
- Compression ratio: Main results near 4×.
Context
Related KV efficiency in Efficiency & Training and Reasoning.
What it means for serving stacks
- Never drop the prompt: That is the fragile region.
- Try random eviction: Benchmark before fancy scorers.
- Measure tokens/sec: Accuracy parity is not enough.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning” (arXiv:2609.03430v2, submitted September 3, 2026). All numbers are taken from the paper’s text, tables and figures; i10X did not rerun the experiments. The chart was redrawn by i10X from abstract and Figure 1b of the paper, which is published under a CC BY 4.0 license.
i10x Perspective
The best eviction policy being “keep the prompt, shuffle the rest” is a deliciously deflationary result. It will annoy a lot of KV-compression papers — and save a lot of GPUs.
