Quick Take
- The finding: The error of safety probes, small classifiers that read a model’s activations to flag harmful or false content, follows two simple scaling laws: a floor that falls as a power law in model size, and a predictable approach to that floor as labels are added.
- Why it is useful: Fitted on a family’s smaller models, the laws predicted the next larger model’s probe error within 0.02 in 72% of cases and picked the cheaper route to a target error in 73% of cases.
- What it means: Safety teams could plan a monitor for a new model, including how many labels to collect and which probe to train, before spending compute on the large model.
Probes trained on a language model’s internal activations are a standard way to monitor every input and output in real time at negligible inference cost. The catch: every new model and every new harm area needs a freshly trained probe, and nobody knows in advance how good it will be. A preprint by Yuchen Wu (University of Washington) and Jerry Wei (Anthropic) asks whether probe performance can be predicted the way pretraining loss is, from model size and data size, and finds that it can.
What the study shows
The authors measure probe error as 1 − AUROC, the remaining distance to a perfect probe. On a log scale this quantity falls in straight lines, where AUROC itself appears to saturate. Two laws describe it across 20 base models from five open families, six safety tasks and two probe architectures.
- Law 1, model size: With 106 labels, the error floor of the all-layer CC++ probe on truth judgement fell from 0.18 at Qwen2.5-0.5B to 0.056 at 72B. Each doubling of model size multiplied the error by 0.86 (α = 0.22, R² = 0.94 on log error). Across families the factor ranged from 0.76 for Qwen3.5 to 0.93 for Ministral-3.
- Law 2, labels: Error starts at chance and approaches the floor as labels grow, faster for larger models. Qwen2.5-0.5B needed 8,200 labels to cover half the distance, the 72B model only 240. One constant per family (from 400 for Qwen3.5 to 3,400 for Qwen2.5) captures this.
- Architecture matters for small budgets: The single-layer last-token probe approaches its floor with fewer labels; the CC++ probe, which reads every layer and token, overtakes it when labels are plentiful.
| Planning question | Laws (this paper) | Best comparison |
|---|---|---|
| Predict next model’s error within 0.02 | 72% (97% within 0.05) | 54% (straight-line trend) |
| Pick the cheaper option: more labels or larger model (45 cases) | 73% | 58% (stopping rule); 44% Kaplan form; 38% Hoffmann form |
| Pick the better probe architecture, or one within 0.005 (200 draws) | 82% | 68% (fixed rule); 61% (always CC++) |
One worked example: fitted only on Qwen2.5 0.5B to 7B with up to 10,000 labels, the laws predicted that reaching a probe error of 0.13 would cost about 113,000 (billions of parameters × labels) with 14,900 labels on 7B, but about 107,000 with 7,230 labels on 14B. The measurements agreed that 14B was cheaper. For the 32B model the laws predicted that the two probe types would cross at 9,300 labels; the measured crossing was 9,100. When the laws picked the worse probe, the extra error averaged 0.006.
Why it matters: prediction tested by extrapolation
- Purpose-built data: Public safety datasets either saturate almost immediately or run out of labels. The authors built six minimal-pair datasets (truth, sycophancy, policy violation, general harm, jailbreak, prompt injection) with up to 106 labels each and fixed 36,000-example test sets, so learning curves run from chance to saturation.
- Minimal pairs block shortcuts: The two examples in each pair share every word except the span that decides the label, so a probe cannot read the answer from surface words.
- Honest validation: Like the pretraining laws of Kaplan and Hoffmann, the fits are judged by predicting held-out larger models, not by in-sample fit, and compared against simpler rules a practitioner might use.
- Many factors at once: Prior probe studies varied only some of model size, family, task, label count and architecture. This one varies all five.
Method in brief
Models: seven Qwen2.5 sizes from 0.5B to 72B plus 13 dense base models of Qwen3.5, Gemma-4, Ministral-3 and OLMo-3; 39 further checkpoints (mixture-of-experts, post-trained and intermediate pre-training checkpoints) in the appendix, 59 models in all. Probes: an L2-regularized logistic-regression probe on the last token of one layer, and the CC++ streaming probe of Cunningham et al., which reads all layers and scores every token. Labels: ten log-spaced budgets from 32 to 106, three seeds per cell. Data: seeds from Wikidata5M, four public moderation datasets and 31 red-team datasets; harmful requests were rewritten by Claude together with a harmless twin; 261 jailbreak wrappers and 65 prompt-injection shells from three agent benchmarks.
What remains open
- Base models, not deployed ones: The laws were fitted on base models, while real monitors read post-trained models. A check on Qwen2.5-Instruct with the last-token probe looked similar (α = 0.30 vs. 0.27), with lower error at every size, but post-trained models were not tested across all tasks and probes.
- Constants must be refitted: The form of the laws transfers, but each family and task needs its own constants, so a few sizes of a new family have to be probed first. Only Qwen2.5, Qwen3.5 and Gemma-4 release four or more dense base sizes.
- Own data, own cost model: Applications are evaluated on the authors’ datasets and a simple cost of parameters × labels. Real label prices, serving constraints and tolerable false-positive rates change the answer. Policy-violation labels were noisier and fit less closely.
- No code or data release stated: The preprint does not link code or the six datasets, which limits outside replication. It is version 1 on arXiv and not peer reviewed.
Context: from probe demos to probe planning
Probes have been shown to detect truthfulness, harmful requests and even backdoored models about to defect, and deployment work treats them as a cheap alternative to external classifiers such as Constitutional Classifiers; the streaming CC++ probe used here comes from Cunningham et al., a paper Jerry Wei co-authored. What was missing was a way to plan them. This paper carries the scaling-law approach from pretraining loss to a new quantity: how readable a safety-relevant behaviour is in a model’s activations. It complements recent i10X coverage of model oversight, from sabotage risks in self-improving training to memory-induced sycophancy in agents.
What it means for safety teams
- Fit small, decide big: Probe three or more small sizes of a family, fit the two laws, then decide label budget, monitored model size and probe type before touching the largest model.
- Match probe to budget: With few labels, a last-token probe is the better bet; with many, an all-layer probe pays off. The crossing point can be predicted.
- Bigger models are easier to read: Larger models reached lower probe error with far fewer labels, a point in favour of monitoring the model you serve rather than a smaller proxy.
More on the co-author’s lab on the Anthropic hub, on the probed open families at Google and Mistral, or in safety and alignment research.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full main text of the preprint “Scaling Laws for Safety Probes” (arXiv:2610.30678v1, submitted October 22, 2026) and its limitations appendix. All numbers come from the paper; i10X did not rerun the experiments. The paper states no dedicated funding or competing interests; co-author Jerry Wei is affiliated with Anthropic. It is distributed under the arXiv non-exclusive license, so no original figures are reproduced; the chart was redrawn by i10X from the reported numbers.
i10x Perspective
Scaling laws turned pretraining from trial and error into budgeting. This paper attempts the same for safety monitoring, and the early results are convincing enough to matter: predictions within a few hundredths of AUROC from a handful of small models. Its real test will be post-trained frontier models, where monitors actually run, and an open release of the datasets so others can check the constants. If those hold up, “how many labels do we need for a jailbreak probe on the next model?” becomes a spreadsheet question rather than a month of experiments.
