Article by John Tribbia
A biosecurity classifier ships, and refusals on biology-adjacent prompts jump from 1.2 percent to 8.2 percent of traffic. Thirty-day retention among the refused group lands 7.2 points below everyone else, the kind of gap that gets a patch rolled back before lunch if nobody reads past the headline.
Almost none of that 7.2-point gap turns out to be a real cost. Most of it is people the classifier was built to remove. A slice of it is an academic calendar that has nothing to do with the patch. What's left, once both of those are pulled out, is one-twentieth the size of the number everyone would have reacted to. Getting from the first number to the second is the exercise below.
What's real here and what isn't
The pressure to ship these classifiers is real. Anthropic's own reporting on disrupted misuse of Claude, spanning cyber operations, biological misuse, and other harm categories, is one instance of why.1
No public dataset pairs a safety-classifier rollout with user-level retention. That pairing doesn't exist to download, so it's built here with a known effect wired into it on purpose. Fifteen thousand synthetic users are split across three archetypes: general users, students who lean on biology-adjacent language for coursework, and a small population of users probing for genuinely dual-use content. Each user has a frozen pre-period intent share $w_{i,c}$ across five prompt categories. A classifier patch goes out at day 30 that correctly suppresses dual-use prompts and, as a side effect, raises the refusal rate on ordinary academic biology writing.
Three mechanisms are injected into the churn outcome, and their true sizes are known because this data was built, not observed: a genuine friction effect from false-positive refusals, a seasonal shock that hits students regardless of whether they were ever refused, and a deliberate departure effect for the small population that got correctly blocked. The question the rest of this post answers is whether a reasonable analysis pipeline can pull those three apart from the outcome data alone, the same way it would have to on real telemetry.
Tier 1: what the telemetry says
The first read is descriptive: track the refusal rate on biology-domain prompts before and after the patch ships. It moves fast and it moves a lot.
That's the whole of what Tier 1 can tell you. It confirms the patch is doing something, and it can't tell you whether that something is a problem.
Tier 2: the naive comparison
The obvious follow-up: do users who got refused actually leave at a higher rate than users who didn't. Split the post-rollout population into two groups by whether they hit at least one refusal, and compare 30-day retention.
This is the number that would go in a postmortem if the analysis stopped here, and it's confounded in two directions at once. Users who trip a dual-use refusal skew toward the population that was already planning to leave once blocked, since getting blocked was the point. Users who trip a false-positive academic refusal skew toward students, and students churn on an academic calendar that has nothing to do with any classifier. Both of those get folded into the same 7.2 points as whatever real friction the false positives caused.
Tier 3: isolating the real number
Two things make the split possible that a plain regression of churn on exposure doesn't get for free. First, which category triggered a refusal is known at the event level, so the population that was correctly blocked on a dual-use prompt can be excluded from the friction estimate rather than averaged into it. That's a separate, already-understood mechanism, not a nuisance parameter to control away. Second, the exposure measure itself is centered within the rollout version, $S_{i,t}^c = S_{i,t} - \bar S_{v(t)}$, so the analysis is asking whether users with an above-average academic-biology mix churn more than other users in the same window, not whether the population churns more after the patch than before.
Neither of those alone is enough. A regression of churn on centered exposure, run across the whole population with no restriction, still returns a coefficient roughly two and a half times the true injected effect. The dual-use-heavy users who left on purpose are still in the sample, and their exposure correlates with the same variable. Restricting to non-malicious users and adding a cohort control for student status brings the coefficient down to 79 percent of the true effect: an undershoot now, from measurement error in the pre-period intent-share estimate itself, which is built from a finite sample of twenty prompts per user and is noisy by construction.
The measurement-error problem has a known fix. Split each user's pre-period prompt history into two halves and correlate the two academic-biology share estimates against each other. That correlation is 0.81, and the Spearman-Brown-corrected reliability of the full-period estimate is 0.90. Dividing the regression coefficient by that reliability is the standard classical-test-theory disattenuation correction, and it moves the estimate from 79 percent of the true effect to 88 percent.
Eighty-eight percent is not a full recovery, and reporting it as one would be the same mistake as the naive number in the other direction. The split-half correction accounts for measurement error in the exposure variable. It does nothing for noise elsewhere in the pipeline: how cleanly a category classifier separates academic from dual-use language, for one. On real telemetry, that residual gap is exactly the line to flag to whoever is deciding how much to trust the number, not the line to explain away.
| Estimate | Sample | % of true effect |
|---|---|---|
| Naive regression | All users, no controls | 250.6% |
| Restricted + cohort control | Non-malicious users, controls for student status | 78.5% |
| Disattenuated | Same, corrected for split-half reliability | 87.5% |
What this changes
The naive 7.2-point number points a fine-tuning team at the wrong problem: it looks like a UX regression large enough to justify loosening the classifier. Decomposed, the number points at a narrower, cheaper fix instead: tune the classifier specifically against academic-biology phrasing. That recovers something on the order of 0.3 to 0.4 retention points among benign users, without touching the 5.7 points of departure the patch was supposed to cause. Chasing the naive framing would have traded away real threat suppression for a number that was never mostly about friction in the first place.
Where this generalizes
The same three-tier progression applies past this one scenario: descriptive telemetry, a naive observational comparison, then a within-version estimator that separates mechanisms the naive comparison conflates. A few adjacent questions a preparedness function runs into, and roughly how each gets answered:
| Question | Method |
|---|---|
| Adversarial drift | How fast are prompt-encoding evasion techniques shifting after a safeguard ships, tracked via population stability index or embedding-space drift over time. |
| Threshold placement | Where a dual-use classifier's decision boundary should sit to hold false-negative risk near zero while keeping false-positive friction under a fixed budget: cost-weighted precision-recall optimization. |
| Capability triggers | Whether a model has crossed a capability threshold that mandates pre-deployment safeguards: continuous automated eval pipelines against a fixed threshold. |
What stays constant across all three is the discipline in Tier 3: know which mechanism you've excluded on purpose, know which measurement error you've corrected for, and report the part of the gap you can't yet explain instead of rounding it away.
scripts/generate_refusal_tax_data.R with three known effects wired into the data-generating process (a benign-friction coefficient, a student-cohort seasonal shock, and a malicious-departure bump). Population size: 15,000 users. Random seed fixed at 42 for reproducibility. Run it yourself in this Colab notebook. The within-version centering and split-half disattenuation approach follows Tribbia, J. (2026), "Identifying Model Quality Effects on User Engagement: A Within-Version Causal Estimator with Synthetic Data Validation," arXiv:2608.17187, arxiv.org/abs/2608.17187, the same estimator applied there to model-quality and engagement is applied here to safety-classifier friction and retention. See also the companion analysis in Does Making AI Smarter Actually Make People Use It More?1 Anthropic, Threat Intelligence Report, September 2026.
Ideas, analysis, and opinions are my own. Generative AI was used as an editor after the writing and analysis were complete — sentence restructuring and light copy-editing. The author reviewed all suggested changes.