When it comes to AI model security, it's not just users' personal information that needs to be protected. The issue of models developed at enormous cost being mass-replicated using competitors’ training data is also becoming increasingly significant. That’s why Anthropic unveiled a defense mechanism to prevent Claude from extracting inference information.
On October 12 (local time), Anthropic announced that after implementing several defensive measures, the number of inference tokens obtained by attackers had decreased by approximately 90–99% compared to before. This does not mean the attacks have been completely eliminated; rather, the figures are based on the amount of tokens extracted as observed by the company on its own platform.
The key point of this news isn’t so much the discovery of a new threat itself, but rather the fact that the company has revealed how to effectively block the previously identified issue and the initial results of those efforts. It also highlights the need to strike a balance between protecting AI companies’ technology and ensuring convenience for general users. In particular, it’s important to consider the costs these measures incur during actual operation.
| What Does a Model Distillation Attack Extract?
Distillation originally refers to a technique that uses the responses of a large model to efficiently train a smaller model. However, the situation changes if someone sends a large number of requests without permission, collects the results, and replicates the functionality of a competing model. Anthropic is addressing organized, unauthorized scraping activities.
In particular, the company explains that the reasoning logs Claude generates before providing its final answer are a valuable learning resource. That's because you can use information not only about the correct answer but also about how the problem was approached. That's why this defense focused on blocking pathways that reveal hidden inferences, rather than simply limiting standard output.
| A simple summary wasn't enough
Anthropic previously used a method that displayed only a summary of the model’s detailed inference history, rather than revealing it in full. In an experiment published by the company, a model trained on the full inference history achieved a score of 80.4% on the SWE-bench Verified evaluation, while a model trained on the summary achieved 71.8%. This is an example of how the amount of inference information can influence subsequent training results.
In response, the attackers repeatedly sent requests designed to bypass the summarization and output the original records as-is. One case involved an organization that tested 12,000 different extraction commands. This revealed the limitation that rules blocking only a single command are ineffective against constantly evolving bypass attempts.
| Controlling the Transfer of Inference Between Accounts and Models
One of the new measures is to limit the ways in which reasoning histories can be transferred to and reused by other conversations, models, or accounts. Anthropic announced that it has introduced "preserved thinking," which prevents the editing of previous context, in Claude Fable 5.1 and Claude Opus 5.5. The company also explained that this restriction applies to new API accounts created on or after August 31, 2026.
He also explained that previous-generation models were prevented from reading the protected inferences of higher-tier models, and that in Claude Sonnet 5.5, inference logs are accessible only to the organization that generated the request. This approach goes beyond simply masking the content of responses; it manages the actual path along which inference data travels.
| Two-stage classifier and iterative detection system
Anthropic built a two-stage classifier to detect requests that attempt to extract inferences. The classifier was designed to distinguish whether an input is a general question or an attempt to uncover hidden inferences. It is also reported that the company underwent a process in which internal and external red teams tested evasion attacks and patched any vulnerabilities that were discovered.
The company reported that the false positive rate for legitimate requests in that classifier is less than 0.1%. However, even if the false positive rate is low, legitimate users may experience inconvenience on services with very high request volumes. Explaining which requests were blocked and providing a way to address legitimate objections will also be part of operational quality.
| How to Interpret a 90–99% Decrease
According to Anthropic, after introducing a two-stage classifier, the number of inference tokens extracted from its platform has consistently been 10 toIt was 100 times lower. Expressed as a percentage decrease, that's roughly 90...which accounts for about 99%. The report also noted that on some dates, the extraction volume was so low that it was difficult to measure.
However, this doesn't mean that attacks have decreased by 90–99% across all external AI services. Interpretations may vary depending on how the company identified the attacker and how many unobserved attempts there were. Anthropic itself has stated that it has not yet encountered enough of the most sophisticated, persistent evasion attempts and does not know how long the measures will remain effective.
| Tens of Thousands of Accounts Blocked and Charged
Anthropic explained that it has blocked tens of thousands of accounts linked to distillation attacks, and that in some cases, accounts were blocked within two hours of the campaign beginning. The company is applying stricter usage limits to users who repeatedly request inference logs, and is managing the system so that persistent attempts result in account suspension.
Starting September 24, we began charging for requests that were blocked in specific defense zones with very low false-positive rates, even before the model responded. While the goal is to increase costs when attackers repeatedly attempt mass bypasses, it could lead to complaints if legitimate users are mistakenly blocked. Another noteworthy change is that the security policy is now linked to the billing policy.
| The Line Between AI Competition and National Security Debates
Anthropic expressed concern that unauthorized distillation could reduce model development time and costs, but could also lead to the creation of powerful models lacking adequate safety measures. The company also cited cases involving Chinese AI firms. However, it is important to distinguish between allegations regarding specific companies and the results of technical verification.
Not all research on model compression and distillation conducted under proper authorization constitutes malicious activity. The key issues are data access permissions, contract terms, and whether there has been any unauthorized bulk extraction. The scope of defense technologies may vary depending on how the AI industry balances the openness of research, the protection of intellectual property, and safety verification.
| Going forward, sustainability and user impact will be key
This announcement combines access controls with detection and account suspension to protect Claude's reasoning, and initial observations show that the volume of data extraction has been significantly reduced. This marks a step forward from past risk warnings toward the disclosure of actual operational performance.
What we need to watch going forward is how quickly they respond to new circumvention methods, whether similar results hold true on other platforms, and to what extent legitimate developers’ work is affected. The current figures are based on Anthropic’s own measurements and do not indicate a permanent solution. Model distillation defense should be viewed as an ongoing operational challenge—where both attacks and defenses continue to evolve—rather than a one-time block.
Tags
#Anthropic #Claude #Model Distillation #Distillation #AI Security #Inference Data #Data Leak Prevention #Cybersecurity #AI Model Protection #AI Safety #Security Technology #Red Team #Generative AI
