A model distillation attack happens when an attacker sends millions of carefully crafted questions to a proprietary AI’s public API, records every answer, and uses those harvested question-and-answer pairs to train a separate model that mimics the original. No access to the model’s internal code or training data is required — only the public API outputs. The result is a near-copy of a competitor’s AI, built at a fraction of the cost of the original research and development.

How a Distillation Attack Works

The mechanics follow a recognizable pattern:

  1. Create accounts at scale. Attackers open thousands of accounts — often through proxy services — to spread traffic and avoid triggering rate limits or anomaly detectors.
  2. Probe targeted capabilities. Rather than random questions, they send systematically varied prompts designed to map specific areas: coding, multi-step reasoning, long-horizon planning, tool use, and so on.
  3. Harvest the outputs. Every question-and-answer pair is recorded, building a large behavioral dataset of the target model.
  4. Train a student model. The attacker fine-tunes an existing open-source base model on the harvested data until it reproduces the target’s responses well enough to be commercially useful.

The name comes from the legitimate machine learning technique called knowledge distillation — where a smaller “student” model learns from the outputs of a larger “teacher” model. A distillation attack applies the same idea without authorization.

Legitimate Distillation vs. an Attack

The distinction is authorization. Knowledge distillation is a well-established, legal technique when applied to a model you own or are licensed to use. Companies routinely distill their own large models into smaller, faster, cheaper variants.

What makes a distillation attack different is that the attacker has no right to use the target model’s outputs for training. Most major AI providers make this explicit in their terms of service. Anthropic’s Acceptable Use Policy, for instance, prohibits “utilization of inputs and outputs to train an AI model (e.g., ‘model scraping’ or ‘model distillation’) without prior authorization from Anthropic.”

See also: What Is AI Model Distillation — the legitimate technique explained

Why AI Companies Prohibit It

Intellectual property. A frontier AI model represents years of research and billions of dollars in compute costs. Systematic API extraction converts that investment into a free capability for a competitor — without any R&D spending.

Competitive and safety incentives. Labs that invest heavily in safety research lose their economic moat when rivals can clone capabilities while skipping safety infrastructure costs. This creates a structural incentive to underinvest in safety: a lab that cuts safety corners can undercut on price.

National security. Distillation attacks can potentially circumvent AI export controls — chip restrictions and regional API bans designed to limit which countries and organizations can access frontier AI. A model distilled in an unauthorized region may also lack the safety guardrails of the original, making it more readily usable for harmful applications.

How AI Companies Detect and Prevent It

Detection depends on behavioral anomaly analysis. Legitimate users ask questions organically; distillation campaigns leave distinct patterns: systematic capability coverage, programmatic query bursts, massive volumes from recently created accounts, and IP addresses inconsistent with the customer base.

Technical defenses include output watermarking — embedding statistical signatures in generated text so that if an extracted model reproduces similar distributions, the watermark propagates, providing evidence of theft. Rate limiting and geographic access controls add friction, though sophisticated campaigns route through proxy networks to evade both.

Legal tools matter too: clear terms of service establish the contractual basis for enforcement, and intelligence-sharing between AI labs, cloud providers, and authorities allows coordinated response. No defense is perfect — determined, well-resourced attackers can often extract meaningful capability despite these measures, which is why detection and deterrence remain as central as prevention.

In the News

In June 2026, Anthropic disclosed that accounts linked to Alibaba’s Qwen AI lab had conducted the largest known distillation campaign to date: approximately 25,000 fraudulent accounts submitting 28.8 million conversations over six weeks, targeting Claude’s advanced software engineering and multi-step agentic reasoning capabilities. Earlier in 2026, Anthropic had also detected campaigns attributed to DeepSeek (150,000+ exchanges), Moonshot AI (3.4 million+), and MiniMax (13 million+). The disclosure escalated into a broader dispute when a hidden country-detection flag in Claude Code became public — Alibaba called it “high-risk spyware” and banned the tool internally.

→ Full story on the Alibaba–Anthropic dispute

FAQ

Is knowledge distillation itself illegal?
No. Distilling a model you own or are licensed to use is a standard, legal practice. What’s prohibited is using a competitor’s API outputs to train a model without authorization — that violates the provider’s terms of service and may constitute trade secret misappropriation.

How many API calls does a meaningful attack require?
Narrower capability extraction can begin in the tens of thousands of queries; cloning broad capabilities requires millions. Anthropic’s disclosed campaigns ranged from 150,000 to nearly 29 million exchanges.

Can watermarking prove a model was distilled?
Watermarking is a promising defense but not foolproof. Post-processing techniques like paraphrasing can strip statistical signatures, and some research has demonstrated differential-noise attacks that scrub watermarks. The field is actively evolving.

What happens to companies caught distilling without authorization?
Immediate account termination and API bans are typical. Legal exposure depends on jurisdiction and may include breach of contract, trade secret theft, or copyright infringement claims. Some labs are pursuing formal legal action as the industry matures.

Sources: Anthropic’s distillation-attack report · Anthropic Acceptable Use Policy