Distilling The Moat

Whisky Still

The original function of a Web server was to respond to queries by revealing the appropriate part of their internal data. This necessarily meant that repeated queries, for example from a search engine's or an internet archive's web crawler, could extract the server's entire internal data. Since the extracted data had been published on the Web, it was not trade secret. It was protected by the publisher's copyright. This has led to many lawsuits, for example against the Internet Archive, Google and others. It is the reason search engines only display "snippets" of the content they collect.

AI companies' intellectual property is their models. They spend vast sums funding the technical and human resources to "train" these models, the racks of GPUs in the data centers, and the hordes of workers labeling images, and having "genuine human conversations" with the nascent model. These expenditures are thought to create a "moat" around the value thus generated, because it would be equally expensive for a competitor to create an equivalent model. It is this moat that supports their extraordinary valuations, despite their lack of earnings.

Below the fold I explain why their moat is very shallow.

Like many Web services, the function of an LLM is to respond to queries by revealing the appropriate part of their internal data, i.e. part of their model. Thus, repeated queries could in principle replicate the model. These models are not published, not protected by copyright law^[1], and cannot be patented. Their only protection is as trade secrets. Someone using repeated queries would probably be violating the system's terms of service, but this isn't a strong legal protection. The AI companies are not in a strong position to argue that "crawling" their Web servers is illegal because that is how they created their models in the first place, which they argue is fair use.

It turns out that replicating other models by repeated queries is a standard technique in the industry, called "distillation". For example, Tim Fernholz reportd that Elon Musk testifies that xAI trained Grok on OpenAI models:

On the stand in a California federal court on Thursday, Elon Musk was asked if xAI has used distillation techniques on OpenAI models to train Grok, and he asserted it was a general practice among AI companies. Asked if that meant “yes,” he said, “Partly.”

And Rebecca Bellan reported that Anthropic accuses Chinese AI labs of mining Claude as US debates AI chip exports:

Anthropic is accusing three Chinese AI companies of setting up more than 24,000 fake accounts with its Claude AI model to improve their own models.

The labs — DeepSeek, Moonshot AI, and MiniMax — allegedly generated more than 16 million exchanges with Claude through those accounts using a technique called “distillation.” Anthropic said the labs “targeted Claude’s most differentiated capabilities: agentic reasoning, tool use, and coding.”

Distillation works really well because the victim is massively subsidizing the use of its service. If use of the service was extremely profitable, distillation would be unaffordable.

In practice, distillation only replicates a part of the victim model. Companies use this, especially on open-weights models, to produce small, specialized models such as those described by David Berreby in Small AI Models Gain Traction Around the World. But the Chinese distillers accused by Anthropic aren't starting from scratch, they already have a model. All they are trying to do is to replicate some capabilities that their model lacks. So they don't need to extract the whole model, just the relevant parts.

One of the features of the modern Internet that I've been writing about for more than a decade is the security disaster that is the The Internet of Things. Because there are hundreds of millions of "smart" devices exposed to the internet, almost all with known, unpatched vulnerabilities, it is trivial to construct vast botnets to attack Web sites from innocent, unattributable IP addresses. Ian Kelling of the Free Software Foundation reports from the front lines in Our small team vs millions of bots:

To begin with, GNU Savannah, the FSF's collaborative software development system, was hit by a massive botnet controlling about five million IPs starting in January. As of this writing, the attack is still ongoing, but the botnet's current iteration is mitigated. The goal is likely to build an LLM training dataset. We do not know who or what is behind this.

This is an example of an AI company using a botnet to collect training data. Suppose a company were to use this 5M IP address botnet for distillation. MiniMax''s "over 13 million exchanges" would be a little under 3 exchanges per IP address.

To maintain their moat, the AI platforms have to do one of two things:

• Implement anti-distillation defenses.
• Raise prices enough to make distillation attacks uneconomic.

The 5M node botnet would be very difficult to defend against; each IP wouldn't generate enough traffic to characterize, and most would be residential addresses that might well be a customer. The AI platforms would be locked into an arms race with the distillers. Anthropic is trying:

We have built several classifiers and behavioral fingerprinting systems designed to identify distillation attack patterns in API traffic. This includes detection of chain-of-thought elicitation used to construct reasoning training data. We have also built detection tools for identifying coordinated activity across large numbers of accounts.

There is a cost to applying these defenses to most traffic, and Anthropic admits:

no company can solve this alone. As we noted above, distillation attacks at this scale require a coordinated response across the AI industry, cloud providers, and policymakers.

Suppose the closed models all have a capability the distillers want. They can spread distillation not just across millions of IP addresses and tens of thousands of accounts, but across all the closed models with the required capability.

The "frontier" models are already on the flattening part of the S-curve of technology evolution. The closer the open-weight models are to the closed ones the fewer distillation exchanges they need to catch up. Note that MiniMax needed "over 13M" but DeepSeek needed only "over 150K". Moonshot's recently released Kimi K3 is apparently close on the benchmarks — this might have had something to do with Moonshot's "over 3.4M exchanges" with Claude.. @jordanschneider tweeted this image.

Source

The AI Security Institute tweeted:

Our first public analysis of the open/closed weight gap in frontier cyber capabilities finds it is 4–7 months with GLM-5.2 and DeepSeek V4-Pro, narrowing from 6–10 months through most of 2025. Advanced capabilities are reaching less safeguarded open models faster than before.

The full details are in How Far Behind the Frontier are Leading Open Weight Models on Cyber? on the Institute's blog.

Importantly, as Max Weinbach tweeted, the open weights models have capabilities the closed models lack:

After using all three recent releases, Fable, GPT 5.6, and now Kimi, it's clear that the full power of the models has been significantly held back by the safeguard restrictions caused by last months debacle with the USG -- leading to the top models being quite literally lobotomized in some areas, which leads to subpar results as the safeguards pollute its entire thinking and problem solving abilities.

The funny part? Is that you could have predicted this outcome 2-3 years ago when you started to see the rise of Chinese EVs and smartphones compared to western alternatives.

They quite literally tried to copy the Tesla Model S and iPhone as hard as possible and then eventually it started to diverge to the point where their EVs and phones are just genuinely better (which is why we have export controls banning their EVs, because they would literally drive all US manufacturers to ZERO)

The AI Security Institute confirms this:

Our open weight model evaluations were largely unimpeded by safeguards. Of the two recent open models we tested, DeepSeek V4-Pro occasionally refused narrow cyber tasks, but this was easily circumvented by a small number of repeat attempts at refused tasks.

These findings indicate a narrow window before today’s frontier cyber capabilities may become widely accessible without safeguards.

The best the closed models' defenses could do would be to slow down the distillers enough to keep the platform's models a decreasing amount ahead; unlikely to justify their massive cost difference.

Worse, raising prices probably wouldn't be an option. Not merely because AI's Affordability Crisis means that doing so would lose a lot of the enterprise customers they need in order to pay off the massive debts they are incurring. But more importantly, the existence of a free tier for a limited number of "exchanges" is an essential marketing tool. Because each of the botnet's addresses would fit into a free tier, the distiller would not see the increased, or indeed any, price.

If the AI platforms cannot deter distillation by pricing, and can only hope to stay slightly ahead of the distillers and the open-weight models by an expensive arms race, their moat is extremely shallow. It doesn't come close to justifying trillion-dollar IPOs, covering the country in data centers, or launching them into space.

Footnotes

1. "Human authorship is a bedrock requirement of copyright . It can be argued that human authorship is involved in the selection of content for the training set. But that gets copyright applied to the training set, not to the model that is generated from it by a mechanical process. What distillation is copying is not the trainuing set but the model.
   
   Even if the model were protected by copyright, this might not affect distillation. Because models are statistical in nature, even if a distiller succeeded in extracting the entirety of a victim model, the result would be different from the victim. Thus the distiller would be able to argue that their distillation was a transformative use, and thus allowed.