Small Is Beautiful (Updated 2x)

Estimates are that, to justify the AI platforms' enormous capex plans, by 2030 they need to be generating around $2T/year in revenue. If every adult resident of the US spent $20/month on AI, it would generate $68.5B/year. Clearly, only the enterprise market stands even a remote possibility of generating the bulk of the $2T.

There are three major threats to the prospect of AI platforms extracting 6% of current US GDP from the enterprise market, and thus to OpenAI's and Anthropic's ambitions to IPO in the near future. First, faced with AI's Affordability Crisis, companies have been placing strict limits on employees' spending on AI tokens.

Source

Second, the gap in performance between expensive, closed-weight US models, such as OpenAI's and Anthropic's, and much cheaper, open-weight Chinese models has been rapidly closing, with the result that the US models are losing enterprise market share. Luz Ding, Spe Chen and Hayley Warren analyze this in US Lead in the AI Race With China Is Rapidly Narrowing:

Bloomberg in partnership with researchers at Vals AI, an independent AI evaluation and benchmarking platform, tested seven models from frontier Chinese and US companies to see how they performed in a real-world task. They were asked to create a fictional coffee e-commerce site called Brewberg using the same prompts. Most of the models scored 100% functional accuracy despite occasional design misses, but with very different price tags. The experiment employed the top performing models in July from Anthropic and all the Chinese firms, as well as more affordable models from OpenAI and Google.

They all did reasonably well, but the two best were Claude Fable 5 at $48.99 and Kimi K3 at $11.99. Chinese models charging much less for almost the same performance are grabbing market share:

the use of Chinese models overtook US platforms globally for the first time in June, and accounted for more than 60% of market share last month, on OpenRouter, a tech platform that offers software developers access to hundreds of AI models. It is a widely watched gauge of model usage despite tracking just a fraction of global AI consumption. The US, parts of Europe and Asia now favor Chinese labs, according to the same data.

On Hugging Face, Chinese AI models account for 41.4% of generative model downloads among developers, 5 percentage points higher than US models.

Third, it isn't just that the Chinese models are cheaper to run remotely, but also that because they are open-weight they can be run on affordable in-house systems, which means that:

• They don't give Donald Trump a kill-switch for your busines.
• They don't give Sam Altman or Dario Amodei a kill-switch for your busines.
• They don't require giving the Chinese, Sam Altman or Dario Amodei all your business' critical data.
• They provide visibility into and control over AI costs.
• They are even cheaper.

The question is "compared to the closed-weight US models, what do you lose by running open-weight models in-house?" Below the fold I discuss a major study from Stanford that answers the question.

The 37-page paper is Intelligence per Watt: Measuring Intelligence Efficiency of Local AI by Jon Saad-Falcon et 14 al. Their abstract reads:

Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can scale. Two advances create an opportunity to rethink it: small, local LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) can host these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? This requires measuring both whether local LMs can accurately answer real-world queries and whether they can do so efficiently on power-constrained devices (e.g., laptops). We propose intelligence per watt (IPW), task accuracy per unit of power, as a unified metric for the capability and efficiency of local inference across model-accelerator configurations. We evaluate 20+ state-of-the-art local LMs, 8 hardware accelerators (local and cloud), and 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy (local LM win rate against frontier models), energy, latency, and power. We find three key results. First, local LMs successfully answer 88.7% of these queries, with accuracy varying by domain. Second, longitudinal analysis from 2023-2025 shows IPW improved 5.3x, driven by both algorithmic and accelerator advances, with locally-serviceable query coverage rising from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for local accelerator optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries, with IPW serving as the critical metric for tracking this transition.

Joachim Klement provides an introduction to this "Small Language Models (SLMs)" paper in If this is true, the hyperscalers are toast:

First, they ran a series of SLMs (QWEN 3, GEMMA 3, GPT-OSS, GRANITE 4.0) that can be downloaded on a local PC and compared their performance with cloud-based state-of-the-art LLMs (ChatGPT 5, Claude Sonnet 4.5, Gemini 2.5 Pro).

They ran these SLMs on local PCs powered either by an Nvidia chip or an Apple M4 chip, as they are readily available in current high-end desktop computers ...

Then they traced the performance of these SLMs vs LLM between 2023 and October 2025 on both chat tasks and reasoning tasks.

Saad-Falcon Fig 2

The results on four benchmarks are in Saad-Falcon's Figure 2, whose caption is:

Local Models Rival Cloud Models Across Diverse Benchmarks: Individual model performance scales with size, ranging from 31.5–69.4% for IBM GRANITE 4-H-S MALL, 30.0–83.6% for GEMMA 3-12B, 51.5–80.4% for GPT-OSS-120B, and 66.5–89.5% for GEMINI 2.5 PRO . Local routing (best local LM per query) achieves 97.8%, 88.3%, 77.0%, and 92.4% on WILDCHAT, NATURAL REASONING, SUPER GPQA, and MMLUPRO respectively, sur- passing cloud routing (100%, 82.9%, 66.5%, 87.4%) on three of four benchmarks.

Klement Fig 1

Klement reformats the results into charts. First for chat tasks:

which still make up the vast majority of requests today. As you can see, in every domain, the best SLM is able to find the same or better answers than an LLM in 90% or more of the cases, with an average across all domains of 98.6%.

It is really hard to justify spending 4-6 times as much for a 1.4% improvement in performance, so LLMs are no longer really necesssary for chat tasks.

Klement Fig 2

Second for reasoning tasks:

which are obviously more demanding, SLMs are catching up fast. On average, they provide a better or at least as good an answer as LLMs in 62.5% of the cases.

SLMs may be catching up fast on reasoning tasks (see IBM's Granite 4.2 below) but it will be a while before LLMs are obsolete for these tasks.

Klement Fig 3

However, in real life as Klement points out:

the tasks for SLMs and LLMs are typically a mix of chat requests and reasoning tasks, so the third chart shows the weighted average of chat request performance and reasoning performance based on the frequency of tasks in each domain. As you can see, on average, SLMs are as good if not better than LLMs in 81.2% of the cases, with the LLMs having a significant advantage only in areas like engineering, life sciences, transportation and computer sciences.

But it’s not just accuracy. SLMs achieve this performance at energy and compute costs that are between 50% and 85% lower than for an LLM, depending on the SLM and hardware used in the computer.

Figure 7

The advantage for LLMs is being eroded quite quickly except for the extremely complex tasks. Saad-Falcon et al's Figure 7 shows how fast SLMs are catching up on reasoning tasks:

For reasoning tasks ... the pattern differs substantially. While levels 1-3 show strong improvements (+24.0, +37.8, and +53.9 pp respectively), levels 4 and 5 exhibit markedly slower progress. Level 4 improves by only +23.8 pp (7.93% to 31.72%), and level 5 remains largely unsolved with just +1.5 pp improvement (3.27% to 4.72%). This suggests that while local models have rapidly closed the gap on moderately difficult reasoning tasks, the hardest reasoning problems (those requiring either massive scale or capabilities beyond current architectures) remain a significant frontier. The presence of 134 level 5 problems (16.5% of the reasoning dataset) that remain 95% unsolved indicates substantial headroom for future model development in complex reasoning domains.

To understand why the rate at which SLMs catch up is critical we need to study Groundbreaker's The Teaser Period: Why the AI Boom Is Built to Break, which starts from the analogy between the subprime crisis and the AI bubble:

Paulson & Co. laid out the arithmetic that same month in a comment letter to the FDIC: Over 80% of recent subprime originations, it observed, were two- or three-year adjustable-rate products. The average subprime borrower’s mortgage payments already consumed roughly 40% of their gross income at the teaser rate. Almost none of them could service the reset rate out of income.

The crisis, in other words, was written in advance by the instruments themselves. The market looked at the reset wall and kept buying, because every participant believed the exit would arrive before the reset: home prices would keep appreciating and the borrower would refinance into a fresh teaser before the old one expired.

Source

Groundbreaker describes the AI bubble's version of the subprime Adjustable Rate Mortgage (ARM):

The take-or-pay compute contract - the instrument at the center of the AI build-out - has a structural feature that almost no one prices: its payments do not begin at signing. They begin at delivery. A lab signs a multi-year capacity commitment today, but the payments do not start until the data center is energized, the capacity is accepted, and the contractual ramp schedule commences - an interval set not by finance, but by construction: siting, powering, and filling a gigawatt-scale campus takes 24-to-36 months from signature - mirroring the two-to-three-year teaser of a subprime ARM.

More than $2.3 trillion of compute contracts now sit on the books of the four largest American cloud providers as remaining performance obligations and contracted backlog - signed, celebrated, capitalized into equity prices, and, critically, not yet billing.
...
The parallel to 2006 is exact and it explains the single most-cited absurdity of this cycle: How does OpenAI, a company with some $40 billion of run-rate revenue, sign $1.4 trillion of compute commitments? The same way a household with $60,000 of income signed a $600,000 mortgage: because the terms at signing do not require the payment yet, and because everyone at the table - borrower, lender, and the market - believes the growth will arrive before the payment does.

The chart shows that, in 2027 and 2028, the AI platforms will need to shell out $852B in cash for compute, whether they use it or not. We don't know how much revenue they are currently generating, but they want us to believe it is in the region of $100B. Ignoring all their other costs, they have to increase their revenue more than 4x next year to cover their contractual payments for compute. That means they have to extract at least $400B from the enterprise market in return for supplying it with technology that is slightly better than technology companies can run in-house around an order of magnitude cheaper.

What matters isn't the relative price/performance of in-house SLMs versus remote LLMs now, it is their relative price/performance when the LLMs' contractual compute payments come due, i.e. next year. It seems very unlikely that companies already balking at the cost of the AI platforms' products by moving to Chinese models will increase their spend 4x next year. It seems equally unlikely that investors will give the AI platforms a few hundred million dollars next year to burn so as to postpone the day of reckoning by another 12 months.

The Stanford study collected data in October 2025. Developments since, with more to come, have already significantly increased the price/performance advantage of SLMs. The include:

• Nvidia's RTX Spark "superchip", disucssed by @pramodchandrayan in NVIDIA Just Put a 120-Billion-Parameter AI Model in Your Laptop. Here’s What That Actually Changes.:
  

  At Computex 2026 in Taipei on June 1st, CEO Jensen Huang announced the RTX Spark superchip — a single piece of silicon that combines a 20-core Arm CPU, a Blackwell GPU with 6,144 CUDA cores, and 128 gigabytes of unified memory, connected by NVIDIA’s NVLink chip-to-chip interconnect. The whole package delivers up to one petaflop of AI compute in a laptop form factor.
  
  The number that matters: RTX Spark can run a 120-billion-parameter language model entirely locally, with a context window of one million tokens, without a single byte leaving your machine.

  This is the guts of Nvidia's $5.2K DGX Spark desktop. It was Portent #20
• The Exxact Valence Nvidia DGX Station discussed by Michal Malewicz in NVIDIA just killed big AI and… You’re the winner?:
  

  Here’s a $94,011.50 desktop computer for sale right now.
  
  It’s not a server rack or some crazy cloud infrastructure monstrosity.
  
  It’s a (rather big) tower PC. Kinda like the one you played Cyberpunk 2077 on.
  
  It sits under a desk and plugs into a wall like a regular desktop. The only difference is that it runs trillion-parameter AI models with no API keys, no per token payment, and no personal data leaving the room.

  It is 18 times as expensive as the DGX Spark but can run models more than 8 times bigger. This was Portent 31.
• Michael Nuñez' Perplexity partners with Nvidia to launch Portable Computer, a fully local AI agent with zero token costs:

  Perplexity is launching Portable Computer today, a version of its agentic "Computer" platform that runs entirely on hardware users already own — starting with Nvidia's DGX Spark desktop supercomputer and Linux machines equipped with Nvidia RTX GPUs.
  
  The launch, developed in close partnership with Nvidia, is one of the most aggressive attempts yet to move serious AI agent workloads off the cloud and onto local devices. The model, the user's files, and the work itself can all stay on the machine. Work completed locally consumes no billing credits, and the company says every task starts on the device by default — with the system asking permission before sending any individual step to a more powerful frontier model in the cloud.

  This was Portent 45.
• Apple's announcement of the latest Mac hardware, discussed by Samuel Axon in Apple’s new desktop computers are designed specifically for local AI development:
  

  Apple announced new iterations of both desktops, along with two new chips: the M6, the first 2nm chip in Apple’s M-series lineup for Macs, and the M5 Ultra, now the most powerful chip in the lineup for most things—especially AI workloads.
  
  There aren’t any major new features for either machine. This is just a specs bump. But based on how Apple is presenting these refreshes, they’re leaning hard into those use cases, which weren’t even a thought when earlier iterations were first engineered.
  
  The devices’ popularity for production inference took off after macOS 26.2 shipped last December. According to Apple’s release notes, 26.2 enabled “low-latency communication between Thunderbolt 5 hosts for use cases including distributed AI inference using MLX.” Thunderbolt 5 is a very fast wired data connection, and MLX is an open source array framework designed to help machine learning workflows take full advantage of the M-series chips’ unified memory.
  
  Since then, both hobbyists and professional developers and researchers have been essentially daisy-chaining Mac minis or Mac Studios to run inference on local large language models that are much bigger than anything that could run a single mass-market device—providing an alternative to ultra-beefy specialized hardware featuring specialized Nvidia GPUs.
  

  This was Portent 46.
• IBM's latest model release, discussed by Samuel Axon in IBM’s new Granite 4.2 models ride the wave of interest in local LLMs:
  

  IBM has rolled out the newest models in its family of open-weight large language models designed to be downloaded and self-hosted. The newly launched Granite 4.2 comes in 3B, 8B, and 30B parameter variants.
  
  Like previous versions, IBM is taking a decoder-only approach here. These new releases offer a 128,000-token context window natively. The 8B and 30B variants (not the 3B one) also go through an agentic reinforcement-learning block; they were trained for expanded capabilities like using the terminal, searching the web, or using external tools. The 3B model supports tools too, but without the same level of specialized training.
  
  Beyond those tweaks, this release is particularly notable because, as IBM itself writes, “Granite 4.2 is the reasoning-focused release of the Granite language-model family.

  This was Portent 47.

Consider not just what Ding et al's graph of the narrowing performance gap between closed= and open-weight models, but also what Saad-Falcon et al's Figure 7 will look like after another year of both hardware and software developments like these.

Saad-Falcon et al argue that it isn't just the raw price/performance that advantages local compute:

System-level benefits offset per-query efficiency disadvantages. While cloud accelerators demonstrate 1.4× to 7.4× higher intelligence efficiency per query, local deployment provides complementary system-level benefits that offset this disadvantage. Local inference avoids datacenter infrastructure costs, network latency, and API pricing, while enabling 88.7% of queries that local models can handle correctly to bypass cloud compute entirely. As demon- strated in Section 4.3, intelligent routing between local and cloud infrastructure can achieve 60–80% reductions in total energy, compute, and cost compared to cloud-only deployment, even when local accelerators are individually less efficient. These findings suggest that the path to efficient AI infrastructure lies not in local accelerators matching cloud efficiency, but in routing systems that leverage the complementary strengths of both paradigms: local processing for the majority of straightforward queries and cloud infrastructure for the minority requiring frontier model capabilities.

It looks increasingly as though Klement is right that the hyperscalers are toast, because the vast majority of inference will happen locally while the massive data centers will be used only for training and for inference on massive models so expensive that almost no-one can afford them.

Update 4^th September 2026

In the three days since I posted this, I have collected more evidence:

• Thomson Reuters built its own AI model on Chinese open-source tech to slash AI costs by Cris Tolomia show a intellectual property company taking aggressive steps to move AI in-house:
  

  Thomson Reuters launched its first in-house large language model on Monday, a move the company says will reduce its reliance on outside AI providers, including Anthropic.
  
  According to Business Insider, Thomson sits atop Snowdon, an intermediate model that Thomson Reuters built by reworking Qwen, an open-source offering from Chinese tech giant Alibaba. A joint team from Thomson Reuters and Imperial College London adapted Qwen over several months to ensure it was "ethically and politically de-biased and safe to use," Chief Technology Officer Joel Hron said.
  
  Thomson Reuters spent roughly $40 million over two years on personnel and computing, the company said. The final training run cost approximately $450,000, according to SiliconAngle. The company chose to forgo developing a foundation model from the ground up, instead taking an existing open-weight model as its starting point and enriching it with proprietary content, specialized training methods, and domain expertise.

• Kyle Orland's Four major AI models suffer rare overlapping downtime shows the risk of outsourcing business-critical operations to shared infrastructure:
  

  Cloud-based AI models operated by OpenAI, Anthropic, xAI, and Google suffered a rare and overlapping set of significant service interruptions over a period of hours Thursday morning.
  
  Anthropic first reported a “partial outage” related to “elevated errors on requests to Claude Mythos 5.1, Claude Fable 5.1, and Claude Opus 5” at 9:23 am (all times Eastern). The company reported that it had “identified the cause” of the error roughly 15 minutes later, before reporting that “a fix has been deployed” and the issue was resolved by 12:16 pm. A separate incident report indicated “elevated errors on requests to Claude Sonnet 5” for a brief period just after noon.
  
  OpenAI, meanwhile, reported “elevated errors across ChatGPT and Codex” were resulting in “degraded performance” as of 10:43 am Thursday morning. A mitigation put in place a little more than half an hour later led to the issue being marked as “resolved” by 12:55 pm.
  
  As of this writing, xAI’s Grok currently displays a user-facing error message that the model “is experiencing issues” and that the company is “working on restoring service as quickly as possible.” User-submitted reports from DownDetector regarding Grok shot up from less than 10 just before 9 am to 1,365 by 9:45 am and have fallen to 273 as of this writing.
  
  While Google has not publicly acknowledged any issues with its Gemini model, reports of problems on DownDetector similarly spiked from just 23 around 10:30 am to 412 just after 11 am. Data collected by API checker StatusGator also shows what it terms a “likely outage” for the Gemini API between 10:45 and 11:15 a.m. before the status returned to normal.
  

  Other Internet services did not experience problems, so this seems to have been solely an AI problem.
• Nvidia announced NVIDIA Personal AI Router (PAIR):
  

  NVIDIA Personal AI Router (PAIR) is software that connects compatible macOS, Windows, and Linux systems with NVIDIA RTX™GPUs and DGX Spark systems into a personal home AI cluster. PAIR distributes local AI inference workloads across available devices while keeping prompts, files, and agent context on the user’s home network.

• Perplexity announced Optimizing On-Device Inference for Apple Silicon:
  

  Hybrid Compute on Apple silicon orchestrates a task between frontier intelligence in the cloud and a local model on the Mac. Cloud models handle research and reasoning, while a local model works with private files and apps on the Mac.
  
  For this division of labor to feel seamless, local inference must keep pace with the rest of the task. That requires an engine that can process prompts quickly and sustain a high token-generation rate.
  
  Lily, our lightweight local inference engine, is built specifically for Apple silicon and Qwen3.6-35B-A3B, with separate optimizations for prefill and decode. A standalone demo is publicly available on GitHub.

  Pair and Lily illustrate the rapidly growing trend of moving as much inference as possible to local hardware and, relatedly, the importance of placinng a router bewteen the user and the inference systems.

Update 14^th September 2026

Alphavillian Toby Nangle's coverage of this issue in How big is the open-model threat to AI hyperscalers? is excellent. He frames the issue thus:

One of the ‘known unknown’ threats that S&P Global Ratings outlined last week hanging over the biggest companies driving the AI juggernaut is the notion that open-weight models close the performance gap with expensive frontier models.

We can break this down further into two component risks. First: that some queries put to frontier LLMs — which require centralised cloud infrastructure — can be answered more quickly and more cheaply by locally hosted set-ups, fitting on something maybe only a little larger than Balakrishnan’s Raspberry Pi. And second: that big complex open-weight models become pretty indistinguishable from big complex closed-weight models, can be accessed at a fraction of the cost, and come with riders that are valuable in their own right.

As regards the small local models, Nangle both conducts his own experiment and also relies on the Saad-Falcon et al paper. He concludes:

So we can see how SLMs might appeal to management seeking to repair their bottom lines and reputations for cost control after token-maxxing experiments backfired so spectacularly.

In fact, given quite how basic most AI queries appear to be, the researchers’ results sound like the sort of thing that might even challenge the economics of building some of the myriad mahoosive data centres being planned.

As regards the open-weight LLMs, he acknowledges that relative to the frontier closed-weight models their performance lag is shrinking:

Cloud-based open-weight models, just like their closed-weight counterparts, gobble up data centre processing capacity. As such, it’s hard to see what direct problem they might pose to the economics of data centre build-out. Although it’s easier to see how they might be a problem for AI labs like Anthropic and OpenAI.
...
Good luck though trying to download Moonshot’s Kimi K3 on to your PC. Operating across 2.8tn parameters, it’s just a different animal. According to Citi Research, Kimi K3 trounces every closed-weight frontier model ever built prior to [checks calendar] three months ago.

Source

It is clear that almost as good but vastly cheaper is winning in the market:

One way to see which way the wind is blowing on open-model versus closed-model usage is by looking at data from router firms like OpenRouter, the New York start-up that Stripe agreed to buy last month.
...
Increasingly, the share of queries that are truly closed-weight frontier-model-worthy is declining. Back at the start of the year, three-fifths of its queries were routed through to closed-weight proprietary models. The latest share is just a quarter.

Among the companies Nangle cites as moving to open-weight models are DoorDash, Siemens, Airbnb, AT&T, Latham & Watkins and other major law firms. As with Thomson-Reuters, many are concerned with security:

Tareq Islam, a strategic adviser to ApexE3, a capital markets AI infrastructure firm, tells Alphaville that some London-based asset managers, as well as not wanting to build dependency on any single proprietary model, are reluctant to post their most confidential data and intellectual property to closed-model providers. Clients of ApexE3 include Vanguard, the world’s second-largest asset manager.

This concern will only grow after:

OpenAI claimed to have found a “singularity” in the Navier-Stokes equations in three dimensions, one of the six unsolved “Millennium Prize Problems” set by the Clay Mathematics Institute. In a statement posted to X.com the lab wrote, “The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.”

A few hours earlier, Tristan Buckmaster, a professor of mathematics at NYU who has been working on closely related problems for some time, posted his own statement in which he went to great lengths to NOT EXACTLY formally accuse OpenAI of stealing an almost complete proof of Navier-Stokes from his sessions on Codex, GPT-5.6 Sol, and Astra. But he did not sound at all happy, and it’s worth a read in full if you’ve got five minutes.