Dark GPUs

Laying dark fiber
Stealth Communications CC-BY-SA

Wikipedia's Dark Fibre article starts:

Dark fibre originally referred to the potential network capacity of telecommunication infrastructure. Because the marginal cost of installing additional fibre optic cables is very low once a trench has been dug or conduit laid, a great excess of fibre was installed in the US during the telecom boom of the late 1990s and early 2000s. This excess capacity was later referred to as dark fibre following the dot-com crash of the early 2000s that briefly reduced demand for high-speed data transmission.

These unused fibre optic cables later created a new market for unique private services that could not be accommodated on lit fibre cables (i.e., cables used in traditional long-distance communication).

Many are analogizing the current AI bubble to the dot-com era, pointing out that although the dark fiber eventually became very valuable infrastructure, the companies that invested in laying it suffered. Wikipedia notes that the buildout:

was based on the assumption that telecoms traffic, particularly data traffic, would continue to grow exponentially for the foreseeable future. The advent of wavelength-division multiplexing reduced the demand for fibre by increasing the capacity of a single fibre by a factor of as much as 100.

According to Gerry Butters, the former head of Lucent Technologies's Optical Networking Group at Bell Labs, the amount of data that could be carried by an optical fibre was doubling every nine months at the time. This progress in the ability to carry data over fibre reduced the need for more fibres. As a result, the wholesale price for data communications collapsed and a number of these companies filed for bankruptcy protection. Global Crossing and Worldcom are two high-profile examples in the United States.

The analogy being suggested is that the AI data centers will eventually be valuable infrastructure even if delays in construction and lack of real demand mean that the companies building them will suffer.

Below the fold I investigate this analogy.

Crowding Out

In Downgrades and Competition For Capital I discussed how AI's insatiable appetite for funds was raising interest rates not just for AI-related debt, but also for everyone else. The impact of these rate increases will take time to become evident, as borrowers roll over existing debt or raise new debt.

Below the fold I look at impacts that we don't have to wait for, because they are already happening on a large scale.

Competition For Capital

Exhibit 1

The same day I posted Downgrades Peter Oppenheimer et al from Goldman Sachs (GS) weighed in on the same topic with Competition for Capital:

"There are two themes dominating our investor conversations: the impact of AI and the rise in interest rates. The two issues are linked as demand for capital from both the private and public sectors increase. In the private sector a surge in capex spending to fund AI infrastructure has eaten into free cash flow and forced companies to raise more in debt and equity markets. Meanwhile, government borrowing needs have increased as priorities shift towards upgrading critical infrastructure, energy security and defense at a time when cyclical inflationary pressures driven by higher energy prices are also resulting in higher policy rates. The combination has pushed up the cost of capital. As recently as 2022, for example, 30-year bond yields in Germany and Japan were close to zero ... A rise in yields, together with more uncertainty (over geopolitics and the future impact of AI) have, in combination, pushed up the cost of capital."

As one would expect from the professionals there is a lot to digest in their report. Below the fold I discuss the details.

Downgrades (Updated)

10-year Treasury yield

Recently, interest rates for government debt have been rising worldwide. As I write the yield of the 10-year US treasury is over 5%. It had been below 4.5% for over a year when, in March, it started a steady rise. This is despite (and probably because of) Scott Bessent's efforts to force it lower, which JP Morgan compared to 'paying your mortgage with your credit card'.

Source

A big part of the reason is governments persistently borrowing to cover deficits, with no plan to reduce them. For example, the Trump administration has borrowed more than the growth in US GDP. Projections are that by the end of their term they will have borrowed $1.40 for every $1 growth in GDP. Half of that GDP growth is from the AI bubble. Remove it and the administration has already borrowed almost 2.5 times the GDP growth.

But another part of the reason is that governments are facing competition for the available funds from other highly-rated borrowers. Hyperscalers such as Microsoft (AAA) and Alphabet (AA+) can no longer pay cash for the immense sums they believe they need in order to keep up with their competitors in the race for AGI. So the bond market is facing a significant increase in demand, which naturally increases interest rates.

Below the fold I look into how this competition for funds is going.

EatingThe Seed Corn (Updated)

The Village Idiom explains:

In the idiom "eat one's seed corn", the phrase refers to consuming or disposing of valuable resources without considering the long-term consequences. It implies a short-sightedness and lack of foresight, often resulting in detrimental effects on future prospects or success.

Salomé Balthus
Uwe Hauth, CC BY-SA 4.0

More than a year ago in Going Out With A Bang I quoted Salomé Balthus:

'The elephant in the room is climate change. Everyone knows it can't be prevented any more,' she said, adding that the 'super rich' could generally be split into two groups on the topic.

'The one group thinks it only affects the poor, the "not-white race", while the others fear that it could get worse but there's no sense in trying to do anything about it so they just enjoy themselves,'

In fact both groups have decided to steal everything they can get their hands on to fund their retreat to bunkers on private islands protected by armed autonomous humanoid robots. Below the fold I look at yet another example of this trend.

Small Is Beautiful (Updated 2x)

Estimates are that, to justify the AI platforms' enormous capex plans, by 2030 they need to be generating around $2T/year in revenue. If every adult resident of the US spent $20/month on AI, it would generate $68.5B/year. Clearly, only the enterprise market stands even a remote possibility of generating the bulk of the $2T.

There are three major threats to the prospect of AI platforms extracting 6% of current US GDP from the enterprise market, and thus to OpenAI's and Anthropic's ambitions to IPO in the near future. First, faced with AI's Affordability Crisis, companies have been placing strict limits on employees' spending on AI tokens.

Source

Second, the gap in performance between expensive, closed-weight US models, such as OpenAI's and Anthropic's, and much cheaper, open-weight Chinese models has been rapidly closing, with the result that the US models are losing enterprise market share. Luz Ding, Spe Chen and Hayley Warren analyze this in US Lead in the AI Race With China Is Rapidly Narrowing:

Bloomberg in partnership with researchers at Vals AI, an independent AI evaluation and benchmarking platform, tested seven models from frontier Chinese and US companies to see how they performed in a real-world task. They were asked to create a fictional coffee e-commerce site called Brewberg using the same prompts. Most of the models scored 100% functional accuracy despite occasional design misses, but with very different price tags. The experiment employed the top performing models in July from Anthropic and all the Chinese firms, as well as more affordable models from OpenAI and Google.

They all did reasonably well, but the two best were Claude Fable 5 at $48.99 and Kimi K3 at $11.99. Chinese models charging much less for almost the same performance are grabbing market share:

the use of Chinese models overtook US platforms globally for the first time in June, and accounted for more than 60% of market share last month, on OpenRouter, a tech platform that offers software developers access to hundreds of AI models. It is a widely watched gauge of model usage despite tracking just a fraction of global AI consumption. The US, parts of Europe and Asia now favor Chinese labs, according to the same data.

On Hugging Face, Chinese AI models account for 41.4% of generative model downloads among developers, 5 percentage points higher than US models.

Third, it isn't just that the Chinese models are cheaper to run remotely, but also that because they are open-weight they can be run on affordable in-house systems, which means that:

• They don't give Donald Trump a kill-switch for your busines.
• They don't give Sam Altman or Dario Amodei a kill-switch for your busines.
• They don't require giving the Chinese, Sam Altman or Dario Amodei all your business' critical data.
• They provide visibility into and control over AI costs.
• They are even cheaper.

The question is "compared to the closed-weight US models, what do you lose by running open-weight models in-house?" Below the fold I discuss a major study from Stanford that answers the question.

Not A Bug But A Feature

Source

Datafinnovation are out with more impressive research described in a blog post entitled Losing The Blacklisting Race and a paper entitled Enforcement Speeds In DeFi: Limits to a Race Against Time by Ben Charoenwong et al.

The blog post is more accessible, so that's what I will quote from. But for the details you need to read the paper. The blog post's TL;DR is:

First, we explore the empirical reality of OFAC’s blacklisting efforts. The results there are stark: nearly all addresses are completely empty by the time OFAC manages to blacklist them. This result is robust across time, attacker and type of attack. Second, we construct a simple model to explain why this outcome is inevitable with competent attackers. Evasion strategies which work in our model match those we find employed in the wild.

Below the fold, I start from Datafinnovation's work and explore its context.