Accelerated Computing

Source

One big problem of the economies of the US and the UK is the cult of the CEO, and the resulting flood of CEO hagiographies that appear after a surge in their company's stock price. These aren't harmless fluff pieces, they contribute to a CEO mindset that is profoundly destructive — see Elon Musk for one example. Will Hutton writes:

But decades of being congratulated and indulged for the relentless pursuit of their own self-interest has turned the heads of too many of our successful rich. They really believe that they are different: that they owe little to the society from which they have sprung and in which they trade, that taxes are for little people. We are lucky to have them, and, if anything, owe them a favour.

Below the fold I continue the "Old man yells at cloud" theme of recent posts by trying to clarify an aspect of the current Jensen Huang hagiography.

Ed Zitron's must-read The Shareholder Supremacy traces the idea of the super-hero CEO revealed by a soaring stock price to:

the famous Dodge vs. Ford Motor Co. case that would define — and ultimately doom — modern capitalism, and in many ways birth the growth-at-all-costs Rot Economy.

The Michigan Supreme Court found that "a business corporation is organized and carried on primarily for the profit of the stockholders [and that] the powers of the directors are to be employed for that end," and intimated that cash surpluses should not be saved to invest in upcoming projects, but distributed to shareholders, because Ford had shown that it was good at making money. Ford was directly forbidden from lowering prices and raising employee salaries, and forced to issue a dividend.

To be clear, the statement around corporations’ duty toward shareholders was made “obiter dicta.” This means it was not actually legally binding, despite over a hundred years of people acting as if it was.

Zitron goes on to detail how "Neutron" Jack Welch destroyed General Electric while being lionized as a super-star CEO. I can testify to his malign influence because, during my time at Sun Microsystems, Scott McNealy was seduced by it.

Now on to the latest CEO whose soaring stock price has caught the attention of the CEO hagiographers, Jensen Huang. Don't get me wrong. I was there when he was a really good startup CEO, and unlike many others he has grown with the company in a very impressive way.

In The Envy of Everyone, M. G. Siegler remarks on:

this incredibly prescient profile of co-founder and CEO Jensen Huang back in 2017 by Andrew Nusca (in naming him Fortune's 2017 'Businessperson of the Year').

At the time Siegler tweeted:

“Video games was our killer app — a flywheel to reach large markets funding huge R&D to solve massive computational problems.”

Genius foresight. Sounds obvious now. Was not then.

Siegler was commenting on Andrew Nusca's 2017 profile entitled This Man Is Leading an AI Revolution in Silicon Valley—And He’s Just Getting Started. As Huang hagiography goes, it was pretty good. The quote in the tweet is from Jensen Huang talking about the early days:

“We believed this model of computing could solve problems that general-purpose computing fundamentally couldn’t,” Huang says. “We also observed that video games were simultaneously one of the most computationally challenging problems and would have incredibly high sales volume. Those two conditions don’t happen very often. Video games was our killer app—a flywheel to reach large markets funding huge R&D to solve massive computational problems.”

I don't disagree with what Huang said. Despite the need to focus on gaming, we did have a vague idea that in the future there would be other application areas in which custom accelerators could make an impact. And it is true that Nvidia's VCs, Sutter Hill and Sequoia, gave us the time to develop a multi-generation architecture rather than rushing out a "minimum viable product". I do quibble with the idea that this was "genius foresight".

90s System Diagram

Even back when I was working with Curtis Priem and Chris Malachowsky on Sun's GX graphics chips the problem we were solving was that the CPU could not write pixels into the framebuffer fast enough for any kind of 3D application. So a chip, then called a graphics chip but now called a GPU, was needed between the CPU and the framebuffer capable of converting a small number of writes from the CPU into a large number of writes to the framebuffer. Designs faced three major performance constraints:

• The limited bandwidth of the bus carrying instructions from the CPU to the graphics chip. I wrote about how we addressed this constraint in Engineering For The Long Term.
• The limited write bandwidth of the framebuffer.
• The limited transistor budget imposed by the chip's cost target.

NV1-based Diamond Edge
Swaaye, CC-By-SA 3.0

Six months after we started Nvidia, we knew of over 30 other companies all trying to build 3D graphics chips for the PC. They fell into two groups. Nvidia and a few others were making fixed-function accelerators, but most were trying to get faster time-to-market by making programmable accelerators in which the functions were implemented in software running on a CPU in the graphics chip.

One problem for the programmable companies was that the transistors needed to implement the graphic chip's CPU's fetch, decode, execute system were not available to implement the actual graphics functions, such as matrix multiply. A second problem was that the CPU in the graphics chip needed to fetch its instructions forming the program that defined the functions from some memory:

• It could fetch them from the system memory that the CPU got its data and instructions from. That wasn't a good idea since (a) it required implementing a DMA engine in the graphics chip, and (b) the DMA engine's program fetches would compete for the limited bandwidth of the bus.
• It could fetch them from a separate program memory private to the graphic chip. This wasn't a good idea since it added significant cost, both for the separate memory itself and also for the extra pins and complexity of the graphics chip's extra memory port.
• It could fetch them from the framebuffer memory. This wasn't a good idea since the program fetches would compete with the limited bandwidth of the framebuffer RAM's random access port.

The result was that for many product generations from NV1 the winning graphics chips were all fixed-function. At the time Nvidia was started, a person having ordinary skill in the art should have understood that fixed-function accelerator hardware was the way to go. We were not geniuses.

Over the years, Moore's Law gradually relaxed the constraints, forcing the design choices to be re-evaluated. As we had expected, the one that relaxed fastest was the transistor budget. The obvious way for the accelerator to exploit the extra transistors was to perform functions in parallel. For parallel applications such as graphics this increased the accelerator advantage over the CPU. But equally important in the competitive market for graphics chips, it sped up the design by making the chip an assembly of many identical units.

The constraints continued to relax. In 2001 Nvidia had enough transistors and bandwidth to release the GeForce 3 series with programmable pixel and vertex shaders. Eventually, various non-graphics communities figured out how to use the increasing programmability of Nvidia's GPUs to accelerate their parallel applications, and Nvidia seized the opportunity to create a software platform with the 2007 release of CUDA. By greatly simplifying the development of massively parallel applications for Nvidia GPUs, CUDA drove their adoption in many fields, AI being just the latest.

Source

The Huang hagiography's focus on Nvidia's current stock price is misplaced and ahistorical. It is notoriously volatile. Look at the log plot of the stock price since the IPO. I count eight drops of 45% or more in 25 years, that's average of about one every three years. One of the questions I asked when interviewing Chris Malachowsky for the 50^th Asilomar Microcomputer Workshop was approximately "how do you manage a company with this much volatility?" His answer was, in effect, "you learn to ignore it".

Source

David Cahn of Sequoia, one of Nvidia's two VCs, writes in AI’s $600B Question:

In September 2023, I published AI’s $200B Question. The goal of the piece was to ask the question: “Where is all the revenue?”

At that time, I noticed a big gap between the revenue expectations implied by the AI infrastructure build-out, and actual revenue growth in the AI ecosystem, which is also a proxy for end-user value. I described this as a “$125B hole that needs to be filled for each year of CapEx at today’s levels.”
...
The $125B hole is now a $500B hole: In the last analysis, I generously assumed that each of Google, Microsoft, Apple and Meta will be able to generate $10B annually from new AI-related revenue. I also assumed $5B in new AI revenue for each of Oracle, ByteDance, Alibaba, Tencent, X, and Tesla. Even if this remains true and we add a few more companies to the list, the $125B hole is now going to become a $500B hole.

Goldman Sachs' provides furtehr skepticism in Gen AI: Too Much Spend, Too Little Benefit?:

Tech giants and beyond are set to spend over $1tn on AI capex in coming years, with so far little to show for it. So, will this large spend ever pay off? MIT’s Daron Acemoglu and GS’ Jim Covello are skeptical, with Acemoglu seeing only limited US economic upside from AI over the next decade and Covello arguing that the technology isn’t designed to solve the complex problems that would justify the costs, which may not decline as many expect.

Another huge drop in the stock price is sure to be in the offing. How great will the hagiographers think Huang is then?

Company	Market	Quarterly	Employees	MktCap per	Income per
	Cap	Income		Employee	Employee
NVDA	$3T	$26B	30K	$100M	$867K
GOOG	$2.3T	$25B	185K	$17M	$140K
AAPL	$3.2T	$28B	160K	$20M	$170K
MSFT	$3.3T	$22B	221K	$15M	$100K

Mr. Market will do what Mr. Market does, the stock price isn't under Huang's control. The things that are under Huang's control are the operating profit margin (53%), revenues ($26B/quarter), and the company's incredible efficiency. Nvidia's peers in the $2-3T market cap range have between 5 and 7 times as many employees. As Huang says, Nvidia is the smallest big company.