The Great MEV Heist

The Department of Justice indicted two brothers for exploiting mechanisms supporting Ethereum's "Maximal Extractable Value" (MEV). Ashley Berlanger's MIT students stole $25M in seconds by exploiting ETH blockchain bug, DOJ says explains:

Anton, 24, and James Peraire-Bueno, 28, were arrested Tuesday, charged with conspiracy to commit wire fraud, wire fraud, and conspiracy to commit money laundering. Each brother faces "a maximum penalty of 20 years in prison for each count," the DOJ said.

The alleged scheme was launched in December 2022 by the brothers, who studied at MIT, after months of planning, the indictment said. The pair seemingly relied on their "specialized skills" and expertise in crypto trading to fraudulently gain access to "pending private transactions" on the blockchain, then "used that access to alter certain transactions and obtain their victims’ cryptocurrency," the DOJ said

Below the fold I look into the details of the exploit as alleged in the indictment, and what it suggests about the evolution of Ethereum.

Background

Lets start with some history. The key issue with MEV is that the architecture of decentralized cryptocurrencies enables a form of front-running, which Wikipedia defines thus:

Front running, also known as tailgating, is the prohibited practice of entering into an equity (stock) trade, option, futures contract, derivative, or security-based swap to capitalize on advance, nonpublic knowledge of a large ("block") pending transaction that will influence the price of the underlying security. ... A front running firm either buys for its own account before filling customer buy orders that drive up the price, or sells for its own account before filling customer sell orders that drive down the price. Front running is prohibited since the front-runner profits from nonpublic information, at the expense of its own customers, the block trade, or the public market.

Note that the reason it is illegal in these markets is that, at the time the front-runner enters their order, the customer's order is known only to them. It is thus "material non-public information". Arguably, high-frequency traders front-run by placing their computers so close to the market's computers that the information about orders on which they trade has not in practice had time to "become public".

I wrote about front-running in cryptocurrencies, describing how it was different, in 2020's The Order Flow:

In order to be truly decentralized, each miner must choose for itself which transactions to include in the next block. So there has to be a pool of pending transactions visible to all miners, and thus to the public. It is called the mempool. How do miners choose transactions to include? Each transaction in the pool contains a fee, payable to the miner who includes it. Miners are coin-operated, they choose the transactions with the highest fees. The mempool concept is essential to the goal of a decentralized, trustless cryptocurrency.

Source

The pool of pending transactions is public, thus front-running is arguably legal and anyone can do it by offering a larger fee. Ethereum's block time is 12 seconds, plenty of time for bots to find suitable transactions in the mempool. It normally contains a lot of pending transactions. Ethereum is currently processing about 1.12M transactions/day (46.7K/hr) and there are around 166K pending transactions, or about 3.6 hours worth. Bitcoin is processing about 700K transactions/day and there are normally around 100K transactions in the mempool, or 3.5 hours worth.

Arguably, this is analogous to high-frequency trading, not front-running by brokers. In The Order Flow I recount how the prevalence of high-frequency trading led institutions to set up dark pools:

When conventional “lit” markets became overrun with HFT bots, investment banks offered large investors “dark pools” where they could trade with each other without the risk of being front-run by algos. But Barclays allowed HFT bots into its dark pool, where they happily front-run unsuspecting investors who thought they were safe. Eventually Barclays was caught and forced to drain its dark pool. In 2016, it was fined $70 million for fraud. It was not the only large bank that accepted money from large investors to protect them from HFT bots and money from HFT traders to allow them access to the investors it was supposed to be protecting.

The Order Flow was in large part sparked by two accounts of attempts to avoid being front-run:

• Ethereum is a Dark Forest by Dan Robinson and Georgios Konstantopoulos:
  

  In the Ethereum mempool, these apex predators take the form of “arbitrage bots.” Arbitrage bots monitor pending transactions and attempt to exploit profitable opportunities created by them. No white hat knows more about these bots than Phil Daian, the smart contract researcher who, along with his colleagues, wrote the Flash Boys 2.0 paper and coined the term “miner extractable value” (MEV).
  
  Phil once told me about a cosmic horror that he called a “generalized frontrunner.” Arbitrage bots typically look for specific types of transactions in the mempool (such a DEX trade or an oracle update) and try to frontrun them according to a predetermined algorithm. Generalized frontrunners look for any transaction that they could profitably frontrun by copying it and replacing addresses with their own.

  Their attempt to rescue about $12K failed because they didn't know a miner and thus couldn't avoid the dark forest in the mempool, where a front-runner bot found it.
• And Escaping the Dark Forest, Samczsun's account of how:

  On September 15, 2020, a small group of people worked through the night to rescue over 9.6MM USD from a vulnerable smart contract.

  The key point of Samczsun's story is that, after the group spotted the vulnerability and built a transaction to rescue the funds, they could not put the rescue transaction in the mempool because it would have been front-run by a bot. They had to find a miner who would put the transaction in a block without it appearing in the mempool. In other words, their transaction needed a dark pool. And they had to trust the cooperative miner not to front-run it.
  
  Ths attempt succeeded because they did know a miner.

Reading both is essential to understand how adversarial the Ethereum environment is.

The 2019 paper that published the MEV concept was Flash Boys 2.0: Frontrunning, Transaction Reordering, and Consensus Instability in Decentralized Exchanges by Philip Daian et 7 al:

In this work, we explain that DEX [decentralized exchanges] design flaws threaten underlying blockchain security. We study a community of arbitrage bots that has arisen to exploit DEX flaws. We show that these bots exhibit many similar market-exploiting behaviors— frontrunning, aggressive latency optimization, etc.—common on Wall Street, as revealed in the popular Michael Lewis expose´ Flash Boys. We explore the DEX design flaws that spawned arbitrage bots, measure and model these bots’ behavior, and illuminate systemic smart-contract ecosystem risks implied by our observations.

Daian and co-authors describe five pathologies: Pure revenue opportunities, Priority gas auctions (PGAs), Miner-extractable value (MEV), Fee-based forking attacks, and Time-bandit attacks. Their results find two surprises:

First, they identify a concrete difference between the consensus-layer security model required for blockchain protocols securing simple payments and those securing smart contracts. In a payment system such as Bitcoin, all independent transactions in a block can be seen as executing atomically, making ordering generally unprofitable to manipulate. Our work shows that analyses of Bitcoin miner economics fail to extend to smart contract systems like Ethereum, and may even require modification once second-layer smart contract systems that depend on Bitcoin miners go live.

Second, our analysis of PGA games underscores that protocol details (such as miner selection criteria, P2P network composition, and more) can directly impact application-layer security and the fairness properties that smart contracts offer users. Smart contract security is often studied purely at the application layer, abstracting away low-level details like miner selection and P2P relayers’ behavior in order to make analysis tractable ... Our work shows that serious blind spots result. Low-level protocol behaviors pose fundamental challenges to developing robust smart contracts that protect users against exploitation by profit-maximizing miners and P2P relayers that may game contracts to subsidize attacks

Because it promised profits, MEV became the topic of a lot of research. By 2022, in Miners' Extractable Value I was able to review 10 papers about it.

Then came Ethereum's transition to Proof-of-Stake. As usual, Matt Levine provides a lucid explanation of the basics:

How does the blockchain decide which transactions to record, and in what order? In Ethereum, the answer is: with money. People who want to do transactions on the Ethereum network pay fees to execute the transactions; there is a flat base fee, but people can also bid more — a “priority fee” or “tip” — to get their transactions executed quickly. Every 12 seconds, some computer on the Ethereum network is selected to record the transactions in a block. This computer used to be called a “miner,” but in current proof-of-stake Ethereum blocks are recorded by computers called “validators.” Each block is compiled by one validator, selected more or less at random, called a “proposer”; the other validators vote to accept the block. The validators share the transaction fees, with the block proposer getting more than the other validators.

The block proposer will naturally prioritize the transactions that pay more fees, because then it will get more money. And, again, the validators are all computers; they will be programmed to select the transactions that pay them the most money. And in fact there is a division of labor in modern Ethereum, where a computer called a “block builder” puts together a list of transactions that will pay the most money to the validators, and then the block proposer proposes a block with that list so it can get paid.

Levine then gets into the details:

I am giving a simplistic and somewhat old-fashioned description of MEV, and modern Ethereum has a whole, like, institutional structure around it. There are private mempools, where you can hide transactions from bots. There is Flashbots, “a research and development organization formed to mitigate the negative externalities posed by Maximal Extractable Value (MEV) to stateful blockchains, starting with Ethereum,” which has things like MEV-Boost, which creates “a competitive block-building market” where validators can “maximize their staking reward by selling their blockspace to an open market,” and MEV-Share, “an open-source protocol for users, wallets, and applications to internalize the MEV that their transactions create,” letting them “selectively share data about their transactions with searchers who bid to include the transactions in bundles” and get paid.

What Is Alleged?

We have two explanations of what the brothers are alleged to have done, one from the DoJ's indictment and one from Flashbots, whose MEV-Boost software was exploited.

Dept. of Justice

The DoJ's indictment explains MEV-Boost:

11. “MEV-Boost” is an open-source software designed to optimize the block-building process for Ethereum validators by establishing protocols for how transactions are organized into blocks. Approximately 90% of Ethereum validators use MEV-Boost.
12. Using MEV-Boost, Ethereum validators outsource the block-building process to a network of “searchers,” “builders,” and “relays.” These participants operate pursuant to privacy and commitment protocols designed to ensure that each network participant—the searcher, the builder, and the validator—interacts in an ordered manner that maximizes value and network efficiency.
13. A searcher is effectively a trader who scans the public mempool for profitable arbitrage opportunities using automated bots (“MEV Bots”). After identifying a profitable opportunity (that would, for example, increase the price of a given cryptocurrency), the searcher sends the builder a proposed “bundle” of transactions. following transactions in a precise order: The bundle typically consists of the (a) the searcher’s “frontrun” transaction, in which the searcher purchases some amount of cryptocurrency whose value the searcher expects to increase; (b) the pending transaction in the mempool that the MEV Bot identified would increase the price of that cryptocurrency; and (c) the searcher’s sell transaction, in which the searcher sells the cryptocurrency at a higher price than what the searcher initially paid in order to extract a trading profit. A builder receives bundles from various searchers and compiles them into a proposed block that maximizes MEV for the validator. The builder then sends the proposed block to a “relay.” A relay receives the proposed block from the builder and initially only submits the “blockheader” to the validator, which contains information about, among other things, the payment the validator will receive for validating the proposed block as structured by the builder. It is only after the validator makes this commitment through a digital signature that the relay releases the full content of the proposed block (i.e. — the complete ordered transaction list) to the validator.
14. In this process, a relay acts in a manner similar to an escrow account, which temporarily maintains the otherwise private transaction data of the proposed block until the validator commits to publishing the block to the blockchain exactly as ordered. The relay will not release the transactions within the proposed block to the validator until the validator has confirmed through a digital signature that it will publish the proposed block as structured by the builder to the blockchain. Until the transactions within the proposed block are released to the validator, they remain private and are not publicly visible.

Note the importance of the relay maintaining the privacy of the transactions in the proposed block.

The indictment summarizes how the brothers are alleged to have stolen $25M:

17. ANTON PERAIRE-BUENO and JAMES PERAIRE-BUENO took the following steps, among others, to plan and execute the Exploit: (a) establishing a series of Ethereum validators in a manner that concealed their identities through the use of shell companies, intermediary cryptocurrency addresses, foreign exchanges, and a privacy layer network; (b) deploying a series of test transactions or “bait transactions” designed to identify particular variables most likely to attract MEV Bots that would become the victims of the Exploit (collectively the “Victim Traders”); (c) identifying and exploiting a vulnerability in the MEV-Boost relay code that caused the relay to prematurely release the full content of a proposed block; (d) re-ordering the proposed block to the defendants’ advantage; and (e) publishing the re-ordered block to the Ethereum blockchain, which resulted in the theft of approximately $25 million in cryptocurrency from the Victim Traders.

The indictment adds:

15. Tampering with these established MEV-Boost protocols, which are relied upon by the vast majority of Ethereum users, threatens the stability and integrity of the Ethereum blockchain for all network participants.

This statement has attracted attention. Why should the DoJ care about "the stability and integrity of the Ethereum blockchain"? Note that the brothers are not charged with this, the indictment has three counts:

1. Wire fraud, Title 18, United States Code, Section 1349.
2. Wire fraud, Title 18, United States Code, Sections 1343 and 2.
3. Conspiracy to Commit Money Laundering, Title 18, United States Code, Section 1956(a)(1)(B)(i).

The steps in 11-14 are charged as wire fraud. The indictment then goes on to detail the steps they are alleged to have taken to launder the loot, leading to the money laundering charge.

Flashbots

Flashbots' explanation starts by explaining the role of a relay:

mev-boost works through a commit and reveal scheme where proposers commit to blocks created by builders without seeing their contents, by signing block headers. Only after a block header is signed are the block body and corresponding transactions revealed. A trusted third party called a relay facilitates this process. mev-boost is designed to allow block builders to send blocks that contain valuable MEV to validators without having to trust them. Removing the need for builders to trust validators ensures that every validator has equal access to MEV regardless of their size and is critical for ensuring the validator set of Ethereum remains decentralized.

Notice the traditional cryptocurrency gaslighting about "trustlessness" and "decentralization" in that paragraph:

• It is true that by introducing a relay they have eliminated the need to trust the validators, but they have done so by introducing "a trusted third party called a relay". The exploit worked because the third party violated its trust. They would likely argue that, unlike the validators, the relay lacks financial incentives to cheat. But a malign relay could presumably also play the role of the malign proposer in the exploit.

• 

  ETH 5/21/24
  

  Saying "the validator set of Ethereum remains decentralized" implies that it is decentralized. It is certainly good that the switch to Proof-of-Stake has increaed Ethereum's Nakamoto coefficient from 2-3 to 5-6, as I pointed out last month in "Sufficiently Decentralized":
  

  A year ago the top 5 staking pools controlled 58.4%, now they control 44.7% of the stakes. But it is still true that block production is heavily centralized, with one producer claiming 57.9% of the rewards.

  But a Nakamoto coefficient of 6 isn't very decentralized. Further, this misses the point revealed by the brothers' exploit. With about 55% of execution clients running Geth and around 90% of validators trusting MEV-Boost's relaying, just to take two examples, the software stack is extremely vulnerable to bugs and supply chain attacks.

Flashbots then explain the bug the brothers exploited:

The attack on April 3rd, 2023 was possible because the exploited relay revealed block bodies to the proposer so long as the proposer correctly signed a block header. However, the relay did not check if the block header that was signed was valid. In the case that the block header was signed but invalid, the relay would attempt to publish the block to the beacon chain, where beacon nodes would reject it. Crucially, regardless of whether the block was rejected by beacon nodes or not, the relay would still reveal the body to the proposer.

Having access to the block body allowed the malicious proposer to extract transactions from the stolen block and use them in their own block where it could exploit those transactions. In particular, the malicious proposer constructed their own block that broke the sandwich bots’ sandwiches up and effectively stole their money.

Then they explain the mitigation:

Usually, proposers publishing a modified block would not only equivocate but their new block would have to race the relay block - which has a head start - to acquire attestations for the fork choice rule. However, in this case, the relay was not able to publish a block because the proposer returned an invalid block header. Therefore, the malicious proposer’s new block was uncontested and they won the race automatically. This has been addressed by requiring the relay to successfully publish a block, thereby not sharing invalid blocks with proposers. The mitigations section covers this and future looking details at more length.

By "equivocate" they mean proposing more than one block in a time slot. Validators responsibilities are:

The validator is expected to maintain sufficient hardware and connectivity to participate in block validation and proposal. In return, the validator is paid in ETH (their staked balance increases). On the other hand, participating as a validator also opens new avenues for users to attack the network for personal gain or sabotage. To prevent this, validators miss out on ETH rewards if they fail to participate when called upon, and their existing stake can be destroyed if they behave dishonestly. Two primary behaviors can be considered dishonest: proposing multiple blocks in a single slot (equivocating) and submitting contradictory attestations.

Discussion

Matt Levine covered this case in Crypto Brothers Front-Ran the Front-Runners by focusing on front-running:

There is a sort of cool purity to this. In stock markets, some people are faster than others, and can make money by trading ahead of a big order, and people get mad about this and think it is unfair and propose solutions. And when money changes hands for speed advantages — “payment for order flow,” “colocation” — people complain about corruption. In crypto it’s like “let’s create an efficient market in trading ahead of big orders.” I once wrote: “Rather than solve this concern about traditional markets, crypto made it explicit.” That feels almost like a general philosophy of crypto: Take the problems of traditional finance and make them, worse, sure, but more transparent and visible and explicit and subject to unbridled free markets.

And then casting the brothers' actions as front-running:

Ethereum and its decentralized exchanges have a market structure that is like “bots can look at your transactions and front-run them if that’s profitable.” And these guys, allegedly, front-ran the front-runners; they turned the market structure around so that they could get an early look at the front-running bots’ front-running transactions and front-run them instead. By hacking, sure, sure, it’s bad. But it leaves the Justice Department in the odd position of saying that the integrity of crypto front-running is important and must be defended.

I think Levine is wrong here. Just as with high-frequency trading, "crypto front-running" is legal because it uses public information. The brothers were not indicted for front-running. What is illegal, and what the DoJ is alleging, is trading on "material non-public informatiion", which they obtained by wire fraud (a fraudulent signature). The indictment says:

this False Signature was designed to, and did, trick the Relay to prematurely release the full content of the proposed block to the defendants, including the private transaction information.

The DoJ is not defending the "integrity of crypto front-running", it is prosecuting activity that is illegal in all markets.

The next day Levine made the first of two clarifications:

First, though I described the exploit as “front-running the front-runners,” I do want to be clear that it was not just that. This is not a pure case of (1) submitting spoofy orders to bait front-running bots, (2) having them take the bait and (3) finding some trade to make them lose money. (There are prior examples of that, using oddly structured tokens to make the front-runners lose money.) Here, though, the brothers are accused of something closer to hacking, exploiting a weakness in software code to be able to see (and reorder) a series of transactions that was supposed to be kept confidential from them. That is worse; it’s sort of like the difference between (1) putting in spoof orders on the stock exchange to try to trick a high-frequency trading firm and (2) hacking into the stock exchange’s computer system to reverse the HFT firm’s trades. Even if you think that the front-running bots are bad, you might — as the Justice Department does — object to this approach to punishing them.

Exactly. Levine's second clarification was:

Second, I said that “they exploited a bug in Ethereum” to do this, but that’s not quite right. They exploited a bug in Flashbots’ MEV-Boost, open-source block-building software that “approximately 90% of Ethereum validators use” but that is not part of the core Ethereum system itself. (Here is Flashbots’ explanation.) They exploited a bug in how blocks are normally built and proposed on Ethereum. From the names “Flashbots” and “MEV-Boost,” though, you might get some sense of why the case is controversial. The way that blocks are normally built and proposed on Ethereum involves “maximal extractable value” (MEV), where arbitrage traders bid to pay validators for priority to make the most profitable trades. These brothers hacked that system, but not everyone likes that system, because it involves predatory traders front-running more naive traders.

This is also important because, as one reader commented: “A a crucial distinguishing factor here is that James and Anton did not re-order committed transactions; they instead picked an ordering of pending transactions that were favorable to them. Under this lens, the integrity of the blockchain is not compromised; the network explicitly ‘allows’ validators to pick whatever arbitrary ordering of transactions they like; it's just that generally it’s economically favorable for validators to prioritize transactions which pay them the most first.”

Part of Satoshi Nakamoto's genius in designing Bitcoin was that he observed KISS, the important software mantra Keep It Simple, Stupid. The Bitcoin blockchain does only one thing, maintain a ledger of transactions. So it the Bitcoin ecosystem has evolved very slowly, and has been remarkably free of vulnerabilities over the last decade and a half. Ethereum, on the other hand, is a Turing-complete environment that does whatever the users want it to. So over the last less than a decade the Ethereum ecosystem has evolved much faster, accreting complexity and thus vulnerabilities.

Look at Molly White's Web3 is Going Just Great. It is full of exploits of "smart contracts" such as "decentralized exchanges" and "bridges". Try searching for "bitcoin". You only find it in the context of the amuonts raided. It is precisely the fecundity of the Ethereum's programmability that leads to an ecosystem full of buggy code vulnerable to exploits such as the MEV-Boost one.

Daniel Kuhn's What the DOJ’s First MEV Lawsuit Means for Ethereum also discusses the details of the case:

“They used a flaw in MEV boost to push invalid signatures to preview bundles. That gives an unfair advantage via an exploit,” former employee of the Ethereum Foundation and Flashbots Hudson Jameson told CoinDesk in an interview. Jameson added that the Peraire-Bueno brothers were also running their own validator while extracting MEV, which violates something of a gentleman’s agreement in MEV circles.

“No one else in the MEV ecosystem was doing both of those things at once that we know of,” he added. “They did more than just play by both the codified and pinky promise rules of MEV extraction.”

The "gentleman's agreement" is important, because what the brothers were doing creates a conflict of interest, the kind that the SEC frowns upon.

Kuhn quotes Consensys General Counsel Bill Hughes:

“All of the defendants' preparation for the attack and their completely ham-fisted attempts to cover their tracks afterwards, including extensive incriminating google searches, just helps the government prove they intended to steal. All that evidence will look very bad to a jury. I suspect they plead guilty at some point,”

He also discusses a different reaction in the cryptosphere:

MEV, which itself is controversial, can be a highly lucrative game dominated by automated bots that often comes at blockchain users’ expense, which is partially why so many in the crypto community have rushed to denounce the DOJ’s complaint.
...
Still, others remain convinced that exploiting MEV bots designed to reorder transactions is fair game. “It's a little hard to sympathize with MEV bots and block builders getting f*cked over by block proposers, in the exact same way they are f*cking over end users,” the anonymous researcher said.

Kuhn quotes Hudson Jameson:

Jameson, for his part, said the MEV is something the Ethereum community should work to minimize on Ethereum, but that it’s a difficult problem to solve. For now, the process is “inevitable.”

“Until it can be eliminated, let's study it. Let's illuminate it. Let's minimize it. And since it does exist, let's make it as open as possible for anyone to participate with the same rules,” he said.

Jameson is wrong in suggesting that MEV could be eliminated. It is a consequence of the goal of decentralizing the system. Even the mechanism in place for "anyone to participate with the same rules" requires a trusted third party.