I am working on a post about my reactions to the first two days (I couldn't attend the third) but it requires a good deal of thought, so it'll take a while.As I recall, I came away from the Summit frustrated. I posted the TL;DR version of the reason half a year ago in Why Is The Web "Centralized"? :
What is the centralization that decentralized Web advocates are reacting against? Clearly, it is the domination of the Web by the FANG (Facebook, Amazon, Netflix, Google) and a few other large companies such as the cable oligopoly.Yet the decentralized Web advocates persist in believing that the answer is new technologies, which suffer from the same economic problems as the existing decentralized technologies underlying the "centralized" Web we have. A decentralized technology infrastructure is necessary for a decentralized Web but it isn't sufficient. Absent an understanding of how the rest of the solution is going to work, designing the infrastructure is an academic exercise.
These companies came to dominate the Web for economic not technological reasons.
It is finally time for the long-delayed long-form post. I should first reiterate that I'm greatly in favor of the idea of a decentralized Web based on decentralized storage. It would be a much better world if it happened. I'm happy to dream along with my friend Herbert Van de Sompel's richly-deserved Paul Evan Peters award lecture entitled Scholarly Communication: Deconstruct and Decentralize?. He describes a potential future decentralized system of scholarly communication built on existing Web protocols. But even he prefaces the dream with a caveat that the future he describes "will most likely never exist".
I agree with Herbert about the desirability of his vision, but I also agree that it is unlikely. Below the fold I summarize Herbert's vision, then go through a long explanation of why I think he's right about the low likelihood of its coming into existence.
Herbert identifies three classes of decentralized Web technology and explains that he decided not to deal with these two:
- Distributed file systems. Herbert is right about this. Internet-scale distributed file systems were first prototyped in the late 90s with Intermemory and Oceanstore, and many successors have followed in their footsteps. None have achieved sustainability or Internet platform scale. The reasons are many, the economic one of which I wrote about in Is Distributed Storage Sustainable? Betteridge's Law applies, so the answer is "no".
- Blockchains. Herbert is right about this too. Even the blockchain pioneers have to admit that, in the real world, blockchains have failed to deliver any of their promised advantages over centralized systems. In particular, as we see with Bitcoin, maintaining decentralization against economies of scale is a fundamental, unsolved problem:
Trying by technical means to remove the need to have viable economics and governance is doomed to fail in the medium- let alone the long-term. What is needed is a solution to the economic and governance problems. Then a technology can be designed to work in that framework.And, as Vitalik Buterin points out, the security of blockchains depends upon decentralization:
In the case of blockchain protocols, the mathematical and economic reasoning behind the safety of the consensus often relies crucially on the uncoordinated choice model, or the assumption that the game consists of many small actors that make decisions independently.
The basic idea of Solid is that each person would own a Web domain, the "host" part of a set of URLs that they control. These URLs would be served by a "pod", a Web server controlled by the user that implemented a whole set of Web API standards, including authentication and authorization. Browser-side apps would interact with these pods, allowing the user to:
- Export a machine-readable profile describing the pod and its capabilities.
- Write content for the pod.
- Control others access to the content of the pod.
In Herbert's vision, institutions would host their researchers "research pods", which would be part of their personal domain but would have extensions specific to scholarly communication, such as automatic archiving upon publication.
Herbert demonstrates that the standards and technology needed to implement his pod-based vision for scholarly communication exist, if the implementation is currently a bit fragile. But he concludes by saying:
By understanding why it is not feasible we may get new insights into what is feasible.I'll take up his challenge, but in regard to the decentralized Web that underlies and is in some respects a precondition for his vision. I hope in a future post to apply the arguments that follow to his scholarly communication vision in particular.
The long explanation for why I agree with Herbert that the Solid future "will most likely never exist" starts here. Note that much of what I link to from now on is a must-read, flagged (MR). Most of them are long and cover many issues that are less, but still, related to the reason I agree with Herbert than the parts I cite.
Cory Doctorow introduces his post about Charlie Stross' keynote for the 34th Chaos Communications Congress (MR) by writing (MR):
Stross is very interested in what it means that today's tech billionaires are terrified of being slaughtered by psychotic runaway AIs. Like Ted Chiang and me, Stross thinks that corporations are "slow AIs" that show what happens when we build "machines" designed to optimize for one kind of growth above all moral or ethical considerations, and that these captains of industry are projecting their fears of the businesses they nominally command onto the computers around them.Stross uses the Paperclip Maximizer thought experiment to discuss how the goal of these "slow AIs", which is to maximize profit growth, makes them a threat to humanity. The myth is that these genius tech billionaire CEOs are "in charge", decision makers. But in reality, their decisions are tightly constrained by the logic embedded in their profit growth maximizing "slow AIs".
Here's an example of a "slow AI" responding to its Prime Directive and constraining the "decision makers". Dave Farber's IP list discussed Hiroko Tabuchi's New York Times article How Climate Change Deniers Rise to the Top in Google Searches, which described how well-funded climate deniers were buying ads on Google that appeared at the top of search results for climate change. Chuck McManis (Chuck & I worked together at Sun Microsystems. He worked at Google then built Blekko, another search engine.) contributed a typically informative response. As previously, I have Chuck's permission to quote him extensively:
publications, as recently as the early 21st century, had a very strict wall between editorial and advertising. It compromises the integrity of journalism if the editorial staff can be driven by the advertisers. And Google exploited that tension and turned it into a business model.How did they do that?
When people started using Google as an 'answer this question' machine, and then Google created a mechanism to show your [paid] answer first, the stage was set for what has become a gross perversion of 'reference' information.Why would they do that? Their margins were under pressure:
The average price per click (CPC) of advertisements on Google sites has gone down for every year, and nearly every quarter, since 2009. At the same time Microsoft's Bing search engine CPCs have gone up. As the advantage of Google's search index is eroded by time and investment, primarily by Microsoft, advertisers have been shifting budget to be more of a blend between the two companies. The trend suggests that at some point in the not to distant future advertising margins for both engines will be equivalent.And their other businesses weren't profitable:
Google has scrambled to find an adjacent market, one that could not only generate enough revenue to pay for the infrastructure but also to generate a net income . Youtube, its biggest success outside of search, and the closest thing they have, has yet to do that after literally a decade of investment and effort.So what did they do?
As a result Google has turned to the only tools it has that work, it has reduced payments to its 'affiliate' sites (AdSense for content payments), then boosted the number of ad 'slots' on Google sites, and finally paying third parties to send search traffic preferentially to Google (this too hurts Google's overall search margin)And the effect on users is:
On the search page, Google's bread and butter so to speak, for a 'highly contested' search (that is what search engine marketeers call a search query that can generate lucrative ad clicks) such as 'best credit card' or 'lowest home mortgage', there are many web browser window configurations that show few, if any organic search engine results at all!In other words, for searches that are profitable, Google has moved all the results it thinks are relevant off the first page and replaced them with results that people have paid to put there. Which is pretty much the definition of "evil" in the famous "don't be evil" slogan notoriously dropped in 2015. I'm pretty sure that no-one at executive level in Google thought that building a paid-search engine was a good idea, but the internal logic of the "slow AI" they built forced them into doing just that.
Another example is that Mark Zuckerberg's "personal challenge" for 2018 is to "fix Facebook". In Facebook Can't Be Fixed (MR) John Battelle writes:
You cannot fix Facebook without completely gutting its advertising-driven business model.John Battelle's “Lost Context is also (MR).
And because he is required by Wall Street to put his shareholders above all else, there’s no way in hell Zuckerberg will do that.
Put another way, Facebook has gotten too big to pivot to a new, more “sustainable” business model.
If you’ve read “Lost Context,” you’ve already been exposed to my thinking on why the only way to “fix” Facebook is to utterly rethink its advertising model. It’s this model which has created nearly all the toxic externalities Zuckerberg is worried about: It’s the honeypot which drives the economics of spambots and fake news, it’s the at-scale algorithmic enabler which attracts information warriors from competing nation states, and it’s the reason the platform has become a dopamine-driven engagement trap where time is often not well spent.
I have personal experience of this problem. In the late 80s I foresaw a bleak future for Sun Microsystems. Its profits were based on two key pieces of intellectual property, the SPARC architecture and the Solaris operating system. In each case they had a competitor (Intel and Microsoft) whose strategy was to make owning that kind of IP too expensive for Sun to compete. I came up with a strategy for Sun to undergo a radical transformation into something analogous to a combination of Canonical and an App Store. I spent years promoting and prototyping this idea within Sun.
One of the reasons I have great respect for Scott McNealy is that he gave me, an engineer talking about business, a very fair hearing before rejecting the idea, saying "Its too risky to do with a Fortune 100 company". Another way of saying this is "too big to pivot to a new, more “sustainable” business model". In the terms set by Sun's "slow AI" Scott was right and I was wrong. Sun was taken over by Oracle in 2009; their "slow AI" had no answer for the problems I identified two decades earlier. But in those two decades Sun made its shareholders unbelievable amounts of money.
In Herbert's world of scholarly communication, a similar process can be seen at work in the history of open access (MR, my comments here). In May 1995 Stanford Libraries' HighWire Press pioneered the move of scholarly publishing to the Web by putting the Journal of Biological Chemistry on-line. Three years later, Vitek Tracz was saying:
with the Web technology available today, publishing can potentially happen independently of publishers. If authors started depositing their papers directly into a central repository, they could bypass publishers and make it freely available.He started the first commercial open-access publisher, BioMed Central, in 2000 (the Springer "slow AI" bought it in 2008). In 2002 came the Budapest Open Access Initiative:
By "open access" to this literature, we mean its free availability on the public internet, permitting any users to read, download, copy, distribute, print, search, or link to the full texts of these articles, crawl them for indexing, pass them as data to software, or use them for any other lawful purpose, without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself. The only constraint on reproduction and distribution, and the only role for copyright in this domain, should be to give authors control over the integrity of their work and the right to be properly acknowledged and cited.Sixteen years later, the "slow AIs" which dominate scholarly publishing have succeeded in growing profits so much that Roger Schonfeld can tweet:
I want to know how anyone can possibly suggest that Elsevier is an enemy of open access. I doubt any company today profits more from OA and its growth!What Elsevier means by "open access" is a long, long way from the Budapest definition. The Open Access advocates, none of them business people, set goals which implied the demise of Elsevier and the other "slow AIs" without thinking through how the "slow AIs" would react to this existential threat. The result was that the "slow AIs" perverted the course of "open access" in ways that increased their extraction of monopoly rents, and provided them with even more resources to buy up nascent and established competitors.
|Elsevier's Research Infrastructure|
Open access advocates point to the rise in the proportion of papers that are freely accessible. They don't point to the rise in payments to the major publishers, the added costs to Universities of dealing with the fragmented system, the highly restrictive licenses that allow "free access" in many cases, the frequency with which author processing charges are paid without resulting in free access, and all the other ills that the "slow AIs" have visited upon scholarly communication in the pursuit of profit growth.
What people mean by saying "the Web is centralized" is that it is dominated by a small number of extremely powerful "slow AIs", the FAANGs (Facebook, Apple, Amazon, Netflix, Google) and the big telcos. None of the discussion of the decentralized Web I've seen is about how to displace them, its all about building a
This is so not going to happen.
For example, you could build a decentralized, open source social network system. In fact, people did. It is called Diaspora and it launched in a blaze of geeky enthusiasm in 2011. Diaspora is one of the eight decentralization initiatives studied by the MIT Media Lab's Defending Internet Freedom through Decentralization (MR) report:
The alpha release of the Diaspora software was deeply problematic, riddled with basic security errors in the code. At the same time, the founders of the project received a lot of pressure from Silicon Valley venture capitalists to “pivot” the project to a more profitable business model. Eventually the core team fell apart and the Diaspora platform was handed over to the open source community, who has done a nice job of building out a support website to facilitate new users in signing up for the service. Today it supports just under 60,000 active participants, but the platform remains very niche and turnover of new users is high.Facebook has 1.37*109 daily users, so it is about 22,800 times bigger than Diaspora. Even assuming Diaspora was as good as Facebook, an impossible goal for a small group of Eben Moglen's students, no-one had any idea how to motivate the other 99.996% of Facebook users to abandon the network where all their friends were and restart building their social graph from scratch. The fact that after 6 years Diaspora has 60K active users is impressive for an open source project, but it is orders of magnitude away from the scale needed to be a threat to Facebook. We can see this because Facebook hasn't bothered to react to it.
Suppose the team of students had been inspired, and built something so much better than Facebook along axes that the mass of Facebook users cared about (which don't include federation, censorship resistance, open source, etc.) that they started to migrate. Facebook's "slow AI" would have reacted in one of two ways. Either the team would have been made a financial offer they couldn't refuse, which wouldn't have made a dent in the almost $40B in cash and short-term investments on Facebook's balance sheet. Or Facebook would have tasked a few of their more than 1000 engineers to replicate the better system. They'd have had an easy job because (a) they'd be adding to an existing system rather than building from scratch, and (b) because their system would be centralized, so wouldn't have to deal with the additional costs of decentralization.
Almost certainly Facebook would have done both. Replicating an open source project in-house is very easy and very fast. Doing so would reduce the price they needed to pay to buy the startup. Hiring people good enough to build something better than the existing product is a big problem for the FAANGs. The easiest way to do it is to spot their startup early and buy it. The FAANGs have been doing this so effectively that it no longer makes sense to do a startup in the Valley with the goal of IPO-ing it; the goal is to get bought by a FAANG.
Lets see what happens when one of the FAANGs actually does see something as a threat. Last January Lina M. Kahn of the Open Markets team at the New America Foundation published Amazon's Antitrust Paradox (MR) in the Yale Law Review. Her 24,000-word piece got a lot of well-deserved attention for describing how platforms evade antitrust scrutiny. In August, Barry Lynn, Kahn's boss and the entire Open Markets team were ejected from the New America Foundation. Apparently, the reason was this press release commenting favorably on Google's €2.5 billion loss in an antitrust case in the EU. Lynn claims that:
hours after his press release went online, [New America CEO] Slaughter called him up and said: “I just got off the phone with Eric Schmidt and he is pulling all of his money,”The FAANGs' "slow AIs" understand that antitrust is a serious threat. €2.5 billion checks get their attention, even if they are small compared to their cash hoards. The PR blowback from defenestrating the Open Markets team was a small price to pay for getting the message out that advocating for effective antitrust enforcement carried serious career risks.
This was a FAANG reacting to a law journal article and a press release. "All of his money" had averaged about $1M/yr over two decades. Imagine how FAANGs would react to losing significant numbers of users to a decentralized alternative!
Kahn argued that:
the current framework in antitrust—specifically its pegging competition to “consumer welfare,” defined as short-term price effects—is unequipped to capture the architecture of market power in the modern economy. We cannot cognize the potential harms to competition posed by Amazon’s dominance if we measure competition primarily through price and output. Specifically, current doctrine underappreciates the risk of predatory pricing and how integration across distinct business lines may prove anticompetitive. These concerns are heightened in the context of online platforms for two reasons. First, the economics of platform markets create incentives for a company to pursue growth over profits, a strategy that investors have rewarded. Under these conditions, predatory pricing becomes highly rational—even as existing doctrine treats it as irrational and therefore implausible. Second, because online platforms serve as critical intermediaries, integrating across business lines positions these platforms to control the essential infrastructure on which their rivals depend. This dual role also enables a platform to exploit information collected on companies using its services to undermine them as competitors.In the 30s antitrust was aimed at preserving a healthy market by eliminating excessive concentration of market power. But:
Due to a change in legal thinking and practice in the 1970s and 1980s, antitrust law now assesses competition largely with an eye to the short-term interests of consumers, not producers or the health of the market as a whole; antitrust doctrine views low consumer prices, alone, to be evidence of sound competition. By this measure, Amazon has excelled; it has evaded government scrutiny in part through fervently devoting its business strategy and rhetoric to reducing prices for consumers.
|Shop, Ikebukuro, Tokyo|
Perhaps the reason for Google's dramatic reaction to the Open Markets team was that they were part of a swelling chorus of calls for antitrust action against the FAANGs from both the right and the left. Roger McNamee (previously) was an early investor in Facebook and friend of Zuckerberg's, but in How to Fix Facebook — Before It Fixes Us (MR) even he voices deep concern about Facebook's effects on society. He and ethicist Tristan Harris provide an eight-point prescription for mitigating them:
- Ban bots.
- Block further acquisitions.
- "be transparent about who is behind political and issues-based communication"
- "be more transparent about their algorithms"
- "have a more equitable contractual relationship with users"
- Impose "a limit on the commercial exploitation of consumer data by internet platforms"
- "consumers, not the platforms, should own their own data"
we should consider that the time has come to revive the country’s traditional approach to monopoly. Since the Reagan era, antitrust law has operated under the principle that monopoly is not a problem so long as it doesn’t result in higher prices for consumers. Under that framework, Facebook and Google have been allowed to dominate several industries—not just search and social media but also email, video, photos, and digital ad sales, among others—increasing their monopolies by buying potential rivals like YouTube and Instagram. While superficially appealing, this approach ignores costs that don’t show up in a price tag. Addiction to Facebook, YouTube, and other platforms has a cost. Election manipulation has a cost. Reduced innovation and shrinkage of the entrepreneurial economy has a cost. All of these costs are evident today. We can quantify them well enough to appreciate that the costs to consumers of concentration on the internet are unacceptably high.McNamee understands that the only way to get Facebook to change its ways is the force of antitrust law.
Another of the initiatives studied by the MIT Media Lab's Defending Internet Freedom through Decentralization (MR) is Solid. They describe the project's goal thus:
Ultimately, the goal of this project is to render platforms like Facebook and Twitter as merely “front-end” services that present a user’s data, rather than silos for millions of people’s personal data. To this end, Solid aims to support users in controlling their own personal online datastore, or “pod,” where their personal information resides. Applications would generally run on the client-side (browser or mobile phone) and access data in pods via APIs based on HTTP.In other words, to implement McNamee's #7 prescription.
Why do you think McNamee's #8 talks about the need to "revive the country’s traditional approach to monopoly"? He understands that having people's personal data under their control, not Facebook's, would be viewed by Facebook's "slow AI" as an existential threat. Exclusive control over the biggest and best personal data of everyone on the planet, whether or not they have ever created an account, is the basis on which Facebook's valuation rests.
The Media Lab report at least understands that there is an issue here:
The approach of Solid towards promoting interoperability and platform-switching is admirable, but it begs the question: why would the incumbent “winners” of our current system, the Facebooks and Twitters of the world, ever opt to switch to this model of interacting with their users? Doing so threatens the business model of these companies, which rely on uniquely collecting and monetizing user data. As such, this open, interoperable model is unlikely to gain traction with already successful large platforms. While a site like Facebook might share content a user has created–especially if required to do so by legislation that mandates interoperability–it is harder to imagine them sharing data they have collected on a user, her tastes and online behaviors. Without this data, likely useful for ad targeting, the large platforms may be at an insurmountable advantage in the contemporary advertising ecosystem.The report completely fails to understand the violence of the reaction Solid will face from the FAANGs "slow AIs" if it ever gets big enough for them to notice.
Note that the report fails to understand that you don't have to be a Facebook user to have been extensively profiled. Facebook's "slow AI" is definitely not going to let go of the proprietary data it has collected (and in many cases paid other data sources for) about a person. Attempts to legislate this sharing in isolation would meet ferocious lobbying, and might well be unconstitutional. Nor is it clear that, even if legislation passed, the data would be in a form usable by the person, or by other services. History tends to show that attempts to force interoperability upon unwilling partners are easily sabotaged by them.
McNamee points out that, even if sharing were forced upon Facebook, it would likely do little to reduce their market power:
consumers, not the platforms, should own their own data. In the case of Facebook, this includes posts, friends, and events—in short, the entire social graph. Users created this data, so they should have the right to export it to other social networks. Given inertia and the convenience of Facebook, I wouldn’t expect this reform to trigger a mass flight of users. Instead, the likely outcome would be an explosion of innovation and entrepreneurship. Facebook is so powerful that most new entrants would avoid head-on competition in favor of creating sustainable differentiation. Start-ups and established players would build new products that incorporate people’s existing social graphs, forcing Facebook to compete again.After all, allowing users to export their data from Facebook doesn't prevent Facebook maintaining a copy. And you don't need to be a Facebook user for them to make money from data they acquire about you. Note that, commendably, Google has for many years allowed users to download the data they create in the various Google systems (but not the data Google collects about them) via the Data Liberation Front, now Google TakeOut. It hasn't caused their users to leave.
No alternate social network can succeed without access to the data Facebook currently holds. Realistically, if this is to change, there will be some kind of negotiation. Facebook's going-in position will be "no access". Thus the going-in position for the other side needs to be something that Facebook's "slow AI" will think is much worse than sharing the data.
We may be starting to see what the something much worse might be. In contrast to the laissez-faire approach of US antitrust authorities, the EU has staked out a more aggressive position. It fined Google the €2.5 billion that got the Open Markets team fired. And, as Cory Doctorow reports (MR):
Back in 2016, the EU passed the General Data Protection Regulation, a far-reaching set of rules to protect the personal information and privacy of Europeans that takes effect this coming May.Doctorow explains that these regulations require that:
Under the new directive, every time a European's personal data is captured or shared, they have to give meaningful consent, after being informed about the purpose of the use with enough clarity that they can predict what will happen to it. Every time your data is shared with someone, you should be given the name and contact details for an "information controller" at that entity. That's the baseline: when a company is collecting or sharing information about (or that could reveal!) your "racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, … [and] data concerning health or data concerning a natural person’s sex life or sexual orientation," there's an even higher bar to hurdle.Pagefair has a detailed explanation of what this granting of granular meaningful consent would have to look like. It is not a viable user interface to the current web advertising ecosystem of real-time auctions based on personal information.
|All of these companies need to get consent|
|The start of a long, long chain of dialog boxes|
There is no obvious way the adtech industry in its current form can comply with these rules, and in the nearly two years they've had to adapt, they've done virtually nothing about it, seemingly betting that the EU will just blink and back away, rather than exercise its new statutory powers to hit companies for titanic fines, making high profile examples out of a few sacrificial companies until the rest come into line.Pagefair's take is:
But this is the same institution that just hit Google with a $2.73 billion fine. They're spoiling for this kind of fight, and I wouldn't bet on them backing down. There's no consumer appetite for being spied on online ... and the companies involved are either tech giants that everyone hates (Google, Facebook), or creepy data-brokers no one's ever heard of and everyone hates on principle (Acxiom). These companies have money, but not constituencies.
Meanwhile, publishers are generally at the mercy of the platforms, and I assume most of them are just crossing their fingers and hoping the platforms flick some kind of "comply with the rules without turning off the money-spigot" switch this May.
Websites, apps, and adtech vendors, should switch from using personal data to monetize direct and RTB advertising to “non-personal data”. Using non-personal, rather than personal, data neutralizes the risks of the GDPR for advertisers, publishers, and adtech vendors. And it enables them to address the majority (80%-97%) of the audience that will not give consent for 3rd party tracking across the web.The EU is saying "it is impractical to monetize personal information". Since Facebook's and Google's business models depend on monetizing personal information, this is certainly looks like "something worse" than making it portable.
I remember at Esther Dyson's 2001 conference listening to the CEO of American Express explain how they used sophisticated marketing techniques to get almost all their customers to opt-in to information sharing. If I were Facebook's or Google's "slow AI" I'd be wondering if I could react to the GDPR by getting my users to opt-in to my data collection, and structuring things so they wouldn't opt-in to everyone else's. I would be able to use their personal information, but I wouldn't be able to share it with anyone else. That is a problem for everyone else, but for me its a competitive advantage.
It is hard to see how this will all play out:
- The Chinese government is enthusiastic about enabling companies to monetize personal information. That way the companies fund the government's surveillance infrastructure:
WeChat, the popular mobile application from Tencent Holdings, is set to become more indispensable in the daily lives of many Chinese consumers under a project that turns it into an official electronic personal identification system.
- The US has enabled personal information to be monetized, but seems to be facing a backlash from both right and left.
- The EU seems determined to eliminate, or at least place strict limits on, monetizing of personal information.
If a decentralized Web doesn't achieve mass participation, nothing has really changed. If it does, someone will have figured out how to leverage antitrust to enable it. And someone will have designed a technical infrastructure that fit with and built on that discovery, not a technical infrastructure designed to scratch the itches of technologists.