ISP Monopolies

For at least the last three years (It Isn't About The Technology) I've been blogging about the malign effects of the way the FAANGs dominate the Web and the need for anti-trust action to mitigate them. Finally, with the recent lawsuits against Facebook and Google, some action may be in prospect. I'm planning a post on this topic. But when it comes to malign effects of monopoly I've been ignoring the other monopolists of the Internet, the telcos.

An insightful recent post by John Gilmore to Dave Farber's IP list sparked a response from Thomas Leavitt and some interesting follow-up e-mail. Gilmore was involved in pioneering consumer ISPs, and Leavitt in pioneering Web hosting. Both attribute the current sorry state of Internet connectivity in the US to the lack of effective competition. They and I differ somewhat on how the problem could be fixed. Below the fold I go into the details.

I've known Gilmore since the early days of Sun Microsystems, and I'm grateful for good advice he gave me at a critical stage of my career. He has a remarkable clarity of vision and a list of achievements that includes pioneering paid support for free software (the "Red Hat" model) at Cygnus Support, and pioneering consumer Internet Service Providers (ISPs) with The Little Garden. Because in those dial-up days Internet connectivity was a commercial product layered on regulated infrastructure (the analog telephone system) there was no lock-in. Consumers could change providers simply by changing the number their modem dialed.

This experience is key to Gilmore's argument. He writes:

The USA never had "network neutrality" before it was "suspended". What the USA had was 3,000 ISPs. So if an ISP did something unfriendly to its customers, they could just stop paying the bad one, and sign up with a different ISP that wouldn't screw them. That effectively prevented bad behavior among ISPs. And if the customer couldn't find an ISP that wouldn't screw them, they could START ONE THEMSELVES. I know, because we did exactly that in the 1990s.

Anyone could start an ISP because by law, everyone had tariffed access to the same telco infrastructure (dialup phone lines, and leased lines at 56 kbit/sec or 1.544 Mbit/sec or 45 Mbit/sec). You just called up the telco and ordered it, and they sent out techs and installed it. We did exactly that, plugged it into our modems and routers and bam, we were an ISP: "The Little Garden".

I was an early customer of The Little Garden. A SPARCstation, a SCSI disk and a modem sat on my window-ledge. The system dialed a local, and thus free, number and kept the call up 24/7, enabling me to register a domain and start running my own mail server. Years later I upgraded to DSL with Stanford as my ISP. As Gilmore points out, Stanford could do this under the same law:

Later, DSL lines required installing equipment in telco central offices, at the far end of the wire that leads to your house. But the telcos were required by the FCC to allow competing companies to do that. Their central office buildings were 9/10th empty anyway, after they had replaced racks of mechanical relays with digital computers.

Gilmore explains how this competitive market was killed:

The telcos figured this out, and decided they'd rather be gatekeepers, instead of being the regulated monopoly that gets a fixed profit margin. Looking ahead, they formally asked the FCC to change its rule that telcos had to share their infrastructure with everybody -- but only for futuristic optical fibers. They whined that "FCC wants us to deploy fiber everywhere, but we won't, unless we get to own it and not share it with our competitors." As usual, the regulated monopoly was great at manipulating the public interest regulators. The FCC said, "Sure, keep your fibers unshared." This ruling never even mentioned the Internet, it is all about the physical infrastructure. If the physical stuff is wires, regulated telcos have to share it; if it's glass, they don't.

The speed of dialup maxed out at 56 kbit/sec. DSL maxed out at a couple of megabits. Leased lines worked to 45 Mbit/sec but cost thousands of dollars per month. Anything over that speed required fiber, not wire, at typical distances. As demand for higher Internet speeds arose, any ISP who wanted to offer a faster connection couldn't just order one from the telco, because the telco fibers were now private and unshared. If you want a fiber-based Internet connection now, you can't buy it from anybody except the guys who own the fibers -- mostly the telcos. Most of the 3,000 ISPs could only offer slow Internet access, so everybody stopped paying them. The industry consolidated down to just one or a few businesses per region -- mostly the telcos themselves, plus the cable companies that had build their own local monopoly via city government contracts. Especially lucky regions had maybe one other competitor, like a Wireless ISP, or an electrical co-op that ran fibers on its infrastructure.

Leavitt makes a bigger point than Glimore's:

The ONLY reason the Internet exists as we know it (mass consumer access) was the regulatory loophole which permitted the ISP industry to flourish in the 1990s. The telcos realized their mistake, as John said, and made sure that there wasn't going to be a repeat of that, so with each generation (DSL, fiber), they made it more and more difficult to access their networks, with the result that John mention--almost no consumer choice, for consumers or business. Last office I rented, there was one choice of Internet provider: the local cable monopoly, which arbitrarily wanted to charge me much more ($85/mo) to connect my office than it did the apartments upstairs in the same building ($49). As is the case in most of that county; the only alternatives were a few buildings and complexes wired up by the two surviving local ISPs, and a relatively expensive WISP.

Gilmore concludes:

The telcos' elimination of fiber based competition, and nothing else, was the end of so-called "network neutrality". The rest was just activists, regulators and legislators blathering. There never was an /enforceable federal regulatory policy of network neutrality, so the FCC could hardly suspend it. If the FCC actually wanted US customers to have a choice of ISPs, they would rescind the FIBER RULE. And if advocates actually understood how only competition, not regulation, restrains predatory behavior, they would ask FCC for the fiber rule to be rescinded, so a small ISP company could rent the actual glass fiber that runs from the telco to (near or inside) your house, for the actual cost plus a regulated profit. Then customers could get high speed Internet from a variety of vendors at a variety of prices and terms. So far neither has happened.

Leavitt shows the insane lengths we are resorting to in order to deliver a modicum of competition in the ISP market:

It's ridiculous that it is going to take sending 10s of thousands of satellites into orbit to restore any semblance of competitiveness to the ISP market, when we've had a simple regulatory fix all along. It's not like the telco/cable monopolies suffered as a result of competition... in fact, it created the market they now monopolize. Imagine all the other opportunities for new markets that have been stifled by the lack of competition in the ISP market over the last two decades!

I have been, and still am, an exception to Gilmore's and Leavitt's experiences. Palo Alto owns its own utilities, a great reason to live there. In September 2001 Palo Alto's Fiber To The Home trial went live, and I was one of 67 citizens who got a 10Mbit/s bidirectional connection, with the city Utilities as our ISP. We all loved the price, the speed and the excellent customer service. The telcos got worried and threatened to sue the Utilities if it expanded the service. The City was on safe legal ground, but that is what they had thought previously when they lost a $21.5M lawsuit as part of the fallout from the Enron scandal. Enron's creditors claimed that the Utilities had violated their contract because they stopped paying Enron. The Utilities did so because Enron became unable to deliver them electricity.

So, when the trial ended after I think five years, we loved it so much that we negotiated with Motorola to take over the equipment and found an upstream ISP. But the Utilities were gun-shy and spent IIRC $50K physically ripping out the fiber and trashing the equipment. Since then, Palo Alto's approach to municipal fiber has been a sorry story of ineffective dithering.

Shortly after we lost our fiber, Stanford decided to stop providing staff with DSL, but we again avoided doing business with the telcos. We got DSL and phone service from Sonic, a local ISP that was legally enabled to rent access to AT&T's copper. It was much slower than Comcast or AT&T, but the upside was Sonic's stellar customer service and four static IP addresses. That kept us going quite happily until COVID-19 struck and we had to host our grandchildren for their virtual schools. DSL was not up to the job.

Fortunately, it turned out that Sonic had recently been able to offer gigabit fiber in Palo Alto. Sonic in its North Bay homeland has been deploying its own fiber, as has Cruzio in its Santa Cruz homeland. Here they rent access to AT&T's fiber in the same way that they rented access to the copper. So, after a long series of delays caused by AT&T's inability to get fiber through the conduit under the street that held their copper, we have gigabit speed, home phone and Sonic's unmatched customer service all for $80/month.

As a long-time Sonic customer, I agree with what the Internet Advisor website writes:

Sonic has maintained a reputation as not only a company that delivers a reliable high-speed connection to its customers but also a company that stands by its ethics. Both Dane Jasper and Scott Doty have spoken up on numerous occasions to combat the ever-growing lack of privacy on the web. They have implemented policies that reflect this. In 2011, they reduced the amount of time that they store user data to just two weeks in the face of an ever-growing tide of legal requests for its users’ data. That same year, Sonic alongside Google fought a court order to hand over email addresses who had contacted and had a correspondence with Tor developer and Wikileaks contributor Jacob Applebaum. When asked why, CEO Dane Jasper responded that it was “rather expensive, but the right thing to do.”

Sonic has made a habit of doing the right thing, both for its customers and the larger world. It’s a conscientious company that delivers on what is promised and goes the extra mile for its subscribers.

Leavitt explained in e-mail how Sonic's exception to Gilmore's argument came about:

Sonic is one of the few independent ISPs that's managed to survive the regulatory clampdown via stellar customer service and customers willing to go out of their way to support alternative providers, much like Cruzio in my home town of Santa Cruz. They cut some kind of reseller deal with AT&T back in 2015 that enabled them to offer fiber to a limited number of residents, and again, like Cruzio, are building out their own fiber network, but according to [this site], fiber through them is potentially available to only about 400,000 customers (in a state with about 13 million households and 1 million businesses); it also reports that they are the 8th largest ISP in the nation, despite being a highly regional provider with access available to only about 3 million households. This says everything about how monopolistic and consolidated the ISP market is, given the number of independent cable and telco companies that existed in previous decades, the remaining survivors of which are all undoubtedly offering ISP services.

I doubt Sonic's deal with AT&T was much more lucrative than the DSL deals Santa Cruz area ISPs were able to cut.

Gilmore attempted to build a fiber ISP in his hometown, San Francisco:

Our model was to run a fiber to about one person per block (what Cruzio calls a "champion") and teach them how to run and debug 1G Ethernet cables down the back fences to their neighbors, splitting down the monthly costs. This would avoid most of the cost of city right-of-way crud at every house, which would let us and our champions fiber the city much more broadly and quickly. And would train a small army of citizens to own and manage their own infrastructure.

For unrelated reasons it didn't work out, but it left Gilmore with the conviction that, absent repeal of the FIBER rule, ISP-owned fiber is the way to go. Especially in rural areas this approach has been successful; a recent example was described by Jon Brodkin in Jared Mauch didn't have good broadband—so he built his own fiber ISP. Leavitt argues:

I'd like to see multiple infrastructure providers, both private for profit, and municipally sponsored non-profit public service agencies, all with open access networks; ideally, connecting would be as simple as it was back in the dial up days. I think we need multiple players to keep each other "honest". I do agree that a lot of the barriers to building out local fiber networks are regulatory and process, as John mentions. The big incumbent players have a tremendous advantage navigating this process, and the scale to absorb the overhead of dealing with them in conjunction with the capital outlays (which municipalities also have).

I think we all agree that "ideally, connecting would be as simple as it was back in the dial up days". How to make this happen? As Gilmore says, there are regulatory and process costs as well as the cost of pulling the fiber. So if switching away from a misbehaving ISP involves these costs there is going to a significant barrier. It isn't going to be "as simple as it was back in the dial up days" when the customer could simply re-program their modem.

My experience of municipal fiber leads me to disagree with both Gilmore and Leavitt. For me, the key is to separate the provision of fiber from the provision of Internet services. Why would you want to switch providers?

• Pretty much the only reason why you'd want to switch fiber providers is unreliability. But, absent back-hoes, fiber is extremely reliable.
• There are many reasons why you'd want to switch ISPs, among them privacy, bandwidth caps, price increases.

Municipal fiber provision is typically cheap, because they are the regulator and control the permitting process themselves, and because they are accountable to their voters. And if they simply provide the equivalent of an Ethernet cable to a marketplace of ISPs, each of them will be paying the same for their connectivity. So differences in the price of ISP service will reflect the features and quality of their service offerings.

The cost of switching ISPs would be low, simply reprogramming the routers at each end of the fiber. The reason the telcos want to own the fiber isn't because owning fiber as such is a good business, it is to impose switching costs and thus lock in their customers. We don't want that. But equally we don't want switching ISPs to involve redundantly pulling fiber, because that imposes switching costs too. The only way to make connecting "as simple as it was back in the dial up days" is to separate fiber provision from Internet service provision, so that fiber pets pulled once and rented to competing ISPs. If we are going to have a monopoly at the fiber level, I'd rather have a large number of small monopolies than the duopoly of AT&T and Comcast. And I'd rather have the monopoly accountable to voters.