The Risks Of Outsourcing

My Cloud for Preservation post was in some sense all about the risks of outsourcing IT infrastructure to the cloud. Below the fold I comment on two recent articles illustrating different aspects of these risks.

One risk that is often underestimated is that the security of IT outsourced to a cloud provider has a single point of failure, the security of the account at the cloud provider. Brian Wilson of Backblaze pointed to the risk that the account suffers a billing problem:

Any vendor selling cloud storage relies on billing its customers. If a customer stops paying, after some grace period, the vendor will delete the data to free up space for a paying customer.

Some customers pay by credit card. We don’t have the math behind it, but we believe there’s a greater than 1 in a million chance that the following events could occur:

• You change your credit card provider. The credit card on file is invalid when the vendor tries to bill it.
• Your email service provider thinks billing emails are SPAM. You don’t see the emails coming from your vendor saying there is a problem.
• You do not answer phone calls from numbers you do not recognize; Customer Support is trying to call you from a blocked number; they are trying to leave voicemails but the mailbox is full.

If all those things are true, it’s possible that your data gets deleted simply because the system is operating as designed.

Matthew Miller's SIM swap horror story: I've lost decades of data and Google won't lift a finger points to another risk, that the account gets hijacked, in his case via a SIM card swap enabled by inadequate processes at T-Mobile. The sub-head says it all:

First they hijacked my T-Mobile service, then they stole my Google and Twitter accounts and charged my bank with a $25,000 Bitcoin purchase. I'm stuck in my own personal Black Mirror episode. Why will no one help me?

Because one of the two-factor authentication (2FA) methods for Miller's Google account was his T-Mobile phone, the attacker could get in and disable his other 2FA methods, locking him out of all his Google services, and then go on to hijack his Twitter and bank accounts, both linked to his gmail address. The gmail address turned out to be a single point of failure.

Accounts controlling outsourced infrastructure need to be very carefully secured. Use of U2F keys such as the Yubikey should be mandatory and any dependency on a SIM, such as SMS or voice 2FA absolutely forbidden and routinely audited. Brian Barrett's How to Protect Yourself Against a SIM Swap Attack is a useful resource for securing personal accounts. In particular, use apps such as Google Authenticator instead of SMS, they are linked to the phone, not to its SIM. Ensure that you do NOT have text or voice messages enabled for 2FA. And think carefully about what would happen if any of your email addresses were hijacked.

In Why public sector outsourcing is less efficient than Soviet central planning Abby Innes of the London School of Economics looks at the history of UK government outsourcing. She analyzes the reasons behind its consistent failure the deliver the benefits promised by neoliberal economists and the politicians seduced by them. This is a much broader picture than just IT outsourcing, but many of the issues she identifies are uniform across the entire range.

Innes' main point is that the contract underlying outsourcing cannot be robust enough to cover all contingencies:

When we put the market rhetoric of New Public Management to one side outsourcing constitutes the central planning of private businesses, and the success of this venture hinges on the viability of the outsourcing contract as an effective junction of instruction and control. What contract theory tells us is that the more complex the service or good, the longer the duration of the contract and the greater the contingencies or uncertainties that the supplier might face, the less the outsourced tasks are amenable to codification and hence to robust contracts that can adequately protect the buyer.

And that, largely because of the inevitable information asymmetry that places the buyer of outsourcing at a disadvantage:

The following market failures are rife in public service markets: high barriers to entry leave public service markets dominated by monopoly or oligopoly firms which render the provider relatively immune from the self–correcting mechanisms of market competition; uncertainty and complexities in contractual requirements create huge information asymmetries between buyer and seller; relationship–specific investments encourage the producer to exploit the loss of bargaining power entailed by sunk costs (i.e. ‘hold-up’ problems); and finally, negative spillovers, that is to say, damaging external effects not reflected in the original price of the transaction are particularly problematic given systemic interdependencies, for example between NHS and social care systems.

Lets look at these market failures in the cloud context:

• High barriers to entry. Check. Cloud service provision is an oligopoly.
• Information asymmetries. Check. The cloud service knows a great deal about the customer's business, the customer knows next to nothing about the cloud service beyond the price list and the EULA.
• Relationship-specific investments & sunk costs. Check. The cloud service can make a good estimate of the cost to the customer of switching services, and price accordingly.
• Negative spillovers. Uncheck (I think).

Especially since most customers for outsourcing are much smaller than governments, they are in a very weak bargaining position. And the whole point of outsourcing is to save costs. Innes notes that this has implications:

The tougher any government tries to be in contract pricing within incomplete contracts the more damaging the consequences of margin-seeking by the firm are likely to prove. The risk under austerity is of chronic adverse selection. Given the objective difficulty of establishing accurate pricing under incomplete contracting, only the most reckless firms with least regard for service quality and those most determined to deploy later strategies of ‘hold up’ will rationally underbid for contracts with no guarantee they can stay within the initial margins. The collapsed Carillion was just such a repeat ‘winner’. Carillion’s management acted rationally under prevailing incentive structures: they were aberrant only in misjudging the moment when the financial markets would baulk at the unsustainability of their value extraction.

The standard counter-argument to ‘the problem of monopoly’ is that the reputational effect on dominant firms acts as a disciplining guarantee against poor contractual behaviour. But in monopoly/duopoly public service industry markets with high barriers to entry under doctrinaire governments who are increasingly structurally dependent on the survival of the dominant firms, the reputational damage to even atrocious providers is apparently nil. A Public Accounts Select Committee investigation found that Serco and G4S were awarded fourteen new contracts by five Departments worth £350 million even as they were being investigated by the Serious Fraud Office for defrauding the Ministry of Justice and after the then Justice Minister, Chris Grayling, publicly committed to withhold awards until the case was resolved: the MoJ was among the five.

See what I mean about a weak bargaining position? See also the oligopoly of the audit market, where "the reputational damage to even atrocious providers is apparently nil" with KPMG just the latest example.

And don't forget that outsourcing to Amazon means paying the generous margins that AWS enjoys. He is talking about compute not about storage, but in Who needs a supercomputer when you can get a couple of petaflops on AWS? Tim Anderson quotes:

Dr Paul Calleja, director of the University of Cambridge Research Computing Services, which operates Cumulus, 107 on the Top500, ... "The cost of running off-prem is significantly higher than the cost of running on-prem," he said. "With our cost models it's roughly 3x which is a big number when you are talking petascale."

Go read the whole of all three posts.