Not Whether But When

Richard Smith, the CEO of Equifax while the company leaked personal information on most Americans (and suffered at least one more leak that was active for about a year up to last March) was held accountable for these failings by being allowed to retire with a mere $90M. But at Fortune, John Patrick Pullen quotes him as uttering an uncomfortable truth:

"There's those companies that have been breached and know it, and there are those companies that have been breached and don't know it,"

Pullen points out that:

The speech, given by Smith to students and faculty at the university's Terry College of Business, covered a lot of ground, but it frequently returned to security issues that kept the former CEO awake at night—foremost among them was the company's large database.

Smith should have been losing sleep:

Though it was still 21 days before his company would reveal that it had been massively hacked, Equifax, at that time, had been breached and knew it.

Two years ago, the amazing Maciej Cegłowski gave one of his barn-burning speeches, entitled Haunted by Data (my emphasis):

imagine data not as a pristine resource, but as a waste product, a bunch of radioactive, toxic sludge that we don’t know how to handle. In particular, I'd like to draw a parallel between what we're doing and nuclear energy, another technology whose beneficial uses we could never quite untangle from the harmful ones. A singular problem of nuclear power is that it generated deadly waste whose lifespan was far longer than the institutions we could build to guard it. Nuclear waste remains dangerous for many thousands of years. This oddity led to extreme solutions like 'put it all in a mountain' and 'put a scary sculpture on top of it' so that people don't dig it up and eat it. But we never did find a solution. We just keep this stuff in swimming pools or sitting around in barrels.

The fact is that, just like nuclear waste, we have never found a solution to the interconnected problems of keeping data stored in real-world computer systems safe from attack and safe from leaking. It isn't a question of whether the bad guys will get in to the swimming pools and barrels of data, and exfiltrate it. It is simply when they will do so, and how long it will take you to find out that they have. Below the fold I look at the explanation for this fact. I'll get to the implications of our inability to maintain security in a subsequent post.

To summarize the explanation, it is that real-world computer systems are embedded in real-world organizations. It might be possible to build a system that was secure when embedded in an ideal organization, but it definitely isn't possible to build a system that remains secure when embedded in a real-world organization.

At Bloomberg, Michael Riley, Jordan Robertson, and Anita Sharpe have a detailed report on the organizational background to the Equifax leak of personal information on most Americans, and the earlier problem that allowed the bad guys to file fake tax returns and claim large refunds. They report that initially, CEO Smith considered security a priority but it didn't last:

Not long after becoming CEO, he hired Tony Spinelli, a well-regarded cyber expert, to overhaul the company's security. ... Apparently, gaps remained. After the breach became public in September, Steve VanWieren, a vice president of data quality who left Equifax in January 2012 after almost 15 years, wrote in a post on LinkedIn that "it bothered me how much access just about any employee had to the personally identifiable attributes. ... Spinelli left in 2013, followed less than a year later by his top deputy, Nick Nedostup. Many rank and file followed them out the door, and key positions were filled by people who were not well-known in the clubby cybersecurity industry.

Smith's replacement for Spinelli was:

Susan Mauldin, a former security chief at First Data Corp., to run the global security team. Mauldin introduced herself to colleagues as a card-carrying member of the National Rifle Association, according to a person familiar with the changes.

That alone should have disqualified her; it shows that in her personal life she preferred the security theater of waving a big gun around to the data showing that gun-owners are a bigger threat to their family than to the bad guys. She wasn't effective at making security a priority:

“Internally, security was viewed as a bottleneck,” one person said. “There was a lot of pressure to get things done. Anything related to IT was supposed to go through security." ... But one former security leader said he finally joined the talent exodus because it felt like he was working with the “B team.”

That's because he was on the B team. Given the incentives facing a CEO, the A team at the company will always be the one boosting the bottom line this quarter, the pervasive effect of short-termism. At Equifax CEO Smith had things he needed to get done:

Smith acquired two dozen companies that have given Equifax new ways to package and sell data, while expanding operations to 25 countries and 10,000 employees. Business was good—the company’s stock price quadrupled under Smith’s watch,

All two dozen companies had incompatible systems that needed to be integrated into Equifax's by the end of the next quarter so that the stock continued to rise. While, of course, ensuring that these external systems contained no vulnerabilities, that none were introduced during the integration, and that none of the new employees posed an insider threat. In at least one case, this process clearly failed. Brian Krebs reported that:

an online portal designed to let Equifax employees in Argentina manage credit report disputes from consumers in that country was wide open, protected by perhaps the most easy-to-guess password combination ever: “admin/admin.”
...
Once inside the portal, the researchers found they could view the names of more than 100 Equifax employees in Argentina, as well as their employee ID and email address. The “list of users” page also featured a clickable button that anyone authenticated with the “admin/admin” username and password could use to add, modify or delete user accounts on the system. ...

Each employee record included a company username in plain text, and a corresponding password that was obfuscated by a series of dots.

However, all one needed to do in order to view said password was to right-click on the employee’s profile page and select “view source,” a function that displays the raw HTML code which makes up the Web site. Buried in that HTML code was the employee’s password in plain text.

A review of those accounts shows all employee passwords were the same as each user’s username. Worse still, each employee’s username appears to be nothing more than their last name, or a combination of their first initial and last name. In other words, if you knew an Equifax Argentina employee’s last name, you also could work out their password for this credit dispute portal quite easily.

It wasn't just employee's information that was at risk:

From the main page of the Equifax.com.ar employee portal was a listing of some 715 pages worth of complaints and disputes filed by Argentinians who had at one point over the past decade contacted Equifax via fax, phone or email to dispute issues with their credit reports. The site also lists each person’s DNI — the Argentinian equivalent of the Social Security number — again, in plain text. All told, this section of the employee portal included more than 14,000 such records.

Clearly no-one at the C-level in Equifax was going to give priority to the security of some penny-ante dispute resolution portal in Argentina. But it is very likely that, had it been the bad guys snooping around, they would have found links from this system to others on Equifax's network, and been able to get in this way too.

The Argentinian dispute resolvers need to get their work done. They didn't have the resources to pay a top-flight developer to build a system for them, the more so because dispute resolution doesn't fatten the bottom line. So they kludged something together and, not being security gurus, made lots of mistakes.

You may be thinking Equifax is unusually incompetent. But this is what CEO Smith got right. It isn't possible for an organization to restrict security-relevant operations to security gurus who never make mistakes; there aren't enough security gurus to go around, and even security gurus make mistakes. It only takes one mistake, in Equifax's case a delay of more than four days in patching a bug in widely used Web infrastructure, to let the bad guys in:

Information [Nike Zheng] provided to Apache, which published it along with a fix on March 6, showed how the flaw could be used to steal data from any company using the software. ... Within 24 hours, the information was posted to FreeBuf.com, a Chinese security website, and showed up the same day in Metasploit, a popular free hacking tool. On March 10, hackers scanning the internet for computer systems vulnerable to the attack got a hit on an Equifax server in Atlanta,

The delay was, in fact, far worse. In Congressional testimony, ex-CEO Smith revealed that:

an Equifax e-mail directing administrators to patch a critical vulnerability in the open source Apache Struts Web application framework went unheeded, despite a two-day deadline to comply. Equifax also waited a week to scan its network for apps that remained vulnerable. Even then, the delayed scan failed to detect that the code-execution flaw still resided in a section of the sprawling Equifax site that allows consumers to dispute information they believe is incorrect.
...
Although a patch for the code-execution flaw was available during the first week of March, Equifax administrators didn't apply it until July 29,

Other recent examples of organizational security incompetence include Deloitte:

analyst firm Gartner ... in June named Deloitte the world’s best IT security consultancy for the fifth year in a row.

On September 25th it was revealed that Deloitte was:

the victim of a cybersecurity attack that went unnoticed for months. ... The Guardian understands Deloitte discovered the hack in March this year, but it is believed the attackers may have had access to its systems since October or November 2016. ... The hacker compromised the firm’s global email server through an “administrator’s account” that, in theory, gave them privileged, unrestricted “access to all areas”. The account required only a single password and did not have “two-step“ verification, sources said.

This motivated others to start looking at. By the next day, it was obvious that  Deloitte's security wasn't up to scratch. For example(9/26):

a collection of Deloitte's corporate VPN passwords, user names, and operational details were found lurking within a public-facing GitHub-hosted repository. These have since been removed in the past hour or so. In addition, it appears that a Deloitte employee uploaded company proxy login credentials to his public Google+ page. The information was up there for over six months – and was removed in the past few minutes.

And also:

Deloitte has loads of internal and potentially critical systems unnecessarily facing the public internet with remote-desktop access enabled. All of this gear should be behind a firewall and/or with two-factor authentication as per industry best practices. ... “Just in the last day I’ve found 7,000 to 12,000 open hosts for the firm spread across the globe,” security researcher Dan Tentler, founder of Phobos Group, told The Register today. “We’re talking dozens of business units around the planet with dozens of IT departments showing very different aptitude levels. The phrase ‘truly exploitable’ comes to mind.”

Not to mention:

a Deloitte-owned Windows Server 2012 R2 box in South Africa with RDP wide open, acting as what appears to be an Active Directory server – a crucial apex of a Microsoft-powered network – and with, worryingly, security updates still pending installation.

If the "security consultancy of the year" for the last five years straight can't get its act together, and nor can security vendor Kaspersky, what chance has a company like Adobe(9/22):

Adobe's Product Security Incident Response Team (PSIRT) took that transparency a little too far today when a member of the team posted the PGP keys for PSIRT's e-mail account—both the public and the private keys.

Or Apple(9/26):

There's a vulnerability in High Sierra and earlier versions of macOS that allows rogue applications to steal plaintext passwords stored in the Mac keychain, a security researcher said Monday.

Or the companies targeted in the attack Cisco found that used the popular CCleaner application as a distribution channel, including(9/18):

at least 18 technology giants, including Intel, Google, Microsoft, Akamai, Samsung, Sony, VMware, HTC, Linksys, D-Link and Cisco itself

Karl Bode at TechDirt has the first one I've found today (10/3), a vehicle tracking company's database in a public Amazon S3 bucket:

this one is notable for its high creep factor. SVR advertises that its technology provides “continuous vehicle tracking, every two minutes when moving” and a “four hour heartbeat when stopped.” That means that a hacker that had gained access to the login data would be able to track everywhere a customer's car has been in the past 120 days.

Richard Forno of Stanford Law School's Center for Internet and Society points out additional reasons for complacency about security:

Companies can purchase insurance policies to cover the costs of response to, and recovery from, security incidents like data breaches. Equifax’s policy, for example, is reportedly more than US$100 million; Sony Pictures Entertainment had in place a $60 million policy to help cover expenses after its 2014 breach.

This sort of business arrangement – simply transferring the financial risk from one company to another – doesn’t solve any underlying security problems. And since it leaves behind only the risk of some bad publicity, the company’s sense of urgency about proactively fixing problems might be reduced. In addition, it doesn’t address the harm to individual people – such as those whose entire financial histories Equifax stored – when security incidents happen.

And:

when cybersecurity problems happen, many companies start offering purported solutions: One industry colleague called this the computer equivalent of “ambulance chasing.” For instance, less than 36 hours after the Equifax breach was made public, the company’s competitors and other firms increased their advertising of security and identity protection services. But those companies may not be secure themselves.  ...  when companies discover that they can make more money selling to customers whose security is violated rather than spending money to keep data safe, they realize that it’s profitable to remain vulnerable.

I could keep going, but you get the idea. The supply of incompetence is endless. So also is the supply of vulnerabilities, as shown by the really important 2010 paper by Sandy Clarke, Matt Blaze, Stefan Frei and Jonathan Smith entitled Familiarity Breeds Contempt: The Honeymoon Effect and the Role of Legacy Code in Zero-Day Vulnerabilities. They show that, the older a software code base, the greater the rate at which zero-day vulnerabilities are found. So even if an organization is staffed exclusively by infallible security gurus, it will still get compromised via a zero-day.