Tuesday, October 3, 2017

Not Whether But When

Richard Smith, the CEO of Equifax while the company leaked personal information on most Americans (and suffered at least one more leak that was active for about a year up to last March) was held accountable for these failings by being allowed to retire with a mere $90M. But at Fortune, John Patrick Pullen quotes him as uttering an uncomfortable truth:
"There's those companies that have been breached and know it, and there are those companies that have been breached and don't know it,"
Pullen points out that:
The speech, given by Smith to students and faculty at the university's Terry College of Business, covered a lot of ground, but it frequently returned to security issues that kept the former CEO awake at night—foremost among them was the company's large database.
Smith should have been losing sleep:
Though it was still 21 days before his company would reveal that it had been massively hacked, Equifax, at that time, had been breached and knew it.
Two years ago, the amazing Maciej Cegłowski gave one of his barn-burning speeches, entitled Haunted by Data (my emphasis):
imagine data not as a pristine resource, but as a waste product, a bunch of radioactive, toxic sludge that we don’t know how to handle. In particular, I'd like to draw a parallel between what we're doing and nuclear energy, another technology whose beneficial uses we could never quite untangle from the harmful ones. A singular problem of nuclear power is that it generated deadly waste whose lifespan was far longer than the institutions we could build to guard it. Nuclear waste remains dangerous for many thousands of years. This oddity led to extreme solutions like 'put it all in a mountain' and 'put a scary sculpture on top of it' so that people don't dig it up and eat it. But we never did find a solution. We just keep this stuff in swimming pools or sitting around in barrels.
The fact is that, just like nuclear waste, we have never found a solution to the interconnected problems of keeping data stored in real-world computer systems safe from attack and safe from leaking. It isn't a question of whether the bad guys will get in to the swimming pools and barrels of data, and exfiltrate it. It is simply when they will do so, and how long it will take you to find out that they have. Below the fold I look at the explanation for this fact. I'll get to the implications of our inability to maintain security in a subsequent post.

To summarize the explanation, it is that real-world computer systems are embedded in real-world organizations. It might be possible to build a system that was secure when embedded in an ideal organization, but it definitely isn't possible to build a system that remains secure when embedded in a real-world organization.

At Bloomberg, Michael Riley, Jordan Robertson, and Anita Sharpe have a detailed report on the organizational background to the Equifax leak of personal information on most Americans, and the earlier problem that allowed the bad guys to file fake tax returns and claim large refunds. They report that initially, CEO Smith considered security a priority but it didn't last:
Not long after becoming CEO, he hired Tony Spinelli, a well-regarded cyber expert, to overhaul the company's security. ... Apparently, gaps remained. After the breach became public in September, Steve VanWieren, a vice president of data quality who left Equifax in January 2012 after almost 15 years, wrote in a post on LinkedIn that "it bothered me how much access just about any employee had to the personally identifiable attributes. ... Spinelli left in 2013, followed less than a year later by his top deputy, Nick Nedostup. Many rank and file followed them out the door, and key positions were filled by people who were not well-known in the clubby cybersecurity industry.
Smith's replacement for Spinelli was:
Susan Mauldin, a former security chief at First Data Corp., to run the global security team. Mauldin introduced herself to colleagues as a card-carrying member of the National Rifle Association, according to a person familiar with the changes.
That alone should have disqualified her; it shows that in her personal life she preferred the security theater of waving a big gun around to the data showing that gun-owners are a bigger threat to their family than to the bad guys. She wasn't effective at making security a priority:
“Internally, security was viewed as a bottleneck,” one person said. “There was a lot of pressure to get things done. Anything related to IT was supposed to go through security." ... But one former security leader said he finally joined the talent exodus because it felt like he was working with the “B team.”
That's because he was on the B team. Given the incentives facing a CEO, the A team at the company will always be the one boosting the bottom line this quarter, the pervasive effect of short-termism. At Equifax CEO Smith had things he needed to get done:
Smith acquired two dozen companies that have given Equifax new ways to package and sell data, while expanding operations to 25 countries and 10,000 employees. Business was good—the company’s stock price quadrupled under Smith’s watch,
All two dozen companies had incompatible systems that needed to be integrated into Equifax's by the end of the next quarter so that the stock continued to rise. While, of course, ensuring that these external systems contained no vulnerabilities, that none were introduced during the integration, and that none of the new employees posed an insider threat. In at least one case, this process clearly failed. Brian Krebs reported that:
an online portal designed to let Equifax employees in Argentina manage credit report disputes from consumers in that country was wide open, protected by perhaps the most easy-to-guess password combination ever: “admin/admin.”
Once inside the portal, the researchers found they could view the names of more than 100 Equifax employees in Argentina, as well as their employee ID and email address. The “list of users” page also featured a clickable button that anyone authenticated with the “admin/admin” username and password could use to add, modify or delete user accounts on the system. ...

Each employee record included a company username in plain text, and a corresponding password that was obfuscated by a series of dots.

However, all one needed to do in order to view said password was to right-click on the employee’s profile page and select “view source,” a function that displays the raw HTML code which makes up the Web site. Buried in that HTML code was the employee’s password in plain text.

A review of those accounts shows all employee passwords were the same as each user’s username. Worse still, each employee’s username appears to be nothing more than their last name, or a combination of their first initial and last name. In other words, if you knew an Equifax Argentina employee’s last name, you also could work out their password for this credit dispute portal quite easily.
It wasn't just employee's information that was at risk:
From the main page of the Equifax.com.ar employee portal was a listing of some 715 pages worth of complaints and disputes filed by Argentinians who had at one point over the past decade contacted Equifax via fax, phone or email to dispute issues with their credit reports. The site also lists each person’s DNI — the Argentinian equivalent of the Social Security number — again, in plain text. All told, this section of the employee portal included more than 14,000 such records.
Clearly no-one at the C-level in Equifax was going to give priority to the security of some penny-ante dispute resolution portal in Argentina. But it is very likely that, had it been the bad guys snooping around, they would have found links from this system to others on Equifax's network, and been able to get in this way too.

The Argentinian dispute resolvers need to get their work done. They didn't have the resources to pay a top-flight developer to build a system for them, the more so because dispute resolution doesn't fatten the bottom line. So they kludged something together and, not being security gurus, made lots of mistakes.

You may be thinking Equifax is unusually incompetent. But this is what CEO Smith got right. It isn't possible for an organization to restrict security-relevant operations to security gurus who never make mistakes; there aren't enough security gurus to go around, and even security gurus make mistakes. It only takes one mistake, in Equifax's case a delay of more than four days in patching a bug in widely used Web infrastructure, to let the bad guys in:
Information [Nike Zheng] provided to Apache, which published it along with a fix on March 6, showed how the flaw could be used to steal data from any company using the software. ... Within 24 hours, the information was posted to FreeBuf.com, a Chinese security website, and showed up the same day in Metasploit, a popular free hacking tool. On March 10, hackers scanning the internet for computer systems vulnerable to the attack got a hit on an Equifax server in Atlanta,
The delay was, in fact, far worse. In Congressional testimony, ex-CEO Smith revealed that:
an Equifax e-mail directing administrators to patch a critical vulnerability in the open source Apache Struts Web application framework went unheeded, despite a two-day deadline to comply. Equifax also waited a week to scan its network for apps that remained vulnerable. Even then, the delayed scan failed to detect that the code-execution flaw still resided in a section of the sprawling Equifax site that allows consumers to dispute information they believe is incorrect.
Although a patch for the code-execution flaw was available during the first week of March, Equifax administrators didn't apply it until July 29,
Other recent examples of organizational security incompetence include Deloitte:
analyst firm Gartner ... in June named Deloitte the world’s best IT security consultancy for the fifth year in a row.
On September 25th it was revealed that Deloitte was:
the victim of a cybersecurity attack that went unnoticed for months. ... The Guardian understands Deloitte discovered the hack in March this year, but it is believed the attackers may have had access to its systems since October or November 2016. ... The hacker compromised the firm’s global email server through an “administrator’s account” that, in theory, gave them privileged, unrestricted “access to all areas”. The account required only a single password and did not have “two-step“ verification, sources said.
This motivated others to start looking at. By the next day, it was obvious that  Deloitte's security wasn't up to scratch. For example(9/26):
a collection of Deloitte's corporate VPN passwords, user names, and operational details were found lurking within a public-facing GitHub-hosted repository. These have since been removed in the past hour or so. In addition, it appears that a Deloitte employee uploaded company proxy login credentials to his public Google+ page. The information was up there for over six months – and was removed in the past few minutes.
And also:
Deloitte has loads of internal and potentially critical systems unnecessarily facing the public internet with remote-desktop access enabled. All of this gear should be behind a firewall and/or with two-factor authentication as per industry best practices. ... “Just in the last day I’ve found 7,000 to 12,000 open hosts for the firm spread across the globe,” security researcher Dan Tentler, founder of Phobos Group, told The Register today. “We’re talking dozens of business units around the planet with dozens of IT departments showing very different aptitude levels. The phrase ‘truly exploitable’ comes to mind.”
Not to mention:
a Deloitte-owned Windows Server 2012 R2 box in South Africa with RDP wide open, acting as what appears to be an Active Directory server – a crucial apex of a Microsoft-powered network – and with, worryingly, security updates still pending installation.
If the "security consultancy of the year" for the last five years straight can't get its act together, and nor can security vendor Kaspersky, what chance has a company like Adobe(9/22):
Adobe's Product Security Incident Response Team (PSIRT) took that transparency a little too far today when a member of the team posted the PGP keys for PSIRT's e-mail account—both the public and the private keys.
Or Apple(9/26):
There's a vulnerability in High Sierra and earlier versions of macOS that allows rogue applications to steal plaintext passwords stored in the Mac keychain, a security researcher said Monday.
Or the companies targeted in the attack Cisco found that used the popular CCleaner application as a distribution channel, including(9/18):
at least 18 technology giants, including Intel, Google, Microsoft, Akamai, Samsung, Sony, VMware, HTC, Linksys, D-Link and Cisco itself
Karl Bode at TechDirt has the first one I've found today (10/3), a vehicle tracking company's database in a public Amazon S3 bucket:
this one is notable for its high creep factor. SVR advertises that its technology provides “continuous vehicle tracking, every two minutes when moving” and a “four hour heartbeat when stopped.” That means that a hacker that had gained access to the login data would be able to track everywhere a customer's car has been in the past 120 days.
Richard Forno of Stanford Law School's Center for Internet and Society points out additional reasons for complacency about security:
Companies can purchase insurance policies to cover the costs of response to, and recovery from, security incidents like data breaches. Equifax’s policy, for example, is reportedly more than US$100 million; Sony Pictures Entertainment had in place a $60 million policy to help cover expenses after its 2014 breach.

This sort of business arrangement – simply transferring the financial risk from one company to another – doesn’t solve any underlying security problems. And since it leaves behind only the risk of some bad publicity, the company’s sense of urgency about proactively fixing problems might be reduced. In addition, it doesn’t address the harm to individual people – such as those whose entire financial histories Equifax stored – when security incidents happen.
when cybersecurity problems happen, many companies start offering purported solutions: One industry colleague called this the computer equivalent of “ambulance chasing.” For instance, less than 36 hours after the Equifax breach was made public, the company’s competitors and other firms increased their advertising of security and identity protection services. But those companies may not be secure themselves.  ...  when companies discover that they can make more money selling to customers whose security is violated rather than spending money to keep data safe, they realize that it’s profitable to remain vulnerable.
I could keep going, but you get the idea. The supply of incompetence is endless. So also is the supply of vulnerabilities, as shown by the really important 2010 paper by Sandy Clarke, Matt Blaze, Stefan Frei and Jonathan Smith entitled Familiarity Breeds Contempt: The Honeymoon Effect and the Role of Legacy Code in Zero-Day Vulnerabilities. They show that, the older a software code base, the greater the rate at which zero-day vulnerabilities are found. So even if an organization is staffed exclusively by infallible security gurus, it will still get compromised via a zero-day.


Fazal Majid said...

Excellent write-up, as always.

One point worth mentioning: the CEO is not "retiring" because he was pushed out, he is doing so to get his $90M payout before the company is sued into bankruptcy, thus preventing him from collecting. It's smart risk management for someone who obviously knows how to look for Number One. Hint: it's neither the Equifux shareholders or the public at large.

In that respect, he is no different in exploiting externalities than the banking executives at too-big-to-fail institutions who loaded up on risky debt because volatility is good for short-term stock options upside, while bad for shareholders and taxpayers.

David. said...

You can't make this stuff up:

"Shortly after we all learned of a massive security breach at Equifax in which the personal information of 143 million 145.5 million Americans and sundry Brits and Canadians was plundered by hackers, the US Internal Revenue Service awarded Equifax a no-bid contract – to provide identity verification services for the tax authority.

People's social security numbers, used by the IRS to identify folks, were among the private information left unencrypted on accessible servers and then stolen from Equifax. Which is now being paid to identify US taxpayers. That totally makes sense."

David. said...

It has only taken Yahoo! 4 years to figure out what happened:

"In a filing on Tuesday to America's financial watchdogs, Yahoo!, now owned by Verizon under the Oath brand, admitted the total number of user accounts illegally accessed by hackers in 2013 wasn't the 500 million earlier reported, nor the one billion it later confessed, but all of them – all three billion accounts."

David. said...

Cory Doctorow writes:

" At yesterday's Congressional hearings on the Equifax breach, Senator Elizabeth Warren took a moment to enumerate all the ways that Equifax will benefit from doxing 145,500,000 Americans.

For starters, there's the "free" credit monitoring service that was Equifax's initial sop to public outrage over its unforgivable carelessness. While the service is indeed free for the first year, every year thereafter Equifax will autobill you $17/month. If less than 1% of the people who signed up for credit monitoring after the breach forget to cancel, the company will make an extra $200,000,000/year."

That should really motivate Equifax to take security more seriously!

David. said...

Brian Krebs has another installment of the rolling security disaster that is Equifax:

"In May, KrebsOnSecurity broke a story about lax security at a payroll division of big-three credit bureau Equifax that let identity thieves access personal and financial data on an unknown number of Americans. Incredibly, this same division makes it simple to access detailed salary and employment history on a large portion of Americans using little more than someone’s Social Security number and date of birth — both data elements that were stolen in the recent breach at Equifax.

David. said...

You would think that after blaming the leak of most Americans' details on a single employee, Equifax would have fired the culprit and bullet-proof security would have been restored overnight. Sadly, no:

"The site that previously gave up personal data for virtually every US person with a credit history was once again under the influence of attackers, this time trying to trick Equifax visitors into installing crapware Symantec calls Adware.Eorezo."

David. said...

Unlike Deloitte, Accenture was not the "world’s best IT security consultancy" five years running. But they are strong competitors in this field:

"Accenture has confirmed it inadvertently left a massive store of private data across four unsecured cloud servers, exposing highly sensitive passwords and secret decryption keys that could have inflicted considerable damage on the company and its customers.

The servers, hosted on Amazon's S3 storage service, contained hundreds of gigabytes of data for the company's enterprise cloud offering, which the company claims provides support to the majority of the Fortune 100.

The data could be downloaded without a password by anyone who knew the servers' web addresses."


"Each server contained a range of different types of credentials, including private signing keys that could be used to impersonate the company, and passwords -- some of which were stored in plaintext.

Vickery said he also found Accenture's master keys for its Amazon Web Service's Key Management System (KMS), which if stolen could allow an attacker full control over the company's encrypted data stored on Amazon's servers."

David. said...

Not to be outdone by Equifax, TransUnion is also competing in the malware distribution market:

"Equifax isn't the only credit-reporting behemoth with a website redirecting visitors to fake Adobe Flash updates. A security researcher from AV provider Malwarebytes said transunioncentroamerica.com, a TransUnion site serving people in Central America, is also sending visitors to the fraudulent updates and other types of malicious pages."

David. said...

The military claim to be experts at cyber-warfare. Maybe at the attack side, but defense? Not so much:

"A cyberattacker nicknamed 'Alf' gained access to an Australian defense contractor's computers and began a four-month raid that snared data on sophisticated U.S. weapons systems. Using the simple combinations of login names and passwords 'admin; admin' and 'guest; guest' and exploiting a vulnerability in the company's help-desk portal, the attacker roved the firm's network for four months. ... Alf obtained around 30 gigabytes of data on Australia's planned purchase of up to 100 F-35 fighters made by Lockheed Martin, as well as information on new warships and Boeing-built P-8 Poseidon maritime-surveillance aircraft, in the July 2016 breach."

Hat tip to phalse phace at /.

David. said...

The result of Equifax distributing malware is that its $7.2M no-bid contract with the IRS:

"The tax-collecting agency is now temporarily suspending the contract because of another Equifax snafu. The Equifax site was maliciously manipulated again, this time to deliver fraudulent Adobe Flash updates, which, when clicked, infected visitors' computers with adware that was detected by just three of 65 antivirus providers. The development means that at least for now, taxpayers cannot open new Secure Access accounts with the IRS."

David. said...

Malvertising from both Equifax and TransUnion was part of a campaign affecting a thousand or so sites via a compromised JavaScript library. Jermome Segura says:

"The reality is that most websites rely on CDNs [content distribution networks] and loading external JS [JavaScript], but that can also be a weakness and expose your visitors to malicious traffic if any of those get compromised."

This is an instance of a huge problem for the Web. As Thomas Claburn wrote back in March:

"The web has a security problem: code libraries. Almost 88 per cent of the top 75,000 websites and 47 per cent of .com websites rely on at least one vulnerable JavaScript library.

As described in a recently published paper, "Thou Shalt Not Depend on Me: Analysing the Use of Outdated JavaScript Libraries on the Web," researchers from Northeastern University in Boston, Massachusetts, have found that many websites rely widely on insecure versions of JavaScript libraries and that there's no immediate way to eliminate this problem."

And, because of Familiarity Breeds Contempt, the older the library the more known vulnerabilities it will have.

David. said...

angryea at Daily Kos makes an excellent point about schedule pressure:

"Programmers, like everyone else, are embedded in an economic system. If there is schedule pressure it does not come from the programmers, who are generally interested in making things. Rather, it comes from the businesses that employ them, who are generally interested in making money. Getting in the way of making money, by, say, holding out for better software, sooner or later leads to unemployment absent any other countervailing factors. Unions are countervailing factors. Professional associations are countervailing factors. Counting on the good will of your employers is not, in most cases, a countervailing factor."

David. said...

This morning's bad news on the security front includes a serious flaw in WPA2 and a flaw in high-security crypto cards:

"The weakness allows attackers to calculate the private portion of any vulnerable key using nothing more than the corresponding public portion. Hackers can then use the private key to impersonate key owners, decrypt sensitive data, sneak malicious code into digitally signed software, and bypass protections that prevent accessing or tampering with stolen PCs. The five-year-old flaw is also troubling because it's located in code that complies with two internationally recognized security certification standards that are binding on many governments, contractors, and companies around the world. The code library was developed by German chipmaker Infineon and has been generating weak keys since 2012 at the latest."

David. said...

Oh, and the news that private companies can make money re-selling the telcos' ability to track you in real time:

"These services are using your mobile phone’s IP address to look up your phone number, your billing information and possibly your phone’s current location as provided by cell phone towers (no GPS or phone location services required). These services are doing this with the assistance of the telco providers."

Tip of the hat to Rob Beschizza:

"It knew my name and address and more besides, and located to me to a few hundred feet's accuracy. I certainly never knowingly opted-in to it."

David. said...

Today starts with cryptocurrency mining in the cloud:

"Security outfit RedLock's security trends report [PDF], out this month, said developers and organizations are not securing their AWS, Azure and Google Cloud Platform systems, allowing miscreants to hijack them to steal processor cycles for digging up alt-coins. It's believed hackers are able to get into boxes by using their default credentials.

RedLock says companies stung this way included security company Gemalto and insurer Aviva.

Its investigators “found a number of Kubernetes administrative consoles deployed on Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform that were not password protected,” the report stated.

It's one way to save yourself the price of enough iron to mine even one Bitcoin. For example, the Bitcoin Energy Index estimates the total energy consumed by miners over the next year will be 21 Terawatt-hours, and it takes 215 kWH for a single transaction."

215KWh/transaction - think about that for a moment.

David. said...

The IRS does not expect the 143M personal details leaked by Equifax, the company that assures your identity to the IRS based on these details, to have a major effect, because:

"the agency believes a “significant” number of the victims already had their information stolen by cyber criminals.

“We actually think that it won’t make any significantly or noticeable difference,” Koskinen told reporters during a briefing on the agency’s data security efforts. “Our estimate is a significant percent of those taxpayers already had their information in the hands of criminals.”

The IRS estimates that more than 100 million Americans have had their personally identifiable information stolen by criminal hackers, he said.


“It’s an important reminder to the public that everyone can take any actions that they can ... to make sure we can do everything we can to protect personal information,” Koskinen said of the breach on Tuesday, in response to a reporter’s question.

The IRS commissioner advised Americans to “assume” their data is already in the hands of criminals and “act accordingly.”"

So that's OK, then.

But wait! "everyone can take any actions that they can ... to make sure we can do everything we can to protect personal information" and "act accordingly". What exactly are these actions? We didn't give the information to Equifax, we didn't leak it, we didn't set up a stupid identity system based on information that wasn't secret even before Equifax leaked it, the IRS did. Oh, and:

"The IRS itself was the victim of a breach in 2015 that exposed personal information associated with more than 700,000 accounts."

David. said...

Dan Goodin's Microsoft never disclosed 2013 hack of secret vulnerability database reveals what might be called a meta-hack:

"Hackers broke into Microsoft's secret, internal bug-tracking database and stole information related to vulnerabilities that were exploited in later attacks."

Why go to all the work of finding vulnerabilities one-at-a-time when there is a whole database full of them ripe for the picking?

blissex said...

«"There's those companies that have been breached and know it, and there are those companies that have been breached and don't know it,"»

There is a legendary quote from the mainframe era, "As far as we know, our computer has never had an undetected error", and a wise friend commented on this "same for an undetected hack".
These are a much deeper quote and comment than it seems.

blissex said...

«holding out for better software, sooner or later leads to unemployment absent any other countervailing factors. Unions are countervailing factors. Professional associations are countervailing factors. Counting on the good will of your employers is not, in most cases, a countervailing factor.»

There are several reports of huge PHB pressure on certified civil and mechanical engineers to "stamp" designs even if they have reservations on them. It is illegal to try, but surely bonuses and promotions go only to "team players".
Fortunately for PHBs a lot of engineering work can be outsourced to countries that seem to have a far more flexible approach.