Monday, September 26, 2016

The Things Are Winning

More than three years ago my friend Jim Gettys, who worked on One Laptop Per Child, and on the OpenWrt router software, started warning that the Internet of Things was a looming security disaster. Bruce Schneier's January 2014 article The Internet of Things Is Wildly Insecure — And Often Unpatchable and Dan Geer's April 2014 Heartbleed as Metaphor were inspired by Jim's warnings. That June Jim gave a talk at Harvard's Berkman Center entitled (In)Security in Home Embedded Devices. That September Vint Cerf published Bufferbloat and Other Internet Challenges, and Jim blogged about it. That Christmas a botnet running on home routers took down the gaming networks of Microsoft's Xbox and Sony's Playstation. That wasn't enough to motivate action to fix the problem.

As I write this on 9/24/16 the preceding link doesn't work, although the Wayback Machine has copies. To find out why the link isn't working and what it has to do with the IoT, follow me below the fold.

The insecurity of the IoT has been a theme of many of my blog posts since 2014, pointing out that it was handing the bad guys, even relatively unskilled bad guys, a weapon that could render the Internet unusable. But nothing has been done to fix the problems and defuse the weapon. Dan Goodin's Why the silencing of KrebsOnSecurity opens a troubling chapter for the ‘Net tells us that we are running out of time:
KrebsOnSecurity, arguably the world's most intrepid source of security news, has been silenced, presumably by a handful of individuals who didn't like a recent series of exposés reporter Brian Krebs wrote. ... On Thursday morning, ... he reported that a sustained attack was bombarding his site with as much as 620 gigabits per second of junk data. ... At 4 pm, Akamai gave Krebs two hours' notice that it would no longer assume the considerable cost of defending KrebsOnSecurity. Krebs opted to shut down the site to prevent collateral damage hitting his service provider and its customers. ... In 2013, attacks against anti-spam organization Spamhaus generated headlines because the 300Gb torrents were coming uncomfortably close to Internet-threatening size. The assault against KrebsOnSecurity represents a much greater threat for at least two reasons. First, it's twice the size. Second and more significant, ... the attacks against KrebsOnSecurity harness so-called Internet-of-things devices—think home routers, webcams, digital video recorders, and other everyday appliances that have Internet capabilities built into them.
Go read the whole article.

This is asymmetric warfare. It doesn't take much skill or many resources to build a DDOS weapon of this kind. But defending against it is beyond the reach of most websites:
Krebs said he has explored the possibility of retaining a DDoS mitigation service, but he found that the cost—somewhere between $100,000 and $200,000 per year for the type of always-on protection he needs against high-bandwidth attacks—is more than he can afford.
So, unless you're seriously wealthy, any time you publish something on the net the bad guys don't like, they can blow your web presence away. Krebs' conclusion is sad:
"Free speech in the age of the Internet is not really free," he said. "We're long overdue to treat this threat with a lot more urgency. Unfortunately, I just don't see that happening right now."
And don't think that knocking out important individual Web sites like KrebsOnSecurity is the limit of the bad guys capabilities. Everyone seems to believe that the current probing of the root servers' defenses is the work of China but, as the Moon Worm showed, careful preparation isn't necessarily a sign of a state actor. There are many bad guys out there who could take the Internet down; the only reason they don't is not to kill the goose that lays the golden eggs.

Pastor Martin Niemöller had it right:
First they came for the security gurus, and I did not speak out -
Because I was not a security guru.
This is probably yet another reason why we need to evolve to a Decentralized Internet (not just a Decentralized Web), perhaps Named Data Networking (NDN). Although, as I wrote, I'm not aware of a major "black hat" analysis of these decentralized proposals, the argument is very plausible.

Why can a large number of small, compromised devices with limited bandwidth upstream bring down a large, powerful Web site, even one defended by an expensive DDOS mitigation service? Two reasons:
  • In today's centralized Internet, the target Web site will be at one, or a small number of IP addresses. The network focuses the traffic from all the compromised devices on to those addresses, consuming massive resources at the target.
  • In today's centralized Web, the target Web site will be be one tenant sharing the resources of a data center, so the focused traffic inflicts collateral damage on the other tenants. It was the cost in resources and the risk to other customers that caused Akamai to kick out KrebsOnSecurity.
In NDN, a request for a resource only travels as far as one of the nearest copies. And in the process it creates additional copies along the path, so that a subsequent request will travel less far. Thus, instead of focusing traffic, large numbers of requests defocus the traffic. They spread the responsibility for satisfying the request out across the infrastructure instead of concentrating it. By moving the load caused by bad behavior closer to the bad actors, it creates incentives for the local infrastructure to detect and prevent the bad behavior.

Denial-of-service attacks are possible in NDN. They take the form of flooding requests for resources that are known not to exist; flooding requests for resources that do exist, such as posts that you don't like, won't work. But both local and cooperative detection and mitigation techniques seem likely to be effective, see for example:
The fundamental problems, as in so many areas, are that the thinking is short-term and the incentives are misaligned. Iain Thomson at The Register reports on a parallel example:
A study by the RAND Corporation, published in the Journal of Cybersecurity, looked at the frequency and cost of IT security failures in US businesses and found that the cost of a break-in is much lower than thought – typically around $200,000 per case. With top-shelf security systems costing a lot more than that, not beefing up security looks in some ways like a smart business decision.

Romanosky analyzed 12,000 incident reports and found that typically they only account for 0.4 per cent of a company's annual revenues. That compares to billing fraud, which averages at 5 per cent, or retail shrinkage (ie, shoplifting and insider theft), which accounts for 1.3 per cent of revenues.
Of course, if because of the insecurity of IoT devices the Internet becomes unusable, or even merely uninteresting once the bad guys have driven anything interesting away, everyone, from the ISPs to the DDOS mitigation services to the IoT device vendors will be out of business. But right now the money is rolling in and it doesn't cost anything to just kick off the targets of the bad guys wrath. Actually fixing things is someone else's problem.

Update 9/25/16: Cory Doctorow writes:
Meanwhile, Krebs was eventually bailed out by Google's Project Shield, one of Jigsaw's anti-"surveillance, extremist indoctrination, and censorship" tools. That right there is another sign of the times: the attacks launched by state-level actors and those who can muster comparable firepower are no match for Google -- so far.
He quotes a post by Krebs called The Democratization of Censorship:
But what we’re allowing by our inaction is for individual actors to build the instrumentality of tyranny. And to be clear, these weapons can be wielded by anyone — with any motivation — who’s willing to expend a modicum of time and effort to learn the most basic principles of its operation.
Krebs post is long but important - go read it now, before it goes away again. If it does, the Wayback Machine has it.


David. said...

Panic over, the Department of Homeland Security is on the case. According to DHS Assistant Secretary for Cyber Policy Robert Silvers:

"What we're going to be doing is drawing on the best approaches, pulling them together and elevating them to get the public's attention."

That should fix the problem in short order.

David. said...

I agree with Dave Farber's note to his IP list. His Stanford talk referenced can be found on YouTube.

David. said...

Some useful links, via Jason Livingood on the IP list:

- Google's Project Shield.

- BITAG has an ongoing review of IoT privacy and security.

- ISOC has a document outlining Mutually Agreed Norms for Routing Security (MANRS).

Dave Taht said...

I note that has had trouble with ddos, too. Burning the library of Alexandria must have been great fun for someone.

I have advocated a do-over in processor design for years now. The system is rotten to its cores.

And as to all your other points, there are a lot of people trying to make IoT constructively better - but it often feels like we're going to lose, no matter what we try. I despair.

David. said...

A lot of reporting on the KrebsOnSecurity DDOS has claimed that it was the largest ever. But as Krebs reported:

"OVH, a major Web hosting provider based in France, said in a post on Twitter this week that it was recently the victim of an even more massive attack than hit my site. According to a Tweet from OVH founder Octave Klaba, that attack was launched by a botnet consisting of more than 145,000 compromised IP cameras and DVRs."

More details on this Terabit DDoS from Dan Goodin here and Pierluigi Paganini here.

David. said...

Symantec has a lot of detail on IoT DDoS attacks.

David. said...

Brian Krebs reports that the source code for the bot network that attacked his site has been released. He writes:

"My guess is that (if it’s not already happening) there will soon be many Internet users complaining to their ISPs about slow Internet speeds as a result of hacked IoT devices on their network hogging all the bandwidth. On the bright side, if that happens it may help to lessen the number of vulnerable systems.

On the not-so-cheerful side, there are plenty of new, default-insecure IoT devices being plugged into the Internet each day. Gartner Inc. forecasts that 6.4 billion connected things will be in use worldwide in 2016, up 30 percent from 2015, and will reach 20.8 billion by 2020. In 2016, 5.5 million new things will get connected each day, Gartner estimates."

David. said...

Daniel Cid has some detailed analysis of a large router-based DDoS, and links to some good advice on router security and a tool for checking one aspect of your router's security.

David. said...

Analysts of the botnet source code are not impressed:

"If mediocre malware can power some of the largest DDoS attacks ever, and considering the sad state of security of the Internet of Things in general, we should probably brace for more cyberattacks powered by our easy-to-hack “smart” Internet of Things, as many, including ourselves, had predicted months ago."

A security researcher said:

“I am just surprised at how such a trivial attack code could be responsible for such a large DDoS. It really says a lot more about the state of IoT security than the specifics of the malware, ... If people still aren't changing default passwords and disabling telnet on Internet connected equipment in 2016 then we are heading to a future with more incidents like this happening.”

David. said...

John Leyden at The Register points to a report from PenTestPartners on potential improvements to the Mirai botnet.

David. said...

Lorenzo Franceschi-Bicchierai at Motherboard writes Internet of Things Malware Has Apparently Reached Almost All Countries on Earth:

"mperva, a company that provides protection to websites against Distributed Denial of Service (DDoS) attacks, is among the ones who have been busy investigating Mirai. According to their tally, the botnet made of Mirai-infected devices has reached a total of 164 countries. A pseudonymous researcher that goes by the name MalwareTech has also been mapping Mirai, and according to his tally, the total is even higher, at 177 countries."

David. said...

Many Things in the Internet are connected via cellular gateways, which turn out to have default passwords and thus are vulnerable to Mirai:

"Sierra said in an alert that the company has “confirmed reports of the ‘Mirai’ malware infecting AirLink gateways that are using the default ACEmanager password and are reachable from the public internet.” Sierra Wireless LS300, GX400, GX/ES440, GX/ES450, and RV50 routers were identified in the bulletin as vulnerable to compromise by Mirai. ... Sierra markets AirLink gateways as a workhorse for IoT deployments in a variety of “mission critical” contexts from energy to smart city. ... While many deployed cellular gateways are protected from roving Internet scans, tens of thousands of devices are publicly accessible. A scan for Sierra GX400 gateways using the Shodan search engine, for example, turned up more than 6,000 such devices that are reachable from the public Internet."

David. said...

John Leyden at flags a report from Fastly’s Director of Security Research, Jose Nazario:

"On average, an IoT device is infected with malware and will launch an attack within six minutes of being exposed to the internet. IoT devices are probed for vulnerabilities 800 times per hour by attackers from across the globe. Every day there is an average of over 400 login attempts per device, an average of one attempt every five minutes and 66 per cent of them on average are successful, according to Nazario."

David. said...

Steve Lohr at the New York Times writes Stepping Up Security for an Internet-of-Things World:

"The Level 3 researchers, working with Flashpoint, an internet risk-management firm, found that as many as one million devices, mainly security cameras and video recorders, had been harnessed for so-called botnet attacks. They called it “a drastic shift” toward using internet-of-things devices as hosts for attacks instead of traditional hosts, such as hijacked data center computers and computer routers in homes."

DARPA has a challenge to address this problem:

"The Cyber Grand Challenge was announced in 2013, and qualifying rounds began in 2014. At the outset, more than 100 teams were in the contest. Through a series of elimination rounds, the competitors were winnowed to seven teams that participated in the finals in August in Las Vegas. The three winning teams collected a total of $3.75 million in prize money."

The result:

"The first-place team, which won $2 million, was a group from ForAllSecure, a spinoff from Carnegie Mellon University. Hours after the Darpa contest, its cyberreasoning software, called Mayhem, went up against the best human teams at Defcon, an annual hacking competition.

In that three-day contest, Mayhem held its own for two days and proved itself to be extremely strong on defense. But by the third day, the human experts had come up with more innovative exploits than Mayhem, said David Brumley, a professor at Carnegie Mellon and chief executive of ForAllSecure."

I don't think that providing better, more intelligent defense at the service is a viable approach. This is asymmetric warfare. It is very cheap for the attacker to collect virtually unlimited resources. It is increasingly expensive for the defense to resist. The end-point is that running a web service becomes unaffordable, and the thus the Web becomes uninteresting. This mismatch between the the cost of sending a request and the cost of servicing it is a problem we identified in our 2003 SOSP paper:

"One application of our principle of inertia is that large changes to the system require large efforts. In a protocol where some valid messages cost nothing to produce, but cause the expenditure of great effort - e.g., a cheap request causing its recipient to hash a large amount of data - this principle is unfulfilled. To satisfy our inertia requirement in LOCKSS, we adjust the amount of effort involved in message exchanges for voting, discovery, and poll initiation, by embedding extra, otherwise unnecessary effort."

DARPA is not being radical enough, because their competition aims to provide better defenses for IP networks. But the vulnerability that leads to the effectiveness of DDoS attacks is fundamental to IP. Services need to be at some small number of IP addresses. The large number of compromised clients can focus huge amounts of traffic on those addresses, so that the service needs large amounts of expensive resource to defend itself. This fundamental cost asymmetry can't be fixed by better defenses.

ISPs have no real economic motivation to implement advanced technologies to prevent their customers IoT devices being used to DDoS sites. The economics of the IoT don't support adequate (or even any) security for the devices. So the raw materials for DDoS botnets are going to be cheap and abundant. We need to re-architect the IP-level network infrastructure so that it isn't vulnerable to cheap, abundant, insecure nodes. This is a very hrad problem - note how long it has taken to deploy IPv6. We likely don't have that long.

David. said...

Karl Bode at TechDirt writes:

"A new report by Akamai warns that hackers are using a 12-year-old vulnerability in OpenSSH to funnel malicious network traffic through IoT devices. SSH certainly can be implemented securely, but as with every other security aspect of the IoT, many hardware vendors aren't bothering to do so. Akamai's data indicates roughly 2 million devices have been compromised by this type of hack, which the firm dubs SSHowDowN.

CVE-2004-1653 is a default configuration in old versions of OpenSSH that can be exploited by an attacker to forward ports,"

He links to Kreb's recent post IoT Devices as Proxies for Cybercrime:

"This post looks at how crooks are using hacked IoT devices as proxies to hide their true location online as they engage in a variety of other types of cybercriminal activity — from frequenting underground forums to credit card and tax refund fraud."

David. said...

Few IoT devices use Intel processors, but to illustrate how hard security is, here is Dan Goodin's Flaw in Intel chips could make malware attacks more potent:

"ASLR, short for "address space layout randomization," is a defense against a class of widely used attacks that surreptitiously install malware by exploiting vulnerabilities in an operating system or application. By randomizing the locations in computer memory where software loads specific chunks of code, ASLR often limits the damage of such exploits to a simple computer crash, rather than a catastrophic system compromise. Now, academic researchers have identified a flaw in Intel chips that allows them to effectively bypass this protection."

The flaw is a side channel in the branch target buffer table that leaks address information.

David. said...

Most Things in the Internet run Linux kernels, because they don't add cost. Dan Goodin at Ars Technica has a post entitled “Most serious” Linux privilege-escalation bug ever is under active exploit with the sub-head Lurking in the kernel for nine years, flaw gives untrusted users unfettered root access:

"The vulnerability, a variety known as a race condition, was found in the way Linux memory handles a duplication technique called copy on write. Untrusted users can exploit it to gain highly privileged write-access rights to memory mappings that would normally be read-only. More technical details about the vulnerability and exploit are available here, here, and here. Using the acronym derived from copy on write, some researchers have dubbed the vulnerability Dirty COW."

It turns out that this 9-year-old is a bit of an survivor:

"Disclosure of the nine-year-old vulnerability came the same week that Google researcher Kees Cook published research showing that the average lifetime of a Linux bug is five years."

With this bug included Cook's analysis is:

"Critical: 3 @ 5.2 years average, High: 44 @ 6.2 years average, Medium: 404 @ 5.3 years average, Low: 216 @ 5.5 years average"

Cook writes:

"While we’re getting better at fixing bugs, we’re also adding more bugs. And for many devices that have been built on a given kernel version, there haven’t been frequent (or some times any) security updates, so the bug lifetime for those devices is even longer. ... The systems using a Linux kernel are right now running with security flaws. Those flaws are just not known to the developers yet, but they’re likely known to attackers,"

The point of this, and of Familiarity Breeds Contempt is that the Things in the Internet are always going to be insecure, and the insecurities are going to last a long time even if means to patch the devices are available. So although regulation to force Things to have an update mechanism would be good, it isn't likely to make much difference.

In fairness, I should point out that 3 critical bugs in 11 years for a codebase the size of the Linux kernel is actually pretty good. But in a way that's not reassuring.

David. said...

The kind of disruption that Mirai and other IoT-based botnets can do is show by this DDoS:

"Twitter, Reddit, Github, Spotify, and many others were knocked offline intermittently on Friday morning as a result of a [...] “global” Distributed Denial of Service or DDoS attack on Dyn, a company that provides core internet services for those popular websites. The attack mainly targeted Dyn’s Domain Name System (DNS) management services infrastructure on the East Coast of the United States, as the company explained in a statement."

David. said...

As Brian Krebs points out, compromised IoT devices can be used for things other than DDoS-ing:

"What he observed was that all of the systems were being used for a variety of badness, from proxying Web traffic destined for cybercrime forums to testing stolen credit cards at merchant Web sites. Further study of the malware files and the traffic beacons emanating from the honeypot systems indicated his honeypots were being marketed on a Web-based criminal service that sells access to SOCKS proxies in exchange for Bitcoin."