Tuesday, April 2, 2019

First We Change How People Behave

Then the system will work the way we want. My skepticism about Level 5 self-driving cars keeps getting reinforced. Below the fold, two recent examples.

The fundamental problem of autonomous vehicles sharing roads is that until you get to Level 5, you have a hand-off problem. The closer you get to Level 5, the worse the hand-off problem.

Source
Sean Gallagher's Lion Air 737 MAX crew had seconds to react, Boeing simulation finds shows the hand-off problem for aircraft:
In testing performed in a simulator, Boeing test pilots recreated the conditions aboard Lion Air Flight 610 when it went down in the Java Sea in October, killing 189 people. The tests showed that the crew of the 737 MAX 8 would have only had 40 seconds to respond to the Maneuvering Characteristics Augmentation System’s (MCAS’s) attempts to correct a stall that wasn’t happening before the aircraft went into an unrecoverable dive, according to a report by The New York Times.

While the test pilots were able to correct the issue with the flip of three switches, their training on the systems far exceeded that of the Lion Air crew—and that of the similarly doomed Ethiopian Airlines Flight 302, which crashed earlier this month. The Lion Air crew was heard on cockpit voice recorders checking flight manuals in an attempt to diagnose what was going on moments before they died.
Great, must-read journalism from Dominic Gates at the Seattle Times, Boeing's home-town newspaper in Flawed analysis, failed oversight: How Boeing and FAA certified the suspect 737 MAX flight control system shows that the fundamental problem with the 737 MAX was regulatory capture of the FAA by Boeing; the FAA's priority wasn't to make the 737 MAX safe, it was to get it into the market as quickly as possible because Airbus had a 9-month lead in this segment. And because Airbus' fly-by-wire planes minimize the need for expensive pilot re-training, Boeing's priority was to remove the need for it.
The company had promised Southwest Airlines Co. , the plane’s biggest customer, to keep pilot training to a minimum so the new jet could seamlessly slot into the carrier’s fleet of older 737s, according to regulators and industry officials.

[Former Boeing engineer Mr. [Rick] Ludtke [who worked on 737 MAX cockpit features] recalled midlevel managers telling subordinates that Boeing had committed to pay the airline $1 million per plane if its design ended up requiring pilots to spend additional simulator time. “We had never, ever seen commitments like that before,” he said.
The software fix Boeing just announced is just a patch on a fundamentally flawed design, as George Leopold reports in Software Won’t Fix Boeing’s ‘Faulty’ Airframe. Boeing is gaming the regulations, and the FAA let them do it. Neither placed safety first. These revelations should completely destroy the credibility of FAA certifications.

Although Boeing's highly-trained test pilots didn't have to RTFM, they did have only 40 seconds to diagnose and remedy the problem caused by the faulty angle-of-attack sensor and the buggy MCAS software. Inadequately trained Lion Air and Ethiopian Airlines pilots never stood a chance of a successful hand-off. Self-driving car advocates assume that hand-offs are initiated by the software recognizing a situation it can't handle. But in this case the MCAS software was convinced, on the basis of a faulty sensor, that it was handling the situation and refused to hand-off to the pilots 24 times in succession.

Self-driving car stopper
Self-driving cars drivers will lack even the level of training of the dead pilots. The cars' software is equally dependent upon sensors, which can be fooled by stickers on the road*, and cannot handle rain, sleet or snow. Or, as it turns out, pedestrians As David Zipper tweeted:
Atrios' apt comment was:
It is this type of thing which makes me obsess about this issue. And I have a couple insider sources (ooooh I am a real journalist) who confirm these concerns. The self-driving car people see pedestrians as a problem. I don't really understand how you can think urban taxis are your business model and also think walking is the enemy. Cities are made of pedestrians. Well, cities other than Phoenix, anyway. I pay a dumb mortgage so I can walk to a concert, like I did last night.
But no-one who matters cares about pedestrians because no-one who matters is ever on the sidewalk, let alone crossing the street. As the CDC reports:
In 2016, 5,987 pedestrians were killed in traffic crashes in the United States. This averages to one crash-related pedestrian death every 1.5 hours.

Additionally, almost 129,000 pedestrians were treated in emergency departments for non-fatal crash-related injuries in 2015. Pedestrians are 1.5 times more likely than passenger vehicle occupants to be killed in a car crash on each trip.
The casualties who don't "know what they can't do" won't add much to the deaths and injuries, so we can just go ahead and deploy the technology ASAP.



* Tesla says the "stickers on the road" attack:
is not a realistic concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should always be prepared to do so
Well, yes, but the technology is called "Autopilot" and Musk keeps claiming "full autonomy" is just around the corner.

29 comments:

David. said...

Sean Gallagher reports that:

"Delivery of Boeing’s promised fix to the flight system software at the center of two 737 MAX crash investigations has been pushed back several weeks after an internal review by engineers not connected to the aircraft raised additional safety questions. The results of the “non-advocate” review have not been revealed, but the Federal Aviation Administration confirmed on April 1 that the software needed additional work."

David. said...

Although they did RTFM, it looks like it didn't help:

"Pilots at the controls of the Boeing Co. 737 MAX that crashed in March in Ethiopia initially followed emergency procedures laid out by the plane maker but still failed to recover control of the jet, according to people briefed on the probe’s preliminary findings."

David. said...

In Whistleblowers: FAA 737 MAX safety inspectors lacked training, certification, Sean Gallagher reports that:

"Multiple whistleblowers have raised issues over the Federal Aviation Administration’s safety inspection process connected to Boeing’s 737 MAX aircraft, according to a letter to the FAA from Senate Commerce Committee chairman Sen. Roger Wicker on April 2. And the FAA’s leadership was informed of these concerns as far back as August of 2018.

The whistleblowers cited “insufficient training and improper certification” of FAA aviation safety inspectors, “including those involved in the Aircraft Evaluation Group (AEG) for the Boeing 737 MAX," Wicker said in his letter to FAA acting administrator David Elwell."

Both Boeing and the FAA have serious credibility problems.

David. said...

Izabella Kaminska and Jamie Powell Uber's conflicting self-driving fleet vision analyzes Uber's IPO documents and shows (a) Uber is betting the future on a fleet of Level 5 cars, and (b) the economics of this bet simply don't work (and of course neither does the technology):

"But here's the really important factor for would-be buyers of the stock on IPO day. Uber says autonomous driving is essential for it to continue to effectively compete, but it also says these development efforts are capital and operations intensive (the opposite of its supposed asset-light business model today)."

The quotes they emphasize from the IPO documents are fairly devastating.

David. said...

Yet again William Gibson was prophetic. In Defense against the Darknet, or how to accessorize to defeat video surveillance, Thomas Claiburn describes a real-life version of the "ugliest T-shirt" from Gibson's Zero History.

David. said...

Julie Bort's An engineer at Uber's self-driving car unit warns that it's more like 'a science experiment' than a real car capable of driving itself shows that in autonomous cars, like everything else, Uber is following the "fake it until you make it" path of today's Silicon Valley startups.

And for the few in the audience who haven't read Gibson, the "ugliest T-shirt" makes the wearer invisible to surveillance cameras. Makes pedestrians even more of a problem for self-driving cars, no?

David. said...

Another good post on the 737-MAX crashes is How the Boeing 737 Max Disaster Looks to a Software Developer by Gregory Travis:

"So Boeing produced a dynamically unstable airframe, the 737 Max. That is big strike No. 1. Boeing then tried to mask the 737’s dynamic instability with a software system. Big strike No. 2. Finally, the software relied on systems known for their propensity to fail (angle-of-attack indicators) and did not appear to include even rudimentary provisions to cross-check the outputs of the angle-of-attack sensor against other sensors, or even the other angle-of-attack sensor. Big strike No. 3.

David. said...

Christine Negroni's What people don’t get about why planes crash stresses the handoff problem:

"In the crash of an Asiana Airlines Boeing 777 landing in San Francisco in 2013, investigators determined that a contributing factor was the pilots’ over-reliance on automated systems which led to an erosion in their flying skills. The investigation of the fatal flight of an Air France Airbus A330 from Rio de Janeiro to Paris in 2009 led to the conclusion that the complexity of the fly-by-wire airplane befuddled the pilots.

The 737 Max probes suggest another variation on the conundrum: Technology intended to protect against pilot error trapped the pilots. Helpless in the cockpit, they were unable to do as Captain Sully did and save the day."

David. said...

Southwest and FAA officials never knew Boeing turned off a safety feature on its 737 Max jets, and dismissed ideas about grounding them by Hillary Hoffower is based on reporting by Andy Pastzor of the WSJ:

"Southwest Airlines and the Federal Aviation Administration (FAA) officials who monitor the carrier were unaware that a standard safety feature, designed to warn pilots about malfunctioning sensors, on Boeing 737 Max jets was turned off when Southwest began flying the model in 2017 ... In earlier 737 models, the safety feature alerted pilots when a sensor called the "angle-of-attack vane" incorrectly conveyed the pitch of the plane's nose, according to Pastzor. In the Max, it functions as such while also signaling when the Maneuvering Characteristics Augmentation System (MCAS) — a new automated system linked to both October's Lion Air crash and March's Ethiopian Airlines crash — could misfire; but these alerts were only enabled if carriers purchased additional safety features"

And:

"Like other airlines flying the Max, Southwest didn't learn about the change until the aftermath of the Lion Air crash, ... the carrier then asked Boeing to reactivate the alerts on its Max fleet, causing FAA inspectors to contemplate grounding the Max fleet until it was determined whether or not pilots needed additional training — but the idea was quickly dropped.

Once the feature was reactivated, some FAA officials again considered grounding Southwest's 737 Max fleet to determine whether pilots needed new training — and again, the discussions, which happened via email, were dismissed after a few days"

It is clear that the FAA's priority was Boeing's competitive position against Airbus, not safety. Additional training would have cost Boeing $1M a plane to Southwest, and would have cost Southwest probably more than that in increased costs covering the grounded planes and unavailable pilots.

David. said...

As usual, Paul Vixie was way ahead of the curve. He wrote Disciplining the Unoccupied Mind in July 2016:

"Simply put, if you give a human brain the option to perform other tasks than the one at hand, it will do so. No law, no amount of training, and no insistence by the manufacturer of an automobile will alter this fact. It's human nature, immalleable. So until and unless Tesla can robustly and credibly promise an autopilot that will imagine every threat a human could imagine, and can use the same level of caution as the best human driver would use, then the world will be better off without this feature."

I wrote Techno-hype part 1 16 moths later, and this post 32 months later, both with esentially the same message.

David. said...

Uber, Lyft, Waymo and many others believe that the key market for semi-autonomous (Level 4) cars is robo-taxis. Via Jamie Powell's The questionable economics of autonomous taxi fleets,

"A new paper out Monday, written by researchers at the Massachusetts Institute of Technology and exclusively shared with FT Alphaville, agrees. It suggests that, at current prices, an automated hive of driverless taxis will actually be more expensive for a consumer to use than the old-world way of owning four wheels.

Drawing on a wealth of publicly available data, Ashley Nunes and his colleague Kristen Hernandez suggest that the price for taking an autonomous taxi will be between $1.58 to $6.01 on a per-mile basis, versus the $0.72 cost of owning a car. Using San Francisco’s taxi market as its test area, the academics examined a vast array of costs such as licensing, maintenance, fuel and insurance for their calculations."

Note the "San Francisco". Waymo can't actually make robo-taxis work in Phoenix. The big markets for taxis are old, dense cities such as San Francisco and New York. Nightmares even for human drivers (try driving through Chinatown in SF, or across Manhattan in rush hour).

David. said...

Boeing Built Deadly Assumptions Into 737 Max, Blind to a Late Design Change is the New York Times longread on the process that led to the 737 MAX disasters. It is a story of a siloed organization, with people making safety-critical decisions based on partial or incorrect information about the system in question. It should make everyone think twice before flying on any Boeing plane:

"But many people involved in building, testing and approving the system, known as MCAS, said they hadn’t fully understood the changes. Current and former employees at Boeing and the Federal Aviation Administration who spoke with The New York Times said they had assumed the system relied on more sensors and would rarely, if ever, activate. Based on those misguided assumptions, many made critical decisions, affecting design, certification and training."

David. said...

Clive Irving's How Boeing’s Bean-Counters Courted the 737 MAX Disaster is another good article on how the crisis arose:

"The origins of the 737 are particularly significant now, with Boeing engulfed in a world crisis of confidence with two crashes of the newest model, the 737 MAX-8, killing 346 people. Specifically, the origins of the design highlight the consequences to Boeing of believing that it could keep upgrading a 50-year-old design indefinitely."

David. said...

April Glaser interviewed self-driving car pioneer Chris Urmson for How Close Are We to Self-Driving Cars, Really?. He didn't disagree with her question:

"I’ve read that you think self-driving cars are about five to 10 years away from a small-scale rollout, but 30 to 50 years away from ubiquity, or a very large rollout."

David. said...

Boeing's disregard of safety in manufacturing and slow-rolling of FAA oversight goes backmany years before the 737 MAX disasters, according to a long story by Michael Laris entitled Long before the Max disasters, Boeing had a history of failing to fix safety problems:

"Repeatedly, safety lapses were identified, and Boeing would agree to fix them, then fail to do so, the FAA said."

David. said...

In Boeing falsified records for 787 jet sold to Air Canada. It developed a fuel leak Katie Nicholson reports that:

"Boeing staff falsified records for a 787 jet built for Air Canada which developed a fuel leak ten months into service in 2015.

In a statement to CBC News, Boeing said it self-disclosed the problem to the U.S. Federal Aviation Administration after Air Canada notified them of the fuel leak.

The records stated that manufacturing work had been completed when it had not."

David. said...

Matt Stoller's The Coming Boeing Bailout? is a good overview of the way anti-trust failure corrupted Boeing:

"The net effect of the merger, and the follow-on managerial and financial choices, is that America significantly damaged its aerospace industry. Where there were two competitors - McDonnell Douglas and Boeing, now there is one. And that domestic monopoly can no longer develop good civilian aerospace products. Hundreds of people are dead, and tens of billions of dollars wasted."

David. said...

Jeffrey Rothfeder's For years, automakers wildly overpromised on self-driving cars and electric vehicles—what now? shows that realism about self-driving cars without trained self-driving car drivers is breaking out, now the Uber IPO is over:

"Starting around May 2016, Uber projected in public and private presentations that it would manufacture 13,000 autonomous vehicles by 2019, only to change that forecast four months later to over 75,000 units. The company also said that human safety drivers, who take over the wheel when an AV needs help, would not be required on its cars by 2020. And in 2022, the company declared, tens of thousands of fully self-driving Uber taxis would be in 13 of the largest cities. ... the Uber employee responsible for the forecasts said that while she was designing them, executives had asked her “to think about a way” to show accelerated Uber AV development."

But now:

"CEO Dara Khosrowshahi said at an Economic Club meeting in Washington, DC, that it will take more than 50 years for all Uber cars to be driverless,"

And:

"Waymo’s CEO John Krafcik told a tech conference that it will be decades before autonomous cars are widespread on the roads, and they may always need human assistance to drive in multifaceted environments, such as bad weather or areas crowded with construction or emergency equipment."

Told you so!

David. said...

The details in Newly stringent FAA tests spur a fundamental software redesign of Boeing’s 737 MAX flight controls seem somewhat confused, but apparently the fact that MCAS, unlike earlier flight control systems, can override the pilots in ways from which they may be unable to recover means that the fundamental architecture of the 737's flight control software is no longer adequate. The FAA is requiring that the software be re-architected to be more resilient to failures. If so, the predicitions of an early return to service are highly optimistic.

David. said...

Gareth Corfield at The Register has more details on the 737 MAX software re-architecture:

"Astonishingly, until the 737 Max crashes, the aircraft was flying with no redundancy at all for the flight control computers. If the active one failed or suffered inversion of critical bits in memory, there was no standby unit ready to cut in and continue. The Seattle Times reported that this has now been redesigned so the two onboard computers run in an active:standby configuration. Previously the units merely swapped over in between flights.

In addition, the computers will receive input from both angle-of-attack sensors rather than just the one. A faulty AoA sensor is thought to have been a contributory factor to the 737 Max crashes, which together cost more than 300 lives."

David. said...

Andy Greenberg's A Boeing Code Leak Exposes Security Flaws Deep in a 787's Guts reports that:

"Santamarta claims that leaked code has led him to something unprecedented: security flaws in one of the 787 Dreamliner's components, deep in the plane's multi-tiered network. He suggests that for a hacker, exploiting those bugs could represent one step in a multi­stage attack that starts in the plane’s in-flight entertainment system and extends to highly protected, safety-critical systems like flight controls and sensors."

This isn't an immediate threat to safety-critical systems which Boeing claims are firewalled:

"But even granting Boeing's claims about its security barriers, the flaws Santamarta found are egregious enough that they shouldn't be dismissed, says Stefan Savage, a computer science professor at the University of California at San Diego, who is currently working with other academic researchers on an avionics cybersecurity testing platform. "The claim that one shouldn't worry about a vulnerability because other protections prevent it from being exploited has a very bad history in computer security," Savage says. "Typically, where there's smoke there's fire."

Savage points in particular to a vulnerability Santamarta highlighted in a version of the embedded operating system VxWorks, in this case customized for Boeing by Honeywell."

Maybe Boeing needs to pay software developers more than $9/hr.

David. said...

Via Atrios, el gato malo has a good explanation of why, even if Level 5 self-driving were possible, Tesla's "full self-driving" is never going to be it.

David. said...

Joining the pile on Tesla's robo-taxi BS, Keubiko's Tesla's Robotaxi Red Herring estimates the cost in crashes and deaths they're projecting:

"Even if autonomous cars are as good has human drivers by 2023, is it reasonable or feasible to think that the news flow, consumer acceptance, politicians, and regulators will accept anywhere near these numbers? If a single Uber test vehicle death can send the industry into a tizzy, what would thousands of crashes per day and a death every 90 minutes or so look like? This even ignores the stats on the miles that would be owner-driven (in autonomous mode) and not “robotaxi”.

As an analogue, look at what Boeing is dealing with on its 737 Max. Air travel is still statistically very safe, and the 737 Max had well over 40,000 flights before the two crashes within 5 months grounded (justifiably so) the fleet.

Does anyone honestly believe that a newly emerging industry can withstand the news flow anywhere close to these numbers?"

David. said...

How does an autonomous car work? Not so great by Youjin Shin, Chris Alcantara and Aaron Steckelberg at the WaPo is a great interactive explanation of many of the limitations of self-driving car technology other than the hand-off problem. Go check it out.

David. said...

Jennifer Elias' Alphabet exec blames media for overhyping self-driving cars, even though Google drove the hype illustrates the slow dawning of the realization that Level 5 is nowhere close:

"[Waymo] has dialed back its enthusiastic tone as it falls behind its original timeline for getting full self-driving cars on the road. The company said in 2017 that it wouldn't need to wait until 2020 ⁠— when analysts expected self-driving cars to go fully autonomous ⁠— but that it would give riders the ability within "months."

Morgan Stanley cut its valuation on Waymo by 40% last month from $175 billion to $105 billion, concluding that the industry is moving toward commercialization slower than expected and that Waymo still relies on human safety drivers, which CNBC reported in August."

Elias provides a timeline of Google's optimistic predictions of "full self-driving".

David. said...

In Hailing a driverless ride in a Waymo, Ed Niedermeyer reports on his first ride in a fully driverless Waymo car, as part of their testing in a small part of Phoenix, AZ:

"There were moments where the self-driving system’s driving impressed, like the way it caught an unprotected left turn just as the traffic signal turned yellow or how its acceleration matched surrounding traffic. The vehicle seemed to even have mastered the more human-like driving skill of crawling forward at a stop sign to signal its intent.

Only a few typical quirks, like moments of overly cautious traffic spacing and overactive path planning, betrayed the fact that a computer was in control. A more typical rider, specifically one who doesn’t regularly practice their version of the driving Turing Test, might not have even noticed them."

But he points out that:

"In 2017, Waymo CEO John Krafcik declared on stage at the Lisbon Web Summit that “fully self-driving cars are here.” Krafcik’s show of confidence and accompanying blog post implied that the “race to autonomy” was almost over. But it wasn’t.

Nearly two years after Krafcik’s comments, vehicles driven by humans — not computers — still clog the roads in Phoenix. The majority of Waymo’s fleet of self-driving Chrysler Pacifica minivans in Arizona have human safety drivers behind the wheel; and the few driverless ones have been limited to testing only."

David. said...

Remember the Uber self-driving car that killed a woman crossing the street? The AI had no clue about jaywalkers by Katyanna Quach describes the NTSB report on the killing of a pedestrian by an Uber self-driving car in March 2018:

"an investigation by the NTSB into the crash has pinpointed a likely major contributing factor: the code couldn't recognize her as a pedestrian, because she was not at an obvious designated crossing. Rather than correctly anticipating her movements as a person moving across the road, it ended up running right into her.

“The system design did not include a consideration for jaywalking pedestrians,” the watchdog stated [PDF] in its write-up.”

The penalty for jaywalking is death.

David. said...

Timothy B. Lee's How terrible software design decisions led to Uber’s deadly 2018 crash has more detail on just how bad Uber's software was.

David. said...

In Another company is dialing back expectations for self-driving taxis Timothy B Lee reports more realism dawning on the self-driving hype:

"Daimler is planning to "rightsize" its spending on self-driving taxis, Chairman Ola Källenius said on Thursday. Getting self-driving cars to operate safely in complex urban environments has proved more challenging than people expected a few years ago, he admitted.

"There has been a reality check setting in here," Källenius said, according to Reuters.

He is just the latest executive to acknowledge that work on self-driving taxi technology is not progressing as fast as optimists expected two or three years ago. Earlier this year, Ford CEO Jim Hackett sought to dampen expectations for Ford's own self-driving vehicles. Industry leaders Waymo and GM's Cruise missed self-imposed deadlines to launch driverless commercial taxi services in 2018 and 2019, respectively.