Tuesday, December 1, 2020

737 MAX Ungrounding

My post 737 MAX: The Case Against Boeing is a year old and has accumulated 58 updates in comments. Now the aircraft is returning to service, it is time for a new post. Below the fold, Bjorn Fehrm has two interesting posts about the ungrounding.

In the first, Boeing 737 MAX changes beyond MCAS, Fehrm lays out the cascade of warnings that resulted from a single angle-of-attack sensor failure:
As FAA and Boeing played through what happened in the MAX crashes in Boeing’s engineering simulators, the cascading alerts triggered by a faulty single Angle of Attack (AoA) sensor stood out:
  • Stick shaker went on on the affected side from rotation and stayed on all the time, despite the aircraft flying with the correct speed and not being close to stall.
  • IAS (airspeed) UNRELIABLE alert triggered
  • ALT (altitude) UNRELIABLE alert triggered
  • AOA (Angle of Attack) UNRELIABLE should have shown but didn’t because of a bug in MAX’s software that tied it to the optional display of AoA on the Pilot’s Primary Flight Display (PFD, the Pilot’s electronic horizon display).
  • The speed tapes on the Pilot’s Primary Flight Display behaved strangely, showing too low speed and high speed concurrently in the ET302 case.
Several trim related failures in such an environment relied on the Pilots identifying the trim misbehavior within four seconds. When flight crews from different airlines were flying these scenarios, it became clear such assumptions were unrealistic.
This is an example of the hand-off problem that is inherent in sophisticated automation (see First We Change How People Behave and the numerous comments). Clearly, giving even expert pilots only 4 seconds to comprehend and react to this confusing rush of warnings would have been unrealistic, even if the pilots had been informed about and trained on the MCAS system that was causing them, which they weren't.

In the second, Fehrm points out an interesting difference between the FAA's and the EASA's requirements for re-certifying the 737 MAX in 737 MAX ungrounding, ANAC’s and EASA’s decisions:
The other condition has its root in the disconnection of Speed Trim, MCAS, Autopilot, and Flight Directors should the two Angle of Attack systems disagree. EASA will temporarily revoke the 737 MAX certification for Required Navigation Performance – Authorization Required (RNP AR) approaches.
...
Should the AoA monitor trip, Speed Trim, MCAS, and more importantly, Autopilot and Flight Directors disconnect, it puts a crew in a very tight spot as the difficulty of such approaches are high (they require special crew training and certification). You need all the tools you have in such approaches and don’t want a sudden disconnect of the Autopilot and Flight Directors combined with Speed Trim warning, followed by AOA, IAS and ALT DISAGREE.

The revoke of the RPN AR approach certification is temporary. One can guess it will be allowed again once a synthetic third AoA sensor is introduced to the MAX. It creates a voting “two versus one” situation when one of the sensors presents suspicious values. It would then result in an AOA DISAGREE warning, but the Autopilot and Flight directors would stay on and IAS and ALT would still get the required AoA corrections. The AOA DISAGREE is then an indication for required maintenance rather than a major system hiccup.
Duplicating systems is never a good approach to fault tolerance, they must be triplicated. In the 70s BA used Tridents on the Edinburgh to London shuttle. Their autoland systems were triplcated, and certified for zero-visibility landing. I experienced my first go-round when, on my way from Edinburgh to Miami for a conference, the approach to LHR in heavy cloud was interrupted by the engines spooling up and an abrupt climb. The captain calmly announced that one of the autopilots disagreed with the other two and, as a precaution, we were going around for another try. On the second approach there was no disagreement. We eventually landed in fog so thick I couldn't see the wingtips. Only the Tridents were landing, nothing was taking off. My Miami flight was delayed and after about 10 hours I was re-routed via LGA.

28 comments:

David. said...

Lori Aratani and Ian Duncan report that Boeing ‘inappropriately coached’ FAA test pilots during review of 737 MAX after deadly crashes, Senate investigators say:

"Senate investigators concluded that Boeing “inappropriately coached” Federal Aviation Administration’s (FAA) pilots for a simulator test last year conducted during the effort to test and recertify the company’s 737 MAX as safe to fly again after two deadly crashes.

The conclusion is contained in a report issued Friday by the Republican majority in the Senate Commerce Committee on an investigation that was launched after the two MAX crashes but that ultimately broadened to unearth numerous safety problems across the FAA.
...
The FAA inspector alleged the Boeing official told the pilots, “Remember, get right on that pickle switch” — meaning an electrical thumb switch on the control column used to pitch up the jet’s nose."

David. said...

Bryan Corliss' Boeing should build 757 replacement in Washington looks ahead to the next big mistake Boeing is going to make:

"Washington, by every relevant objective measure The Teal Group could think of, is categorically the best place for Boeing to keep its commercial operations. And Washington, by every subjective measure that Boeing uses, is absolutely a terrible place for Chicago to run its business the way it wants.

Since the 1997 McDonnell Douglas merger, Boeing adopted the Jack Welch/GE management model. Former CEO Harry Stonecipher was a Welch acolyte. So was his successor, Jim McNerney. Current CEO David Calhoun spent 26 years at GE, sitting on that company’s board of directors for eight years and rising to the rank of vice chairman.

The GE model is based on a foundation of ruthless competition, among suppliers, among employees and among communities where a company bases its business units. Welch famously once said his dream would be to put factories “on a barge” where he could move them – or threaten to move them – to lower cost sites that are “more competitive.”

That business culture clashes severely with the traditional Seattle culture, which is heavily, maybe obsessively, based on consensus building and “win-win” solutions. It also views unions favorably."

Boeing is very late responding to the A321 Neo. Moving the headquarters to Chicago and the 787 to Charleston have proven to be disasters:

"With the benefit of 20/20 hindsight, we can see that the road to where we are today was paved the day in June 2005 when the Boeing board chose McNerney to be CEO, thus embracing the ethos of ruthless competition and rejecting Alan Mulally’s brand of “Working Together” leadership. Mulally went on to guide Ford through one of the greatest crises of its history; McNerney’s tenure led Boeing to … this, a situation so bad they’ve sold off the corporate yacht."

David. said...

Dominic Rushe reports that US fines Boeing $2.5bn following fraud charges tied to 737 Max crashes:

"Boeing has been fined $2.5bn by the US justice department after being charged with fraud and conspiracy in connection with the fatal crashes of its 737 Max airliner.

The Max was grounded worldwide in March 2019 after crashes in Indonesia and Ethiopia that killed 346 people. A March 2020 congressional concluded that Boeing promoted a “culture of concealment” and was “grossly inefficient” in its oversight of the Max’s development.

“The tragic crashes of Lion Air Flight 610 and Ethiopian Airlines Flight 302 exposed fraudulent and deceptive conduct by employees of one of the world’s leading commercial airplane manufacturers,” acting assistant attorney general David Burns of the justice department’s criminal division wrote in a release."

David. said...

It turns out that the headline Boeing to pay $2.5 billion to settle fraud charges tied to 737 MAX crashes
is misleading, and this is actually the usual "cost-of-doing-business" settlement:

"Boeing has agreed to pay just over $2.5 billion to resolve a federal charge of “criminal misconduct” for how its employees misled regulatory officials during certification of the 737 MAX, the Department of Justice announced Thursday.

Of that amount, only $243.6 million, less than 10%, is a fine paid to the U.S. government for the criminal conduct, “which reflects a fine at the low end” of the sentencing guidelines, the court agreement states.

The rest includes an additional $500 million Boeing commits to pay in compensation to the families of the 346 people who died in two crashes of the MAX.

However, 70% of the $2.5 billion cited in the settlement, or $1.77 billion, is compensation to Boeing’s airline customers that the company has already agreed to pay. (Indeed, that’s just a fraction of what it has agreed to pay them.)"

David. said...

The Economist summarizes Boeing's predicament in Can Boeing fly without government help?. (Spoiler: No):

"Boeing got away with a slap on the wrist from regulators over lax safety practices. But the firm, which has destroyed $140bn in shareholder value over the past two years, is not in the clear. Without America’s doting government, it may never be.
...
Its gross debt has more than doubled to $63bn over the past year. ... Its free cashflow ... was minus $20bn in 2020. Operational fragility, meanwhile, extends beyond the MAX. Last quarter Boeing shipped just four 787 Dreamliners, after wrinkles were discovered on the plane’s fuselage. Compensation claims for delayed deliveries may result in a charge of up to $3bn. Boeing is taking a $6.5bn charge against another wide-body model, the 777X, delivery of which has been pushed back three years owing to uncertain demand for air travel.
...
Boeing’s capital expenditure slumped from $1.7bn in 2019 to $1.3bn in 2020. Despite pandemic-related cuts, that of Airbus is closer to $2bn"

Cutting investment won't help it compete with:

"Airbus, which bagged 1,200 more orders than Boeing in 2015-19"

David. said...

Richard Aboulafia at Aviation Week writes Opinion: Will Boeing Become The Next McDonnell Douglas?. The only scenario envisaging Boeing's survival is:

"Yet if Boeing does not launch a new product, industry competitive dynamics might still save BCA or at least prolong its decline. The slow death of Douglas Aircraft took 30 years. But Douglas faced two very aggressive competitors that were constantly investing in the future. If Airbus refrains from new clean-sheet launches and is content with a 60-65% market share, Boeing, even without a new airplane, might be able to maintain its current position for decades. This remains an industry with very high barriers to entry."

David. said...

Reuters reports that Boeing 777s grounded by airlines after FAA issues emergency order:

"Boeing 777s have been grounded in the US and Japan after the US Federal Aviation Administration issued an emergency airworthiness directive following a catastrophic engine failure on one of the planes in Denver on Saturday.

United Airlines said it was temporarily grounding all 24 of its Boeing 777s in active duty after one of its Boeing 777-200s had to make an emergency landing at the weekend, scattering engine debris across the ground.
...
Japan’s transport ministry said a JAL flight from Naha to Tokyo had to return to the airport on 4 December last year due to a malfunction in the left engine. That plane is the same age as the 26-year-old United Airlines plane involved in Saturday’s incident."

David. said...

Reuters reports that Engine parts drop from Boeing 747 cargo plane in Netherlands:

"Dutch authorities are investigating after a Boeing 747-400 cargo plane dropped engine parts shortly after takeoff from Maastricht airport.

The Longtail Aviation Flight 5504 cargo plane scattered mostly small metal parts over the southern Dutch town of Meerssen on Saturday, causing damage and injuring a woman.

The Bermuda-registered plane, which was headed from Maastricht to New York, was powered by Pratt & Whitney PW4000 engines, a smaller version of those on a United Airlines Boeing 777 involved in an incident in Colorado on Saturday."

Two incidents in one day.

David. said...

Linea Ahlgren's 222 Boeing 787s Affected By New FAA Airworthiness Directive reports on yet another Boeing safety issue:

"On Friday, February 19th, the US Federal Aviation Administration (FAA) will publish an Airworthiness Directive (AD) requiring the inspection of approximately 222 Boeing 787 Dreamliner jets. The deadline for initial checks of the jets’ decompression panels in the bilge barriers is set to 45 days from the AD’s publication."

David. said...

Dominic Gates' FAA safety engineer goes public to slam the agency’s oversight of Boeing’s 737 MAX
reports that:

"A week after the Lion Air crash on Oct. 29, 2018, Jacobsen received an email from a colleague asking if there was an issue paper on MCAS.

“This was the first day that I heard about MCAS,” he wrote. “We had no issue papers, and if we had, I would have been the engineer responsible for providing technical content and comment on such an issue paper.”

When he did get a look at the system, Jacobsen said he was “shocked to discover that the airplane was purposely designed and certified to use just one AOA (Angle of Attack) input for a flight critical function.”

If given the chance during the original certification, he’s certain that he and “6 to 8 of our most experienced engineers in the Seattle office” would have identified that as a serious design flaw because there’s “a long history of AOA sensor failures.”

Instead, Boeing minimized MCAS and kept the details of its assessment to itself."

Gates earlier reported that Boeing 737 MAX program leaders who approved flight control system say they didn’t know key details:

"In testimony to congressional investigators probing the fatal crashes of two 737 MAX jets,Michael Teal, the chief engineer on Boeing’s 737 MAX program who signed off on the jet’s technical configuration, said he was unaware of crucial technical details of the flight control system that triggered inadvertently and caused the crashes."

David. said...

Boeing tells airlines to stop flying some 737 Max planes because of an electrical issue by Niraj Chokshi starts:

"Boeing said Friday it had notified 16 customers of a potential electrical issue with its troubled 737 Max plane and recommended that they temporarily stop flying some planes.

The affected airlines should verify “that a sufficient ground path exists for a component of the electrical power system” on certain Max planes, Boeing said.
...
American Airlines, Southwest Airlines and United Airlines, the top three operators of the Max, said they had removed more than 60 planes from service. It is not clear how long the planes will be sidelined."

David. said...

In Pontifications: More of the same expected at Boeing’s annual meeting Scott Hamilton points out that no-one on Boeing's board has experience either making or using their major product, commercial aircraft.

David. said...

Reuters reports that Boeing still working on fix for 106 grounded 737 MAX planes -U.S. FAA:

"Boeing disclosed an electrical power system issue on April 7 and recommended operators temporarily remove these airplanes from service.

The problem involved the electrical grounding - or connections designed to maintain safety in the event of a surge of voltage - inside a backup power control system. The FAA said Thursday "subsequent analysis and testing showed the issue could involve additional systems."

David. said...

Bloomberg reports that Boeing faces FAA audit of latest manufacturing flaw in 737 Max:

"The flaws in the electrical components raise new questions about Boeing’s ability to monitor safety issues within the company. The lack of a robust internal safety review and oversight system was cited repeatedly by multiple reviews of the 737 Max crashes.

Until recent months, Boeing didn’t have what is known as a Safety Management System, which requires an organization to conduct more robust risk analyses of design features, open channels for employees to raise concerns and involve senior management."

David. said...

Bryan Corliss reports in Boeing rehires aircraft inspectors that:

"The Boeing Co. has quietly recalled at least some of as many as 900 quality control inspectors who were laid off in 2019 as part of a drive to adopt car-industry manufacturing processes in aerospace manufacturing.

The move comes after the union for the inspectors – Machinists District Lodge 751 – pushed the company to prove that getting rid of inspectors could be done without risking quality issues and would actually improve production times."

Who could have predicted that firing the inspectors would cause quality problems?

David. said...

Another Boeing quality problem. Reuters reports Emirates warns Boeing it will refuse 777x jets if they don't meet commitments:

"In an interview broadcast on Monday, President Tim Clark said he had not received any performance details of the jet's engines so far even though test flights began in 2020.

The influential industry veteran has raised concerns that Boeing had a recent history of over-promising on performance of new jets, including the in service 737 MAX and 787 Dreamliner."

David. said...

The drip-drip-drip continues. Gareth Corfield reports that Boeing fined $17m after fitting uncertified sensors to 737 Max and NG airliners for 4 years:

"Boeing has paid the US Federal Aviation Administration $17m after fitting hundreds of 737 NG and Max airliners with heads-up guidance system sensors (HUGSS) that hadn’t been properly certified as safe for use.
...
The affected sensors have since been formally tested and certified as safe to use. Nonetheless, Boeing will still pay $17m for fitting the sensors in violation of US airworthiness regulations."

David. said...

Marie Christine Duggan's Boeing Hijacked By Shareholders and Execs! is a detailed look at Boeing's slow corporate suicide:

"The failure of Boeing management to fund innovation and quality control has had repercussions for the tens of thousands of workers at Boeing. The company’s supply chain even reaches into my town in rural New Hampshire, where a firm that supplies parts to commercial airplanes trains high school students as machinists and pays employees to get computer science degrees. One of the reasons Boeing is “too big to fail” is that those independent, well-managed suppliers hold up what is left of the social fabric in a multitude of small towns across the country.

The mismanagement of Boeing—and the struggle of its employees to stem the decline—is not simply the story of one company gone terribly wrong. It is the story of incentives for all publicly held corporations in the United States gone horribly wrong. And it is also the story of many in lower levels of the hierarchy inside the firm putting their jobs on the line in a struggle to retain a focus on quality."

It is well worth a read.

David. said...

In addition to the 737MAX and 787 problems I've written about, I should also have mentioned the debacle of their space program, the KC-46 tanker's persistent fuel leaks, and now the long delay in certifying the 777X. Boeing's engineering culture is in ruins.

David. said...

The FAA just mandated that every single 737 be inspected:

"The Federal Aviation Administration (FAA) is instructing airlines to inspect their Boeing 737 fleet for faulty altitude pressure switches that could potentially pose a safety risk.

The switches are part of a system designed to warn flight crew of cabin depressurization. Planes are equipped with two cabin altitude pressure switches so there is a backup if one fails. But the crew and maintenance personnel are not alerted of switch failures."

David. said...

Boeings deluge of quality problems continues, as Andrew Tangel reports in Boeing faces new Dreamliner production problem:

"A new production problem has surfaced with Boeing Co.'s 787 Dreamliner, likely further delaying deliveries of the popular wide-body jets, people familiar with the matter said.

Boeing expects the newly discovered defect to take at least three weeks to address, according to some of these people. That means its customers may not get new Dreamliners for much of the traditionally busy summer travel season.

Boeing halted handing over Dreamliners to airlines in late May, after federal air-safety regulators declined to approve the plane maker’s proposed method of inspecting the jets for previously disclosed production defects. It was the second such pause in the past year."

David. said...

Dominic Gates reports that FAA demands that Boeing flight manuals give more detail on pilot emergency procedures:

"Boeing, under intensifying regulatory scrutiny after the fatal MAX crashes, has been directed by the Federal Aviation Administration to rework its flight manuals for both the 777X and MAX 10 to include detailed emergency pilot procedures.

The FAA has told Boeing to incorporate into the Airplane Flight Manuals, formal documents that are required for certification of both jets, precise details of the procedures and checklists the crew must follow to handle the kind of emergencies that killed 346 people in the MAX crashes in Indonesia and Ethiopia."

David. said...

Alan Levin reports that Boeing Employees’ Safety Independence Under Scrutiny by U.S. FAA:

"U.S. aviation regulators are opening a new review of Boeing Co. after a survey of company engineers found a sizable percentage said they couldn’t raise safety concerns without interference.

A survey conducted by the Federal Aviation Administration from May through July found that 35% of Boeing employees raised issues of conflicts of interest and a lack of independence, according to an Aug. 19 letter from the head of the agency division overseeing the company."

David. said...

Dominic Gates reports that Ethiopian Airlines settles with Boeing following 737 MAX crash and expects to fly the jet again by January:

"The financial terms of the settlement were not disclosed. The Seattle Times reported exclusively in January that Boeing had then offered an amount on the order of $500 million to $600 million, a large portion of which was not cash but concessions, including discounts on future airplane sales and waivers on maintenance costs.

A Chicago law firm advising the airline wrote a letter to CEO Tewolde then urging him to reject a settlement and instead to sue the manufacturer for punitive damages in the U.S., hoping to win “not less than $1.8 billion in cash.” That advice wasn’t taken."

But:

"This summer, the FAA slowed certification of Boeing’s next new plane, the 777X; directed Boeing to rework its flight manuals for the 777X and MAX 10 to include detailed emergency pilot procedures; and ordered Boeing to improve the independence of engineers working on airplane certification, after a third of those surveyed by the FAA said they feel they cannot raise safety concerns without interference."

And:

"In January, Boeing escaped serious consequences from a criminal investigation into the MAX crashes when the Department of Justice imposed a fine of $244 million, a relatively small amount for Boeing.

Boeing’s ongoing estimate of the total cost of the MAX crisis in financial filings has stabilized at about $21 billion, of which almost $9 billion is compensation to airline customers. Wall Street doesn’t expect that to grow."

$21B is about 2/3 of what it would have cost Boeing to deliver the first of a new airframe replacing the 737.

David. said...

Scott Hamilton's Pontifications: David Joyce fills key void on Boeing’s Board reports:

"Last week’s election of David Joyce to the Boeing Board of Directors fills a glaring hole of talent and expertise that’s been missing from the Board for years.

Joyce, an outside director, brings commercial aviation and engineering experience to a Board that has been dominated by political, defense and financial expertise."

But it has a string in the tail:

"Calhoun, a GE alumni, picked a former GE exec to succeed former EVP and CFO Greg Smith. The GE influence at Boeing that began with the 1997 acquisition of McDonnell Douglas Corp., where another GE alumni, Harry Stonecipher, was CEO, is viewed by many as a long-term contributor to Boeing’s slide from an engineering company to one overly focused on financial performance and shareholder value.

The GEntrification of Boeing has long been a concern, especially as the financial world saw the decline of GE itself as a global leader that has been in a long financial decline of its own."

David. said...

Daniel Thomas reports that Boeing: Directors to face investor lawsuit over fatal crashes:

"Boeing's board of directors must face a lawsuit from shareholders over two fatal crashes involving its 737 Max plane, a US judge has ruled.

Morgan Zurn said the first crash was a "red flag" about a key safety system on the aircraft "that the board should have heeded but instead ignored".

She said the real victims were the dead and their families but investors had also lost billions of dollars."

David. said...

Dominic Gates reports that Criminal indictment imminent for former Boeing 737 MAX chief technical pilot, report says:

"Federal prosecutors plan to criminally indict Mark Forkner, the former Boeing 737 Chief Technical Pilot who is alleged to have deceived aviation regulators and airlines about a critical new flight control system on the 737 MAX, the Wall Street Journal reported Friday.

Bringing Forkner to trial could shed more light on why the flaws in the MAX flight controls that killed 346 people in two crashes were overlooked during certification."

E-mails show he misled the FAA. Boeing's Deferred Prosecution Agreement exonerated Boeing's senior management but:

"Internal emails show Forkner felt intense pressure to comply with a “program directive” to keep down costs and protect the jet’s development schedule by ensuring that regulators and airlines perceived the MAX as so minimally changed from the previous 737 model that pilots would find little difference."

So this looks like Forkner is the scapegoat so that Boeing's disastrous management can escape accountability. I agree with Ellen Pao that the comparison between Elizabeth Holmes and, for example Adam Neumann and Travis Kalnick (and Elon Musk) shows that there is a different standard of accountability for women and men (none).

David. said...

The DoJ announced the sacrifice of Mark Forkner to avoid accountability for Boeing's top management.