Boeing, Complexity, Culture
Late Winter Dispatch
Salutations, travelers!
The last Signal, Beyond the State?, featured images from Berlin. Given the topic, where else? Too cute, maybe. I started to caption the images, something about Romanticism and Classicism, the push toward regularity and order entailed in rational modernity, the bonds of sense slipping into violence, the collective mind turning toward madness, and so forth. It’s difficult to be in Berlin without falling into a rut.
But using captions to “draw a map,” as it were, seemed a bit heavy handed. One of the things I love about images is that they are factually overdetermined (lots of details) but theoretically underdetermined. I therefore left it to the reader/viewer to indulge such musings, or not.
I was also told that the last Signal was opaque, too hard. Sorry, I guess, but some thoughts are hard, or just different, and so take work. I do try to be clear, without being too bossy. At any rate, this Signal is lighter, about plane crashes! Well, maybe “light” isn’t the right word.
These images have nothing to do with the text. I simply like photographing in snow. Moonlight: a photo essay. While the cherry blossoms are blooming in D.C., a bit early, it is still very much winter in the Colorado high country. I will have to trudge back to the house, and dig out tomorrow.
* * *
Like many people, I’ve been thinking a lot about Boeing. And, as I’ve said here before, one of the joys of my life is a range of interlocutors, people with very different backgrounds, who help me think, or at least explore. What follows is a sort of synthetic roundtable, lightly edited and with permission.
“PA” is a computer scientist; “CW” is a corporate executive and now management school teacher with a substantial part of his career in manufacturing; “KD” is a pilot. All are very experienced.
By way of introduction or reminder: on January 5, 2024, shortly after takeoff, an Alaskan Airlines Boeing 737 Max 9 was gaining altitude when a “plug” toward the rear of the plane blew out, leaving a door-sized hole in the fuselage. The plane depressurized. Fortunately, cruising altitude had not been attained so the depressurization was not as violent as it might have been, nobody was sucked out, and the plane landed without loss of life. The plug landed in a backyard, and did not kill anybody.
How does this happen? To start with the basics, an additional rear door is an option on the Max 9, and therefore, fuselages are manufactured with the opening. If the buyer does not choose the rear door, then a plug is bolted in place. The plug is not visible to the casual observer. Unless, as here, the plug is not bolted in very well and blows out, in which case the hole is more than obvious to the passengers, and for that matter, the plug was obvious to the homeowner.
This accident occurred in easy memory of two much sadder accidents. On October 28, 2018 a Lion Air Boeing 737 MAX 8 crashed, killing all 189 people on board. This is the deadliest accident in the entire long history of the 737. A few months later, on March 10, 2019, an Ethiopian Airlines Boeing 737 MAX 8 crashed, killing all 157 people on board.
Although Delaware law and corporate practice organized thereunder generally hold management harmless for failures to manage well, Boeing’s management was SO delinquent that they might be deemed disloyal, and so potentially personally liable for the company’s performance. The law, and therefore the cases, are quite complicated, but for present purposes, it is enough to say simply that the company was monumentally, shockingly, badly managed.
Since I started this little project, these two items of interest were reported: Boeing is likely to face criminal charges, and a whistleblower has committed suicide.
We’ve all had experiences with companies that run a good business, that do whatever it is they do well. Boeing, evidently, is not such a company. But why not? If you had given much of your working life to Boeing, wouldn’t you want to be proud of your work? I’m sure many of the people who work there are embarrassed. How does this happen?
I teach business law. I began asking myself, how do we teach “good business”? When things go wrong, really wrong, or there is bad intent, we might hold management responsible as a matter of corporation law. But, as I’ve already suggested, this is very rare. Similarly, in law schools, we teach “compliance” systems, basically accounting systems, especially in financial industries, but in other regulated industries, too, including aviation. We also teach transparency, especially in the context of raising money, “securities” law. We sometimes talk about “corporate social responsibility,” (CSR) or “Environmental, Social and Governance investing,” (ESG) that is, managing tensions between good citizenship and making money.
These are all significant legal endeavors. As a junior lawyer, I worked around some of these areas. Since then, I’ve published or taught some of this stuff. None of these legal regimes, though, really address the issue of doing the job well – whatever the job is, perhaps building a plane – by the terms of the business, not the terms of some regulatory regime. At most, these laws punish outright failures to comply. Maybe law cannot do more. Law school tries to teach its students hard work, quality, integrity, the kinds of things we lump together under the heading of “culture,” but that does not mean we address the culture of other enterprises.
For a bit, I thought maybe business schools were doing this, somehow. But I have it on information and belief that they don’t really teach “do your job well,” either. How would they? It’s like Spike Lee said, “do the right thing.” Yeah, right. To sound very traditional, lots of people do good work because it’s the right thing to do. And they take pride in their work.
To put the same point in terms of self-interest, one might think that if a business does not do good work, its customers will take their trade elsewhere. This last, of course, assumes the existence of competitors. But competition, in the case of aircraft and many other things, is a heroic assumption. It’s beyond the scope of today’s outing, but one of the things that happened in the US economy, even as it turned towards a market focused ideology in the age of deregulation, was the decline of competitive markets. Digital businesses, which now loom so large, seem to be especially prone to concentration. A marvelous introduction is here: Review Interview: Five Books on Market Concentration.
And, as we shall see, the question arises even when, as in the case of Boeing, there is considerable market concentration, as well as competition.
PA to DAW
Excellent article. An example of how we deal with complexity in the context of profit and regulation. From my discussion with colleagues in the aerospace industry, this is spot on.
Gregory Travis, How the Boeing 737 Max Disaster Looks to a Software Developer, IEEE Spectrum, April 18, 2019, updated February 3, 2024, in response to the Alaska Airlines incident.
* * *
Explains the aircraft design choices, and the computational choices, that led to the Max and its vulnerabilities.
[See also Erik Larson, Accidents, Abduction, and the Devil, and the linked article in The Atlantic, on how computational thinking may make a dangerous situation worse. The trick is knowing when . . . ]
DAW to PA
Fantastic, terrifying argument. But I’m not sure “regulation” is the right way to think about it. Lack of regulation, if anything. But more deeply, engineers seem to think that what they do is prior to the social. We are just making stuff, doing science, etc., and then profit motives, regulation, etc., somehow show up and screw the pooch.
This is a fundamental, flattering (to the engineers) and sometimes lethal, error. Their jobs as engineers are dependent on being hired, i.e., are always already social before they start.
To be blunt, engineers delude themselves.
It’s almost a class thing. Engineers want to have agency. Bless them.
PA to DAW
“Engineers delude themselves.” Best quote of the day. I’m going to use it in class.
Of course, we believe that we work in a technical utopia that is shattered by the social. What else should we believe? Engineering is art under the guise of some kind of economic contribution. Great engineers are constantly looking for the most elegant solution of available solutions. Aesthetics are important - just as we talked about months ago and should go back to. The importance of elegance often works in opposition to economics which is where we tend to get in trouble.
I’m ranting. Sorry...
DAW to PA
Not ranting, imho.
To be generous, it might be that to really see a problem, you have to block the other stuff out. The delusion might be sort of necessary. I once read that bloodhounds had flaps over their ears in part so they wouldn’t hear so well. And some had actually been hit by (loud) trains because so focused on tracking by scent. Whether or not true, might be kind of the point.
Rephrased, who cares what some talented 20 something nerd thinks about capitalism, or even the specifics of how he got the project? We ask him (usually) to play to his strengths, solve this discrete problem.
But that addresses the engineer. Not the problem. Military cognates abound. I spent much of today talking [. . .] about some strategy course they teach senior colonels up for promotion to general.
PA to DAW
Bloodhound is the perfect spirit animal for engineers. Single minded and focused. When P____ and I were dating I had to make sure that I ended every day with a problem I could finish. Otherwise I would not leave work, or worse yet, leave work physically but not mentally. Funny that P____ now does the same thing.
CW to DAW
Thanks for sending, and thanks for the sleepless night . . .
This is something that literally kept me up when I learned that the 737 Max was software-related (which I had, incidentally, immediately suspected).
The author is spot on about the incomprehensible disconnect between traditional redundancies – ignored! – and the takeover of software in the implementation of the most dangerous safety system ever implemented in the consumer transportation space.
This is a deep-seated problem that, funny (komisch, aber nicht lustig) enough, is difficult to solve. The allure of the software patch (while you sleep!) over hardware changes and product recalls have become mainstream in all industries, barring, perhaps, the luxury watch segment.
The reason the problem is deep seated is because as consumers, we have conditioned ourselves to beta test everything. And what should a producer do, logically, other than cater to its customers' customs?
The article is really well-written, although one qualm I have is the quick excuse the author comes up with about "money flowing the wrong way." This seems all too easily accepted as an answer. But the question behind the article’s account is how could this happen? I think not enough effort is devoted to the connection between money and (ir-)responsibility. How did THAT happen? In other words, even if the FAA runs out of money and talent to keep up with the private sector, how did the private sector’s rewards system get turned around so severely, and concurrently?
[Whether or not a focus on cost cutting/making money is an adequate explanation, Boeing has lost astonishing amounts of money, as reflected in direct settlements, stock prices, and in other, more difficult to quantify ways, including opinions like these.]
...which brings me to your call the other day. The recent failure with Alaska Airlines was, in one sense, "purely hardware" related. The work instructions literally say, "tighten the bolts to the required torque spec, Goddammit." And this wasn't done. So, like the software problems of the same OEM, of all the things to go wrong – WiFi not working, WC toilet clogged, temperature in cabin main too cold and economy rear too hot – how could a maker not tighten the bolts that could suck every soul out of the hole blown out? I can't believe I just typed that sentence. So, I guess I really, or at least mostly, agree with Gregory Travis.
The cavalier attitude that an auto manufacturer could use its customers on public roads to validate a software level has always made my skin crawl. And yet Tesla customers pay for this privilege. Alas, NHTSA finds itself in the same situation as the FAA, unable to keep up and now focusing on punitive reactions rather than industry direction, or at least collaboration.
One last question I would loft in Travis' direction: is every plane crash an existential threat to the industry? This is not true for cars, motorcycles, or guns. Maybe nuclear power plant meltdowns. (AIR Europe?) I think your question about what is business school really teaching, if not to run a noble, profitable business, is well taken. The "existential threats" in my business were, e.g., bad press (back when we had 3 or 4 buff books that told us all about cars) or shareholder value. Both aren't really true anymore – social media has diluted the legacy press, and shareholder value has become a result, not a competence.
Finally, pure embarrassment likely isn't enough anymore. For a company to change what it's doing, it needs to reexamine its own values and commit to a different, ideally higher, set of goals. And thus the conversation shifts into culture, money, politics, social values, etc...
As I say in class, "Complexity is easy. It's the simplicity that's hard." Ergo nobody, in any one of a myriad of decision-making processes, stated, simply, that the MCAS didn't have a cross-checking mechanism built into it. Because redundancy is one of the very few things I know about airplanes, I myself can't really fathom it.
DAW to CW
Hey, sleepless nights are what they pay me for. And, again, I am not an assassin.
Thanks for all the thinking. Yes, and, but . . . more anon.
Oh, and as a writer:
"Complexity is easy. It's the simplicity that's hard."
Brilliant. Really. I’ll spare you the cognate statements in the arts, at least tonight, but perfect.
PA to DAW
I use a different quote in my slides:
"It is easier to be gigantic than beautiful" - Friedrich Nietzsche
Then again, I also quote Foghorn Leghorn:
"It’s mathematics, son! You can argue with me, but you can’t argue with figures!”
I don’t know what that says about me.
DAW to KD
This article from my buddy PA (a computer scientist) is really good, both on the planes and the computers.
Over beers, you told a story about landing a plane in your skivvies and boots. I was really interested in a point you made about the military plane, the name of which I didn’t know/forgot, but which, like the 737, had been redesigned and redesigned.
This is part of – an example of – a longer effort of something I’m thinking about, which is why does technology sometimes get worse.
KD to DAW
Yes, we were discussing the way in which Boeing fancies taking analog airplanes, and scabbing them with digital “controls.” In the 737 MAX, it’s flight control software to compensate for an inherently unstable, stretched airplane. In the MD11, that I fly for FedEx, McDonnell Douglas morphed a DC10 into a jet that has had a challenging safety record in the landing phase (tail is too small). In the Air Force I flew the KC135R for 23 years. It was originally an underpowered 707-ish airplane, that burned water to augment takeoff thrust at heavy gross weights. That was KC135A – the infamous “water-wagon.”
[It is said that the plane was “built when men believed they could burn water.” This was too weird, so I had to do a little research. The plane injected water into the combustion chamber to increase density and thereby thrust. Bizarre and tricky. An Air Force historian explains here: Fairchild KC 135 Airframe.]
I started flying the “R-model” in 1998. [The KC135R] was fitted with the grossly powerful GE CFM56 engines. In the event of an outboard engine failure (4 engine jet, so 1, or 4), it had so much asymmetric thrust that the plane couldn’t be controlled without a large powered rudder, and/or reduced thrust on the opposite outboard engine. This rudder system is robust, and very reliable … until it isn’t. Google SHELL 77, and report back . . .
[“Robust” is a huge understatement. I didn’t know a plane could snap.]
In response, the AF ended up inspecting, fixing, and then writing this super complicated/confusing emergency checklist to be run in the event of “uncommanded rudder, yaw, or roll.” Many pilots became apprehensive of the airplane. “What was that?! Did you feel that!?”
By 2017, I was a very senior evaluator pilot, confident in the airplane and its systems. I was asleep in the back of a jet, heading to Al Udeid air base, from Mildenhall air base, UK. shortly after takeoff, at cruise, autopilot engaged. The airplane was exhibiting symptoms of Dutch roll. The young aircraft commander called me up front (in gym shorts, boots, and t shirt), to assess with him. He concluded this was the dreaded “uncommanded rudder, yaw, or roll,” to which I said perhaps, but let’s just shut the series yaw damper off, and see if it settles down. (Symptoms thus far were just very slow lazy 8’s about the longitudinal axis, unperceivable to a passenger.) The yaw damper was a function of the powered rudder, that dampened Dutch roll with minute rudder inputs. When he shut it off, the plane settled down. Diagnosis complete – except that the AF had scared everyone into running this absurd checklist for fear of death or disqualification. This young lad was determined that we had uncommanded rudder, yaw, or roll, which technically -- splitting hairs – we did. The pilot was insistent on declaring an emergency, and landing at Ramstein, now 80 miles off the nose. He also wanted me to take the landing due to crosswinds. Landed uneventfully in Germany, plane was impounded for 3 weeks for inspections, that determined some worn out rudder fittings.
Post “incident” my boss questioned my non-application of the choose your own adventure checklist. First step is called a “critical action” in bold face: RUDDER POWER-OFF. This would have saved the crew of SHELL 77. It would have also fixed our issue as well, but several steps later rudder power would have been restored when we isolated the problem to the yaw damper system (I knew this in my initial diagnosis). That’s called “pilot judgement/aircraft commander discretion.” The context of our discussion was AI, single pilot ops would be challenged by complex emergency procedures. My leadership supported me, once discussion was had. The checklists are made for the dumbest pilot, so the master system is isolated and then sub systems are restored, as needed . . .
Clear as mud . . .
* * *
Let me close by circling back. We tend to see development, especially “innovation,” in positive terms. Let’s build a better mousetrap. Competitive markets compel companies to offer better products. Evolve or die, the Silicon Valley mantra once had it. And it is easy to present such Darwinian, or Smithian, accounts of progress in heroic terms. Surely there is much truth here.
But not the only truth. Companies have to sell, this year. How else can you justify to the board, or to shareholders, paying for the R&D, the designers, the assembly lines?
CW told me another story, meeting with a car club in something like a VFW lodge in some snowy city, guys talking cars. It’s all going well and he is about to leave when an old codger says “you guys are not making REAL cars any more. You can’t get ____.” And, CW confesses, the codger was right. The soul of the company, its culture, centered on things like _____. But, he said, the codger hasn’t bought a car in 30 years. He just loves his old car. We have to sell to today’s consumer.
Like sharks that must move forward to survive, companies must offer product and sell it as somehow “new and improved.” (I know, a few sharks, and a few companies, can rest.) Maybe it is only put fins on the back of the car. Sometimes a product is designed to have a short shelf-life, planned obsolescence. In a digital age, one may simply add functionality (and now my car can find liquor stores while reading my texts), which often means complexity, which often compromises reliability, and lnly the manufacturer can repair . . . Software companies, of course, do all of those things, as one would expect from de facto monopolies. But sometimes something more substantive is required to stay in the game. To stay employed.
Competition turns the screws. Boeing, evidently, was terrified of the new Airbus A320 Neo. (The Neo is a sweet plane, partially built outside Hamburg, across the Elbe from villages, now suburbs, where my family has much history. The plant is still “new” to me.) It is said that the famed duopoly in big civilian planes has become more like 2/3 to 1/3 in favor of Airbus. Thus, one way to read the 737 MAX debacle is that Boeing was reacting, not acting. There was not enough time or money to design, build, and certify, a new plane, much less retrain pilots in countries and companies all over the world. Boeing strapped on bigger engines, jerry-rigged the dynamics with baling wire (software), thereby compromising the design and ultimately safety of the airframe, and sold the whole thing as a familiar plane – in the name of “improvement.” Boeing, in short, did not have – or did not act as if it had – the sort of agency we associate with design, and with engineering generally.
“Progress,” of a sort, here and elsewhere.
Time for spring skiing, and the emergence of flowers, which I also love to photograph.
Safe travels.
— David A. Westbrook
Good article! It had not occurred to me to include the latest Boeing debacle. I'm from Washington State, where we've forgiven Boeing for moving its headquarters to Chicago. Still, Boeing is a major employer in Washington and I wish they'd get their act together.
This is not my knowledge area, but I still found it compelling. Thanks!