Cyberspying on Supply Chains; Against Reading; Inland Navigation
Respite from the Election Dispatch
Howdy, Pilgrims,
Hard rain. I’m sitting directly under the Eastern Continental Divide, the roofline of my parent’s farmhouse. Above me, drops are being sorted between the Atlantic Ocean at Savannah and the Gulf of Mexico at New Orleans.
In this Signal, the images are once again in some dialog with the text. Well, they can hear each other in adjacent rooms. As always, it is much better to view on a decent screen.
The events of the last weeks have led political writers to abandon their boring debating points to ask whether something fundamental is happening, not just in the United States. An unusual number of interesting essays have crossed my transom, which is welcome. I know that this sounds snarky, but I mean it: lots of people are suddenly writing and thinking creatively. (Don’t fret, sarcasm to come.)
So, for this Signal I’m going to talk about U.S. Presidential politics, specifically: structural differences between the two parties, traditionally and during the last few election cycles; Roundheads and Cavaliers, i.e., preaching vs performance and different conceptions of “truth” in political discourse; nobody trusts the people; the constraints and conditions for Enlightened political discourse, and what might be possible in a mass society with a nearly universal franchise; blood and soil vs the American experiment; whether this is a Hegelian moment, in the US but also elsewhere; male and female approaches to party politics; silos and security failures (I co-wrote a book about that); and . . . just kidding. Deep breaths. You can get your fix elsewhere.
Cyberspying Part II
A few weeks ago, car dealers across the land were shut down by a ransomware attack. CDK Global, which provides operational software to car dealerships, was hacked by an outfit called Blacksuit. CDK appears to have paid a $25 million dollar ransom in Bitcoin, which is of course completely anonymous and unbreakable . . . you cannot make this stuff up. Reportedly some $1.1 billion was paid in ransoms last year. It is said that North Korea funds much of its military budget by hacking.
A few days ago, computers worldwide were shut down due to a bug in a distribution of CrowdStrike cybersecurity software. Yes, the security software makes the software less secure. Beautiful. The CrowdStrike failure does not appear to be an attack, that is, the bug appears to be accidental. Or so they would have me believe! All I know is what I read, as my Grandmama used to say, to my annoyance. If true, however, incompetent programming was innocently distributed by a careless company, as opposed to the Xz case, in which malicious programming was innocently distributed by a foolhardy company. Completely different situations. But again, multiple systems, across nominally discrete enterprises fail together because they share fundamental software, and are therefore vulnerable to failure. Much like the software in my wife’s car. Antitrust? Design flaw? Naïve? Lazy?
Which brings us to supply chain failures, and back to the Xz attack, which I began discussing with my buddy Perry Alexander some time ago. See Digital Spies, Infrastructure (In)security, and the Culture of Computing for Part I of the resulting account. When we left our story, a malicious person or persons, quite probably state sponsored, had positioned themselves within the Linux developer community, with the authority to make changes to code that would be widely distributed and incorporated. But how was this accomplished in secret, in a community said to be “open source,” that is, transparent? Perry explains, with a bit of editing, back and forth, comment, etc.
Anatomy of a Supply Chain Attack
Now that the adversary (“black hat”) is where they want to be, they can introduce arbitrary code into the Linux distribution. Sounds simple - just check in your nefarious code and watch the attack run. Not so simple. If the black hat just inserts their code into the source repo they will get caught almost immediately. Either a different maintainer or one of the apocryphal 10,000 pairs of eyes will catch it. When they do, the ID associated with the pull request will also be known. The black hat might as well put up a big, flashy, pointy arrow sign pointing to the bad code with their name on it. Somehow, the bad guy has to avoid the eyes.
The GitHub repos that contain source code are viewed by everyone. Can't hide there. So the black hat did not put their hack in the repo at all. They snuck into the Linux supply chain. They executed a supply chain attack. To understand how this was done, we need to understand how Linux is distributed.
There are any number of Linux distributions to choose from. I use Fedora. Red Hat, Debian, Ubuntu, Yocto, and Centos are all popular distributions that I’ve used. The Xz attack was orchestrated on the Debian distribution, but any distribution is vulnerable. Distributions take the Linux kernel and build all kinds of utilities and packages around it. Linux is the engine and distributions are the cars built around the Linux kernel.
Linux distributions all ultimately get their Linux kernel from the same place, as already discussed (see earlier post). But the distributions don’t check out code from repos like Linux developers. Distributions grab “tarballs,” created by the Linux kernel maintainers, that contain the current Linux source, and also contain other stuff used to support the kernel. (A “tarball” is the name given to file archives created with the Linux “tar” command. For our purposes just think of this as a collection of files.) The distributions (Debian, etc.) use these tarballs to build their specific distributions.
So, here, the black hat acquired the right to make tarballs; the black hat inserted their malicious code directly into tarballs. Distributions just grabbed premade, in this case, maliciously made, tarballs. In hindsight, this might seem foolhardy on the part of the distributions, but it makes some sense. If you have authorization to change source code, the kernel, surely you have authorization to make a tarball to run and otherwise support the source code? Specifically, instead of just grabbing a clean Linux kernel, Debian grabbed a dirty Linux tarball, containing the kernel and other goodies.
The black hat’s plan is ingenious because the bad code will never be in the git repos. No “white hat” maintaining the Linux source will ever see it. Maintainers are looking at the engine.
Worse yet, if the attack is detected, white hat developers will check out source from the repos to do their debugging. That's where developers work. They don't see the bad code either. They will see good code, and no matter how they change the code, the back door will still be installed when the tarballs are executed.
It gets worse. Let’s assume the corrupted tarballs make it into the Linux source distribution and the attack is installed. The bad code is not reflected in the Linux source repository, but nonetheless, a bad executable is running on actual computers, imported via a tarball.
Assume now that some white hat discovers anomalous behavior and tracks it to the guilty executable. The white hat needs to isolate and repeat the behavior to fix it. To do this, they restart the offending code by itself, and make observations. When this happens, they don't see a thing. The bad code isn't bad at all, and the software behaves normally.
Why don’t they see it? There are several ways that software starts on a system. One is you double click on an icon or run a command at the command line in your account. We say this is running code in user space, because the code has only the privileges of the user who starts the code. But, another way that software starts is that it is started by the operating system when the system reboots. The vast majority of software starts this way: operating software is built out from the initial boot code. This is the system space or privileged space, and this is where a black hat wants their software running.
The trick played here is the bad guy’s code only runs if it is started during system boot. If a user tries to start the bad code in isolation by making a command line call, the bad code starts the good code instead. System maintainers trying to debug the attack see the original, good version of the software running. The bad code will only start if it is started by the system during boot.
Now we have bad code running that does not appear in the source repos and cannot be debugged in isolation. The only way to see it is (i) to boot the system and (ii) know exactly where to look. This is what makes the discovery and foiling of the Xz attack so improbable, so lucky. Andreas Freund and friends (!) found the attack only because they were interested in the specific functionality leveraged by the bad code. The black hats used an application for their attack that just happened to be of interest to the white hats who noticed the code running slowly, and therefore discovered the bad code running. We'll come back to that.
A comment: what we see here is somewhat like the security problems that have emerged in cryptocurrencies. Absent quantum computing, a cryptographic distributed ledger (blockchain) makes perfect sense, is virtually unbreakable, and so forth. But it is royal hell to actually get access to the ledger, track things, and so forth. In practice, everybody uses exchanges. And exchanges get hacked all the time.
To generalize, we have highly secure, relatively mathematical cores (blockchain, Linux kernel) that are hard to use in practice. When we make them more usable (through exchanges, or tarballs, respectively) we give up, often unwittingly, security.
In a later Intermittent Signal, we will discuss the attack itself.
Against Reading
Only a fool answers his critics, that is, tries to correct a reader’s inexplicable failure to understand his writing. So, without further ado, here goes.
In Philadelphia, Writers Problems, US Presidents, French Gifts, Patriotism I wanted to suggest, substantively, that our consciousness of U.S. politics was perhaps too dark, that even in a troubled city like Philadelphia there was a lot to love. Artistically, I also wanted to put some things into conjunction with one another, aesthetically and intellectually, without being too blatant.
Judging from the responses, it did not go so well. I was, it was said, more generous with questions than answers. I teach law sometimes, and so am more than capable of telling you, gentle reader, what to think. But I don’t want to, and tried to quote LeGuin to that effect. Still, I think the piece was too diffuse, too allusive (nicer than “vague”), too piano.
So, let me clarify. We live in this vast country. I spend more time than most moving across it, literally and figuratively out among ‘em. And what I see is far different from what I read. If the medium is the message, as Marshall McLuhan famously said, how does the medium inform my view of the world represented therein?
Most of what I read – I wager most of what most Substack readers mostly read – is short form argument. Whether in what remains of legacy journalism, e.g., the New York Times, or in social media, discourse tends to be loud, dramatic, aggrieved and aggressive. Bright, but monochromatic. Writers compete for attention in a maelstrom of texts and images. Most of what I read is fairly superficial, because it cannot afford to be otherwise, much like a billboard along a highway. In either case, the message must be transmitted loudly, clearly, quickly, before the audience speeds on.
More deeply, however, writers write. And even Benjamin or Hemingway writing about Paris is not the same as actually walking on a Parisian street, feeling the air on your skin, smelling, sensing the presence of other people, and so forth.
In encountering America, and especially American politics, through short form argument, we are like people who understand restaurants by watching the Food Network on television. The T.V. shows are not wrong per se, but watching T.V. is not the same thing as eating in a restaurant. By the same token, watching different shows, or reading political opinions “from the other side,” may be salutary, but will not solve the problem. We are still just watching T.V., still just reading. Fine, as far as it goes. But no further.
It is often said, and every literary person must at some level believe, that reading somehow broadens the mind. In the United States especially, we also may believe that reading politics is incumbent on the citizen, somehow doing politics. The Federalist Papers and all that. Sure.
But reading may work in the opposite direction, too. We often have little direct experience (because we are staring at the screen!) and so tend to mistake our reading for reality, to forget that the medium, like every medium, exercises its power aslant reality. All too often, the picture of America that I and my fellow readers form in our heads simply reflects the character of our reading writing meant for the internet, both aggrieved and aggressive. Many of us readers are deeply unhappy. Reading has prejudiced us to see the world as it is in our texts, and our prejudices may not only cloud our judgment, they may sour our spirits.
Inland Navigation
The longer I spend in the West, the more I’m impressed by the temperate rain forest of Southern Appalachia, where I spent much of my youth. Yeah, banjo music, and swimming in the Chattooga. Coming up over the spine from Tennessee, the Smokies to my south and then my west, the green, the sheer density of the forest. I know it, I grew up with it, and still, it is new. Fantastic.
So much water. I left the Great Divide a week or so ago, and now am on the Eastern Divide. I crossed much of the continent, almost all drained by the Missouri/Ohio/Mississippi, the fourth longest river system in the world, and the tenth by discharge. So says Wikipedia. The first 700 miles, not quite, were pretty dry. But then I crossed the Kansas (the “Kaw”), the Missouri repeatedly, the Mississippi, and the Ohio. In Paducah, “Quilt Capital of the World,” a “UNESCO Creative City,” where the Tennessee meets the Ohio, I drank bourbon, ate well, and slept hard. Huge barges. Just upstream, the Cumberland flows into the Tennessee, and just downstream, the Ohio joins the Mississippi at Cairo. Huck Finn and Jim missed Cairo and the Ohio, the path towards Jim’s freedom, and the raft drifted into the Deep South and literary history. That Cairo. Western Kentucky has the highest concentration of major navigable rivers in the world, a sign says. Sure, whatever. I’m an internationalist and an Atlanticist, shipping ancestors, so naturally I think in terms of oceans, and all of this comes as kind of a surprise, at my age, really? The Great Lakes where I teach (20% of the worlds fresh surface water) and now these huge rivers, all inland. Somehow (how?) I’d never really thought about it . . . so much water, bounded by land, big enough for ships. Wondrous.
Safe travels, and enjoy the dog days.
— David A. Westbrook
Reading IS good, but screens... meh. Mine eyes roll out of their eye holes and across the room, where they lie, staring at me, begging to be immersed in Nature.
Thank you for brilliantly using (and trusting your readers to understand) long-form, fact based, argument (beautifully illustrated too) to make, what I perceive, two critically important points: (1) the modern world is operating at incredible speed to create, maintain and use exceedingly complex systems, and there is major risk, whether intentionally or carelessly created, of great damage to any one of those systems and those who depend on it, of any malfunction in the system; (2) the principal, perhaps ultimate, risk is contingent on whether we humans who have created, maintain, and use these systems, ostensibly for our common and individual benefit, have (a) the character required to create and maintain them for “good” purposes (I.e., whether each of us wears a black or while hat all or almost all of the time) and (b) the intellect - and willingness to use it - to discern the difference between “good” and “bad” purposes and the critical importance of maximum risk analysis, mitigation and avoidance, when creating, maintaining and using any of these complex systems. [I encourage your use of all your skills, especially the pedagogical ones, to teach potential lawyers, judges, and all others within your reach or influence, to think and act coherently, in recognition of the truth or reality of the points your long-form argument describes.] Thank you again!