The XZ Attack and Digital Infrastructure Insecurity
Analysis by, conversation with, and gloss on my friend computer scientist Perry Alexander
Content Warning: Intermittent Signal varies wildly. If you do not have much background in computing, this post is quite difficult, despite best efforts to be clear. The next post will be very different. So, if this isn’t your cup of tea, please be patient.
Greetings, travelers,
One of the things I think and write about, and sometimes try to suggest through photography, is “security,” in several senses of the word. Which, of course, means insecurity. Even among relatively comfortable people these days, a sense of insecurity is pervasive. I wonder why.
What does such insecurity mean for the structure of thinking, and more specifically, for the kind of work I (and many others) try to do, a critical articulation (appreciation?) of our situation? To express what we understand, at least to ourselves, and with luck others, even if we are not quite afraid? The Zeitgeist is anxious; how should we respond? Stepping back, I just realized that I’ve been trying to do this for a while, now, in books and here on Intermittent Signal. Perhaps this is an expression of my own sense of insecurity, perhaps it is merely human, but I believe the contemporary moment is nervous, anxious.
Computing must contribute to this nervous sensibility, no? If that is true, we might say that this mood is specifically modern, a function of widespread and historically recent habitation of digital environments. I don’t think this is accidental. Computers, long before we talk about AI, are spooky. See Aphorisms for Digiterati
Consider, not at all incidentally, a Turing test, military code-breaking, and homosexuality in a society that condemns it, and most importantly of all, a Turing machine, i.e., the idea of code: language is what it seems, and something quite different, at the same time. Most essentially, representation and instruction. But instruction for what? Not simple. Not, in a way, trustworthy. Surreal? And since we are all living in code, we are all living in contexts that speak to us, sometimes quite politely, and doing other things, things we generally do not know.
But there is not knowing in the sense that life, including digital life, is a mystery, and there is not knowing because you are being fooled. As intellectuals need to be reminded that sometimes a cigar is just a cigar, sometimes a spy is just that, a spy. The XZ attack on Linux is a story about one of those times, not mere spookiness but outright deceit.
The early parts of this story have been in previous Signals, and are here reproduced and modestly amended.
* * *
Preface
On March 29, 2024, Andres Freund, a Microsoft engineer, announced that Linux, the free operating system that runs untold personal, commercial, and governmental computers worldwide, had been attacked and was at imminent risk of serious compromise. Freund Announcement of XZ Attack.
In broad outline, the attack worked like this. Linux is updated regularly via distributions of software to users. A forthcoming distribution contained software that inserted a “backdoor,” which is a form of malware that negates a system’s normal authentication processes. Someone using the backdoor would have total access to the computer’s programming and data, that is, complete control. Given the critical missions run on Linux machines, this is a very scary possibility. The malware was caught, and the distribution was not made, and calamity was averted. This appears to have been the computer security equivalent of 1983’s Able Archer nuclear scare, in which the Soviets became convinced that a NATO exercise was in fact mobilization for a first strike.
My friend Perry Alexander is a computer scientist of considerable renown, who uses “formal methods” for “high assurance” systems, often in military contexts. Over the years, I have spent time with Perry and some of his colleagues, mostly talking about the meanings of what they do. I find such discursive “colleges” fascinating. In particular, there are important similarities, and of course differences, among the understandings of “security” in cybersecurity, counterterrorist, and military discourses, to say nothing of diplomacy or finance. Mark Maguire and I tried to theorize some of this thinking in Getting Through Security: Counterterrorism, Bureaucracy, and a Sense of the Modern.
The XZ attack is Perry’s nightmare. Shortly after the attack, we started talking, in person and via emails. I suggested he try to explain the situation to a non-specialist audience, which I would be happy to publicize. He agreed. What follows is a barely edited synthesis of Perry’s communications. As you’ll see, the diction provides a real window onto a world. In what follows, “you” is me, David and whoever is looking over my shoulder. A few notes are in brackets, and there are more extensive comments along the way, along with some other relevant sources. It’s a fascinating, and terrifying, story. Enjoy.
Introduction
Perry: On the weekend of 29 March 2024 an engineer from Microsoft announced that the software community had a problem. An adversary, almost certainly nation state, had run a brilliant multiphase attack. The attack combined social engineering, exploitation of supply chain vulnerabilities, and technical hacking, run over a number of years. Fake identities were established and a circle of trust infiltrated. Once trusted, the fake identities compromised a key path for delivering software to Linux distributions. The software they delivered used an innocuous utility for compressing data to insert a backdoor that allowed executing anything at any privilege level. The keys to the kingdom.
I’m going to tell you the story of this hack and how the culture I work in every day prevented it from going live. If you are a hardcore computer scientist, this story is not for you. It’s for all my friends who want some insight into cybersecurity and what the bad guys are up to. I’ve elided some details to make things simpler. Please forgive any omissions.
David: I disagree with this passage for reasons that I hope will become clear. It is, however, typical, and revealing of how this discursive community frames issues, and where its collective understanding is hazy at best. Perry knows this, hence his effort here.
Perry: Before diving into the attack, it’s important to understand what open source software is and how it works.
Open Source
If we go back in time, all software was free. You bought your hardware and the software you ran came with it, usually as source, or you wrote it yourself. Software was thought to have no value, constantly changing, and easy. Turns out we were wrong on all counts, but that’s another story. The community around the hardware you bought shared and maintained software with the vendor. But you bought hardware, not software.
David: At the outset, Perry has implicitly introduced a profound question: how do different discursive communities, e.g., computer scientists, regulators, citizens, the defense establishment, etc., conceptualize “computing” or “software” or engineering” and the like? The answers are neither obvious nor stable over time, but are critical to how we can think about digital infrastructure security.
Perry: Sometime in the ‘70s this changed, and companies started charging for compiled software systems. Companies that wrote software exclusively began to emerge and become profitable. Their software was licensed, and closed for most major commercial systems. Closed in the sense that you could not see source or modify source to fix bugs or add features. Licensed in the sense that if you shared it, you got in trouble. If the vendor goes out-of-business or drops support for your product, you’re screwed. It’s a business model and a perfectly reasonable one if you think buying a tractor whose engine can only be accessed by its manufacturer is a good idea.
David: These are, of course, fighting words in agricultural America. Perry grew up in Kansas, where he now teaches, and has family who own farms. Note how the shift to compiled software not only changes the relationship between the user and the machine, it changes who can be responsible.
Perry: Then several things happened - Unix, FSF, and Linux. Unix was developed by AT&T and you could get a copy by sending them a mag tape and $50. Unix was hardware independent and came with all the tools needed to build and install it. One could read the source. Still, it was owned and licensed by AT&T and we were governed by that license.
The Free Software Foundation (FSF) and the GNU (“GNU’s Not Unix”) project to a large extent reintroduced the idea that software should be free and treated as community property. FSF was formed by Richard Stallman who worked as a student researcher for Gerry Sussman at MIT. Both continue to provide leadership for FSF to this day although Stallman stepped away from the presidency of FSF in 2019. (You may remember Gerry from NDIST.)
As an aside, GNU’s Not Unix is a recursive acronym, which is just really clever for programming, which relies on recursion . . .
As community property, this software is maintained by what you called collections of “volunteers.” I have never thought of us as volunteers. We use software and as we improve upon it, we make our improvements freely available to the community. We are allowed to say how the software we make can be used, but it is open and free.
David: This is really interesting. For scientists, colleagues, actors engaged in an important, indeed historically significant collective project, the notion of “volunteer,” implying charity, gift, need to be helped, comes across a little odd, maybe even offensive. At the same time, the techie Weltanschauung, well, maybe it needs a little provocation. Just saying.
Perry: What GNU is for compilers, editors and debuggers, Linux is for operating systems. It is a cleanroom implementation of Unix originally developed by Linus Torvalds. Linux is free and open. I used to jokingly refer to Linux as software developed by hippies for free. Usually in reference to Linux being better than certain commercial operating systems developed by certain unnamed mega corporations. Maybe I shouldn’t tell you this, but MacOS is built on Darwin which is in turn built upon FreeBSD and Mach. Both FreeBSD and Mach are open source projects maintained by their communities. Darwin is now a full-fledged open-source UNIX maintained by Apple.
David: Remember, this is the version for “friends who are not computer scientists.” We are all prisoners of our worlds! “Cleanroom” refers to a process of designing functionally similar but not IP infringing software. It is, in a technical sense, not copying.
Perry: Linux and FSF’s GNU project revolutionized software development forever by making software free again. I can confidently say that every computing system you interact with is derived in part from open source software. No kidding. Every system. Open source is everywhere and is critical. We could not build or maintain the software infrastructure we have without sharing. [Emphasis supplied.]
A colleague who has 30+ years of experience developing secure software told me explicitly that had the XZ attack target been a closed source system we would have never found it. Why? Because the developer who found the bug could look at the source when the system behaved unexpectedly. Closed source you live with what you get and the developer could not see either the development process or the source code. The open source adage is 10,000 pairs of eyes look at source. (This turns out to be false, but that’s a different thread.)
Okay, but the developers behind all those 10,000 eyes can change the source anonymously. This, you say, is absurd. It would be if that were what actually happens.
David: In an earlier version of this conversation I had said that we did not know who the people behind the screenname were. So the attack was anonymous. As Perry says, “almost certainly a nation state.” I think the point to some significant degree stands.
More generally, Perry is defending the procedures and indeed culture of the open source “community” (insofar as that word makes sense in a world of digital anonymity), largely because he’s been a part of it, or adjacent to it, for decades. So ignorant criticism from an unwashed barbarian like me stings, not least because it is of course wrong. At the same time, the process is deeply wrong, as this entire piece shows. So, Perry’s position is deeply conflicted, which I of course appreciate since that’s how I feel about so many things! For now, Perry defends the process of open source modification as follows.
Perry: To understand what happens you need to understand git repos. Public domain software is stored in repositories (“repos”) that are maintained by adding patches that represent code changes. If I want to make a change to a package I have git [a system that tracks changes in software, also developed by Torvalds] create a patch and submit that patch to the repo as a “pull request.” [You are requesting the maintainer to “pull” the suggested code from your version of the software (“fork”) and add it to the repository. Like accepting a suggested edit.] Each pull request is evaluated by a system maintainer or owner, who determines if the change should be added. Nothing is added to the repo without review by at least one maintainer. Even maintainers themselves must have changes they develop reviewed.
David: Developers may suggest changes to the code. Such changes have to be accepted by “maintainers,” who add, or do not add, the changes to the official version of the code.
Perry: The repo itself becomes a history of additions to the repo. When the repo is updated, the patch is added to the existing contents. So, a repo is not just source code, it is a collection of patches applied in sequence to the initial commit. The repo becomes a ledger that tracks all changes. Plus, each change is signed by the submitter. Given a repo you know what changes were made and who made them. Just like a bank ledger. Any change can be undone, parallel branches can be created and merged, and the entire process is auditable.
Trust
Now that we know how repos work and how they are maintained, we need to deal with trust. How do you trust maintainers and owners? How do you identify submitters of patches?
What is trust? Strong identity, observed good behavior, indirect good behavior observed by a trusted third party. Strong IDs are identifiers bound uniquely to an entity. On most sites, IDs are created and associated with passwords. Logging in using your password verifies your right to use the associated ID. It does not reveal who you are, just that you own the ID. Maintaining a secret password is a big deal. If someone has your password, they can be you. This is why multi-factor authentication has become a big deal. Also, why sharing a password in my shop will get you walked immediately. No discussion.
Anyone can create one or many ids and passwords and participate in the process of maintaining software. There are two kinds of people we might need to trust - patch submitters and system maintainers.
Let’s discharge the issue of submitters first. We don’t care who they are. They submit a patch and the maintainers review the patch. Maintainers do not need to trust any submitter because they always verify the patch is good before adding it. [David: !!!] Submitters have no privileges for writing to the repo, so all they can do is ask. If a maintainer can’t establish a patch is good, they never add it. If a submitter tries to do something nefarious, the maintainer can block or ignore them in the future. Regardless, there is no need to trust because the maintainer is evaluating the submission and the submitter can only ask.
What about maintainers? If they are adding to the repository, they must be trusted and authenticated. More specifically, their identities need to be trusted. There are two common ways of establishing trust in an identity. The first is building a chain of trust from a root id that is trusted. Whenever you use a notary public, you are using this mechanism. The notary establishes their trust with the government using physical identification. They are issued a certificate that memorializes that trust and we trust the notary because we trust the certifier. The notary extends that trust to you by observing your signature in the presence of your ID. They add a stamp that certifies your signature can be trusted because they are trusted by the government. Now extend that. Let’s say the notary certifies that you are trustworthy and gives you a certificate that allows you to be a notary. Do this over and over and you end up with a “tree” of trust from the government “root.”
There are good things about this tree model. The biggest of which is one can trace trust in an identifier back to a root they trust. Maybe the government, maybe your company, or university. You can always trace your trust in an identifier back to this authority you have decided to trust.
There are also bad things about the tree. If a bad actor is certified or a bad actor compromises the certification process, everything under their node in the trust tree is now untrusted. Furthermore, the certifying individual completely controls who they choose to certify. If the government decides you are a subversive, they don’t certify your identity. We see this with passports in nefarious countries all the time. Try functioning without WeChat in China.
The set of maintainers uses a less formal “web of trust.” You use this model all the time, but likely don’t realize it. In a web of trust there are no roots. You decide who you directly trust. You also record that trust and announce it to everyone else. Because trust is transitive, you trust also IDs trusted by those you trust. That announcement adds to the IDs reputation. The more IDs that trust you, the greater your reputation. In a formally implemented web of trust you indicate trust in an ID by signing it with your private key. That signature can be checked to make sure you really do trust it. (In the old days we used to have key signing parties where everyone on a project would get together, verify identities visually, and sign keys.)
When an ID is compromised in a web, the reputation gained from that ID is deleted. If lots of other IDs trust you, the deletion of one link does very little. If not many IDs trust you, your trust could be heavily compromised or even be eliminated. Regardless, there is no central authority establishing trust. It is the community.
The web of trust among maintainers is not constructed formally by signing keys. It’s far more informal. Just like joining a club or getting a security clearance or granting tenure, those that want to trust you check that you have lots of links from diverse places. Then they indicate their trust to the larger community. [Emphasis supplied]
David: Fantastic passage. Thinking about trust is much of what Perry does (“high assurance”). Here he distills and conveys a great deal of thought in a disarmingly simple manner. Indeed, as I sometimes warn my students, Perry, like many very smart people, often adopts an aw shucks delivery that is not to be trusted! Actually, I completely trust Perry, but after years of talking, I’ve learned that much of what he says must be treated with some care. You may not understand yet.
The Attack
The social part of the XZ attack was successfully launched against the web of XZ maintainers and was classic espionage tradecraft. Nothing particularly novel, other than the target. “Jia Tan” managed to convince the XZ maintenance community to trust them, when Jia Tan should not be trusted.
The perpetrator of the attack appears to have established multiple IDs on GitHub. One of those IDs is Jia Tan. They proceeded to make legitimate pull requests to maintainers using the Jia Tan ID over a period of years. They were establishing a basic presence in the community. At some point they targeted XZ. Likely because it had a small maintainer group - a single person who was dealing with medical issues. They then flooded the XZ maintainer with pull requests from many IDs.
David: Note that we do not know who Jia Tan is, or even if it is a single person, probably not, especially as the number of IDs in the attack multiply. Also, in an earlier draft, ID was written “id,” which seems somehow perfect.
Perry: Now the XZ maintainer can’t keep up. The attacker who has been legitimately contributing patches offers to help. The attackers then start badgering the maintainer and the maintainer’s colleagues. Jia Tan continues to offer help. The maintainer’s reputation is at stake. [Emphasis supplied. For some of the dialog, see The XZ Attack and Timeline.]
Finally, the attackers win. The barrage has hurt the reputation of the maintainer who wants to retain his status. So, the maintainer gives in and grants access to the attacker, Jia Tan. Now the attacker is where they want to be.
This is classic espionage. The attacker threatened something valuable to the maintainer. The attacker appeared trustworthy, based on earlier behavior. The maintainer, the target moves to protect the valuable thing. Boom. In human systems the valuable thing could be anything - family, wealth, status. Here the valuable thing was status, the maintainer’s reputation in an online community.
Do not underestimate the difficulty of pulling off this attack. It took many years. It took knowledge of the Linux maintenance mechanism. It took an understanding of human behavior and spy tradecraft. It is very likely several people were involved, likely a nation state.
David: I have comments about virtually every sentence of the foregoing, but I’ve been thinking about security in various contexts for a long time. Here I’ll restrict myself to the following.
It is difficult to believe that this system can continue in its present configuration. If we do not care who the submitters are, then the maintainers are a single line or plane of defense. If there is only one maintainer, then we have a single point of failure. Which, here, failed. It’s a bit professionally, shall we say convenient, to frame this as a success by Freund and the open source community. That little Dutch boy saved the dike, too.
Second, the relationships among transparency, secrecy and certification as sources of trust, even within this text, are endlessly fascinating. In conversation with Perry, I spoke about “volunteers,” i.e., there is no licensing process for submitters. We do not know who the submitter was, or, evidently more likely, who was on the team represented by “Jia Tan” and other ids. As a law professor, in the business of licensing, to say nothing of a car driver, I find this bizarre. Perry’s account adds nuance, and tells us a great deal about how we got to this point, but it hardly convinces me that all is well with that world, that is, the control systems that sustain contemporary life.
Third and most importantly, as with oral history generally, Perry’s diction is significant, points to a way of looking at the world. Roughly understanding, analyzing, and gently critiquing such worlds is much of what I do. More generally, in a society constructed, and here barely protected, by experts, with their own blind spots and limitations, amateur conversation, with members of the public, is critical. Contra my friend, public accounts of what they do are exactly what powerful experts (“hard core computer scientist”), in the military and elsewhere, should be doing. And I thank Perry for doing so. Not so incidentally, this is a central theme in my book, Social Thought From the Ruins: Quixote’s Dinner Party.
To recap: Perry has discussed the structure of open source software and how it is maintained and updated, and he has discussed how a “social engineering” attack unfolded at a particular point, the maintenance of an specific, widely used, utility. The good folks at XKCD have captured this pretty well.
XKCD on Infrastructure Vulnerability
Anatomy of a Supply Chain Attack
Perry: Now that the adversary [the attacker = “the adversary,” note military diction] is where they want to be, they can introduce arbitrary code into the Linux distribution. So the attack sounds simple – the adversary checks in their nefarious code, and watches the attack run. Not so simple. If the adversary just inserts their code into the source repo, the bad code will get caught almost immediately. Either a different maintainer or one of the apocryphal 10,000 pairs of eyes will catch it. When they do, the ID associated with the pull request, Jia Tan in this case, will also be known. The adversary might as well put up a big, flashy, pointy arrow sign pointing to the bad code with their name on it. Somehow, the adversary has to avoid the eyes.
The GitHub repos that contain source code are viewed by everyone. Can’t hide there. So, the adversary did not put their hack in the repo at all. They snuck into the Linux supply chain. They executed a supply chain attack. To understand how this was done, we need to understand how Linux is distributed.
There are any number of Linux distributions to choose from. I use Fedora. Red Hat, Debian, Ubuntu, Yocto, and Centos are all popular distributions that I’ve used. The XZ attack was orchestrated on the Debian distribution, but any distribution is vulnerable. Distributions take the Linux kernel and build all kinds of utilities and packages around it. Linux is the engine and distributions are the cars built around the Linux kernel.
Linux distributions all ultimately get their Linux kernel from the same place, as already discussed. But the distributions don’t check out code from repos like Linux developers. Distributions grab “tarballs,” created by the Linux kernel maintainers, that contain the current Linux source, and also contain other stuff used to support the kernel. (A “tarball” is the name given to file archives created with the Linux “tar” command. For our purposes just think of this as a collection of files.) The distributions (Debian, etc.) use these tarballs to build their specific distributions.
So, here, the adversary acquired the right to make tarballs; the black hat inserted their malicious code directly into tarballs. Distributions just grabbed premade, in this case, maliciously made, tarballs. In hindsight, this might seem foolhardy on the part of the distributions, but it makes some sense. If you have authorization to change source code, the kernel, surely you have authorization to make a tarball to run and otherwise support the source code? Specifically, instead of just grabbing a clean Linux kernel, Debian grabbed a dirty Linux tarball, containing the kernel and other goodies.
The adversary’s plan is ingenious because the bad code will never be in the git repos. No “white hat” maintaining the Linux source will ever see it. Maintainers are looking at the engine.
Worse yet, if the attack is detected, white hat developers will check out source from the repos to do their debugging. That’s where developers work. They don’t see the bad code either. They will see good code, and no matter how they change the code, the back door will still be installed when the tarballs are executed.
David: This is fantastic. The system is designed to protect source, but applications are source plus the other “goodies” that “support” (or infect) the computer in operation. So the goodies are vulnerable. Hah! Take the apple, Snow White, and behind that, the Garden, of course.
Perry: It gets worse. Let’s assume the corrupted tarballs make it into the Linux source distribution and the attack is installed. The bad code is not reflected in the Linux source repository, but nonetheless, a bad executable is running on actual computers, imported via a tarball.
Assume now that some white hat discovers anomalous behavior and tracks it to the guilty executable. The white hat needs to isolate and repeat the behavior to fix it. To do this, they restart the offending code by itself, and make observations. When this happens, they don’t see a thing. The bad code isn’t bad at all, and the software behaves normally.
David: So that’s how the attack was structured, a vulnerability and an explanation of how that vulnerability could be exploited. But how was the attack carried out? Through the internet!
Executing the attack
Perry: Let’s review where we are. Jia Tan has injected themselves into the software build process by becoming a maintainer for XZ. Knowing they cannot hide easily in the Linux repos, they decide to hide in libraries, and the build and install system around the kernel, using delivery tarballs.
Now let’s turn to what Jia Tan hides in the tarballs and how is it used, specifically, the backdoor code or “kit.” [The attacker has access to a place to hide a “bomb,” so let’s examine the bomb.] First, I’ll talk about the backdoor and how it uses Linx infrastructure to receive and execute nefarious code without detection. Then, I’ll describe how Jia Tan gets the back door from the tarball into the operating system.
Ever wonder how the internet works? How we get messages from one place to the other? Let’s say I want to send you email. Somehow my mail server needs to talk to your mail server in order to move the message. Your server needs to somehow know where mine is and my server must find yours. We do this with something called a port. Your email server sends a message to a port where my email server is listening. There is a protocol behind the scenes that sets things up, but in essence your email server answers the door for messages that arrive on the email port. Your web server answers the door for messages that arrive on the HTTPs port. SSH on the SSH port and so on and so forth. It all works when everyone agrees which port is the email port, the HTTP port, the SSH port and so forth.
As you might guess, it’s not quite that simple. What about messages that ask a server to do something wrong or exploit a bug in their associated server? This happens more than you think, sometimes by accident and sometimes by intent. Worse yet, what if a bad guy can replace your message handler with one that does what they want. Remember all the work Jia Tan did to get in a place where they could control code distribution? This is why. They are going to replace a legitimate message handler with a backdoor hidden in the tarball. While there are a number of mechanisms in place to prevent these kinds of things, an adversary like Jia Tan will sometimes find a way around them. It’s complicated and complicated benefits the attacker.
The data packets that arrive for processing by your servers are datagrams containing an address and a payload. The address identifies the port that should process the packet and the payload is data to be processed. In a sense, the address of the message recipient and the content of the email. The adversary is going to put their exploit code in the packet payload. Pretty simple, except payloads are not executed. They simply arrive and are processed.
Remember, what the bad guy ultimately wants to do is run code at elevated privilege on your computer. Do whatever they want and then hide their tracks. As it turns out, these programs that process messages from ports run at elevated privilege. They have to in order to do their jobs. There are a number of mechanisms in place to prevent misuse, but among the biggest is they do not ever execute code. Message payloads are sent where they need to go, but they do not execute.
But Jia Tan is going to get a message processing program to execute their code at elevated privilege. The message processing program will still do what it’s supposed to, but the malicious tarball will add a special feature that identifies adversary code in payloads and executes it.
One last nifty trick. The adversary is going to embed a public key in their new message processing program and keep the associated private key. They will encrypt messages with their private key, send them to the message processing program that will decrypt them. The new processing program will only process messages it can decrypt. The owner of the private key is the only source for encrypted messages that can be decrypted by the public key. This is ingenious. Only the adversary who possess the private key can send attacks for execution and those attacks are encrypted.
The adversary writes a new message processing backdoor that watches for messages whose payload is encrypted with their private key. When that code arrives, it decrypts the message and executes the code. At elevated privilege. The hacker’s holy grail.
But now they need to put their elevated social status to work to get their code in the Linux distribution. To do this, they will take advantage of how code is tested and built prior to release.
To install the bad executable the adversary must hide the code from maintainers. The delivery tarball is easily inspected and read, thus the attacker must obscure their backdoor code in the tarball to hide it. To do this they created a binary test file that contains the bad code. Specifically, they hide the bad code by including it in its compressed form rather than readable text. For compression libraries like XZ, it is common to include compressed files in the test suite to test decompression. The bad guy simply compresses their bad code and includes the compressed file in the test suite. No one thinks twice about yet another compressed test file, thus the new compressed test file sits in the tarball, appearing completely innocent.
During the testing process the rogue binary is decompressed and installed. To do this build scripts are modified to extract the bad code and inject it into the delivered system. The build is performed using a common system called GNU Autotools (configure.ac and m4) whose purpose is executing build scripts that construct sophisticated systems like Linux. Another bit of social engineering plays out here. The configure.ac file is read by autoconf, a program that generates a configure shell script. Most engineers look only at configure.ac because the configure shell script may be thousands of lines long. The attackers knew this and modify the configure shell script itself rather than configure.ac.
[A]utoconf uses another tool, m4, to expand macros during generation of the configure script. [C]onfigure.ac uses these macros and m4 translates them. An m4 directory often exists that contains custom macro translation scripts for a particular project. This directory is so common and expected that no one bothers checking them, so custom scripts from the adversary will not attract much attention if any at all.
Bad code is now installed, but there is still one more safety feature to defeat. When GNU Autotools builds the deliverable system, it checks for sandboxing support. Sandboxing is a way of putting code in a controlled environment that restricts the operations it can perform. In this case, the sandbox is the last line of defense and should be enabled for any code build. If Jia Tan’s code must run in a sandbox it will not be able to perform many operations requested of it because the sandbox blocks them. The sandbox must be removed.
How Jia Tan defeats the sandbox is ingenious. A single period is included in the GNU Autotools build script that causes a check for the sandbox to always fail. That failure is not a failure that causes the build to crash, but a failure that causes the build to believe the sandbox code is simply not available. Thus, the build continues without the sandbox. Now when the backdoor executes, it does so at elevated privilege, and with no sandbox restrictions.
When executed, the backdoor installation works like this. A malicious m4 macro that looks legitimate at first glance is added to the m4 directory. This macro decodes the binary test file containing be backdoor payload. Adding one more test file is innocuous, so no one pays much attention. Now the build script causes the sandbox that is our last line of defense to fail silently, allowing the build to proceed unprotected. The m4 file that causes all this to happen exists in the delivery tarball, not the git repo. No one inspecting the git repos as a normal part of development would ever know the difference. (Remember that Jia Tan controls the building of the tarballs, due to their social attack on the legitimate maintainers.)
The compromise is serious. Once installed, Jia Tan can send a packet with encrypted attack code as its payload. The attackers illegitimate message processing system will decrypt the attack code. The backdoor will execute the attack at elevated privilege with no sandbox. Jia Tan has found the Holy Grail of system compromise – they can run any code they want at any privilege they choose. By extension, they can install themselves and live on the target system undetected indefinitely.
Jia Tan can even reinstall the legitimate XZ system after they have established themselves on the target, and no one will be the wiser. This is devastating and had it succeeded we would never know.
Postmortem
How did Andres Freund discover this carefully implemented attack? By accident. Two things raised suspicions that caused Andres to look for a problem. One was a timing issue. Andres is a PostgreSQL database programmer who cares deeply about runtime. The PostgreSQL code they write must execute efficiently because of how many times and where it runs. What got noticed when Jia Tan got their code installed was a slight slowdown. Very slight. Only someone with Andres’ attention to detail would notice. I certainly would not have. When their code started running a bit slower, antennas twitched and the game was on.
Unwinding this attack was no small feat requiring reverse engineering and lots of detective work. [So why didn’t all hell break loose if the code was installed?] Thankfully, Andres was working on a pre-release beta, and the attack was not delivered beyond the beta users.
One thing I always say in my public presentations on cybersecurity is to be cautious of anyone who tries to scare you. Particularly when they offer you a solution for your fears. I’m going to make an exception here and ask you to be scared precisely because I don’t have a viable solution. This attack was discovered because of one individual’s attention to detail and perfectionist obsessions. Nothing systematic. Nothing planned. Just one programmer behaving like, well, a great programmer.
David: In a controlled environment, beta testing by a relatively small number of users.
Perry: This combination of social and technical attack techniques over an extended period of time is highly sophisticated. This attack was coordinated and took years to unfold. Almost certainly executed by a state actor. Regardless, the attack represents an escalation in adversary capabilities like none I’ve seen discussed publicly. Clearly our open source software supply chain needs improvement, but this is far easier said than done. Here, the checks and balances actually worked, but barely.
If Jia Tan was successful, we would never know the attack happened. If the backdoor were distributed, Jia Tan’s friends would be able to execute whatever code they wanted to, at high privilege and to whatever effect they choose. Then they could clean up. The adversary gets in, installs their kit, and puts everything back the way it should be. Except, of course, the adversary is present on your system, and cannot be detected. It is not an exaggeration to say that every program that runs on every device you own depends on open source software. It’s not going away, the supply chain is not easily changed, and our society depends on it.
David: Not like we need more to worry about, so thanks, Perry.
* * *
In the year or so since Perry has been teaching me the XZ exploit, much more has happened. The situation seems to be getting worse. A great deal of money, both through ransoms and operational losses and efforts at defense, is lost through a rising wave of cyberattacks. Much of this goes unreported, and so is difficult to quantify. And while many hacks are motivated by simple greed, greater dangers loom. As noted, the XZ hack was almost certainly sponsored by a nation state. The software associated with a fair amount of material infrastructure, e.g., water treatment plants, has been found to be infected. Governments, including the US government, have been infected. Just this March, Claude Code got leaked. Engineer's Codex analyzes in substantial detail, and adds:
The [Claude] leak did not happen in isolation. It landed one of the worst months in AI developer security on record.
In the same 30-day window:
· Axios (100M weekly npm downloads): maintainer account hijacked, a remote access trojan deployed across macOS, Windows, and Linux. The malware self-destructs after execution and deletes itself from node_modules. Google suspects its from North Korean bad actors.
· LiteLLM (97M monthly PyPI installs): backdoored with a three-stage attack: credential harvester sweeping SSH keys, AWS/GCP/Azure credentials, Kubernetes configs, crypto wallets, and every LLM API key; then Kubernetes lateral movement; then a persistent systemd backdoor. Live for three hours before PyPI quarantined it
· Railway (2M users, 31% of Fortune 500 as customers): CDN misconfiguration leaked authenticated user data to wrong users for 52 minutes
· Delve: YC-backed compliance startup allegedly generating fraudulent SOC 2 audit reports with identical boilerplate across 494 reports, fabricated board meeting evidence, and audit conclusions written before any client submitted evidence
· Mercor AI: alleged LAPSUS$ breach of 939GB source code and 4TB of total data via their TailScale VPN
· OpenAI Codex: command injection via branch names discovered December 2025, patched February 2026, disclosed March 2026. Attackers could steal GitHub auth tokens via unsanitized branch name parameters
· GitHub Copilot: injected promotional ads into 1.5M+ pull requests as hidden HTML comments without developer consent. GitHub VP confirmed it and said it was “the wrong judgement call”
The Claude Code leak is, in some ways, the least technically dangerous item on that list.
Engineer’s Codex: Diving into Claude Code’s Source Leak
* * *
There is much, much more to say, of course, and not only at a technical level. This Signal is already the longest ever, so I’ll conclude with a few brief observations mostly gleaned from Perry and his colleagues, perhaps to be developed in due course.
It’s remarkable that software is so unregulated. Computer “engineers,” unlike all other “engineers,” are not licensed. I’m not sure exactly why, but it must have something to do with our deep and unspoken societal assumptions, the imaginary, about what computing is, as an activity. For many of us, computing is still “just math,” done in boxes, often by boys drinking diet coke and eating bad pizza in a basement. But Moore’s law and all that, and we have bolted computers onto everything.
Software has a very basic test, often overwhelmingly compelling from a commercial standpoint. Does it run? If so, then the buyer is likely to say that’s good enough. Making software safe is an entirely different question. In fact, the easiest way to get workable software is to cut and paste, and now use generative AI (misnomer) to cut and paste. Relevantly, Joshua Saxe argues that it looks like AI tends to advantage attackers over defenders. Saxe on Dual Use Gen AI Advantages Attackers. Be that as it may, cheap and unsafe software propagates. Except for obviously mission critical systems, and not always even then, developers do not take (are not paid to take) the time and expense required to build safer software. But there are lots of better practices, some mentioned in the Saxe piece. Replacing old but functional software with new, more robust, software, preparing for an attack that may never come, is hard to fund, absent disaster. This is an old theme in security.
Systems probably cannot be made completely safe, not least due to the humans, but that’s easy to understand in principle. In contrast, one of the things that has been hardest for me to really learn, grok, is how opaque computing systems generally are. Malicious software is “undetectable,” or systems or conclusions are “not auditable” all the time. Again, my imaginary is off: I still subconsciously think of computing as done on clean white boards. Worse still, I don’t think there’s any principled way to understand “bug” ex ante. (Much like “weed.”) Worse still, the layered structure of computing (hence “substack”) means that the machine must move up and down layers of abstraction, losing and gaining specification, and hence allowing for “bugs” in principle. (I think.)
Perhaps on another outing I’ll try to tie together this anxiety that I now see runs like a subterranean river through my work over the last half dozen years at least. Much of it has to do with computing, our age would not be what it is without digitization, but it’s not all hardware, software, and the rag tag gang of military interests and oligarchs that work to make our history, such as it is. Maybe. For now,
Safe travels and no worries,
—David A. Westbrook