4 What We Almost Lost
The responders never quite stopped to ask the question aloud, because they were too busy answering it: what, exactly, had almost happened? Strip away the panic and it has a clean shape. Suppose the half-second had gone unremarked. Suppose Andres Freund had been traveling that week, or had written the latency off as a bad benchmark, or had simply never run the test, and the two poisoned releases of xz, versions 5.6.0 and 5.6.1, had ridden the ordinary update machinery out of the fast-moving development channels where they had surfaced and into the stable distributions that most of the world actually runs. What would the world have woken up to?
It helps to begin with what the compromised software is, because the answer is the first measure of the stakes. xz is a compression utility, one of the small, unglamorous programs that shrink files for storage and transport and expand them again on the other end; liblzma is the library inside it that does the actual work, a piece of code that other programs call without ever thinking about it. It is precisely the kind of component that is everywhere and noticed nowhere. Nadia Eghbal, in the most careful account of the labor beneath digital infrastructure, drew the distinction that makes the danger legible: “we directly rely on open source code to keep our phones, laptops, cars, banks, and hospitals running smoothly,” she observed, and “if an open source project goes down, it can literally break the internet” (Eghbal 2020, 14). The phrase is bounded, not breathless. But it names the right order of magnitude. The thing with a backdoor in it was not an app. It was part of the floor.
The contemporaneous record reached, almost at once, for the language of the near miss. Akamai, summarizing the incident days later, allowed itself one controlled flourish: “This backdoor almost became one of the most significant intrusion enablers ever — one that would’ve dwarfed the SolarWinds backdoor. The attackers were almost able to gain immediate access to any Linux machine running an infected distro, which includes Fedora, Ubuntu, and Debian. Almost” (Akamai Security Intelligence Group 2024). The triple “almost” is doing real work. The word can be read two ways. It can mean the catastrophe did not happen, so the system held. Or it can mean the catastrophe was averted, not absent: that it was fully built, fully capable, and stopped by something outside the design rather than by the design itself. The second reading is the one the record supports. Freund himself supplied the first caution, in the disclosure email: “Luckily xz 5.6.0 and 5.6.1 have not yet widely been integrated by linux distributions, and where they have, mostly in pre-release versions” (Freund 2024). Close, then, but not yet everywhere. The task is to say precisely what the thing could have done, and to whom, without letting “what could have happened” inflate into a claim the evidence does not carry.
Start with what the backdoor did, because the first public account of it was wrong in an instructive way. The earliest widely read disclosure-day report, in the open-source news outlet LWN, framed the threat as a login problem: “It appears that the malicious code may be aimed at allowing SSH authentication to be bypassed” (Corbet 2024). Authentication is the step where a server checks that a visitor is who they claim to be, and an authentication bypass means slipping past that check, logging in without the right credentials. That was bad, but it was not the whole of it, and within days the reverse engineers had corrected the frame. As Akamai put it, “It was originally reported as an SSH authentication bypass backdoor, but further analysis indicates that the backdoor actually enables remote code execution (RCE)” (Akamai Security Intelligence Group 2024). Remote code execution is the more serious capability, and the one worth defining carefully: it means the ability to make a machine run commands of the attacker’s choosing, issued from somewhere else on the network, with the privileges of the vulnerable process. Here, as the demonstration below shows, that meant root. Not “log in as someone.” Run anything.
The cryptographer Filippo Valsorda, narrating the live teardown as it unfolded, drew the line in a single terse sentence: “It’s RCE, not auth bypass, and gated/unreplayable” (Valsorda 2024, post 3kowjkx2njy2b). Russ Cox, whose timeline became one of the standard references, stated the same capability in a sentence a general reader can hold: the backdoor “watches for the attacker sending hidden commands at the start of an SSH session, giving the attacker the ability to run an arbitrary command on the target system without logging in: unauthenticated, targeted remote code execution” (Cox 2024). Three properties are folded into that sentence, and each matters. The commands arrive at the start of a session, before any login: pre-authentication, in the jargon, meaning the server runs the attacker’s instructions before it has decided whether the visitor is allowed in at all. They are targeted, aimed by the operator rather than sprayed. And they execute arbitrary code, anything the operator chooses to send.
In a later reconstruction for a DEF CON audience, the Microsoft security researcher Thomas Roccia inventoried the capability as three commands the hidden code would honor. The first was, in his words, “an SSH authentication bypass using root login,” the power to log in as root, the all-powerful administrative account, with no valid password; the second “a remote code execution,” the power to run an arbitrary command; the third simply closed “the pre-authentication [connection] that [was] previously open[ed],” tidying up after itself (Roccia 2024, 29:22). The machinery behind that second command was a single, ordinary function in the standard C library, system(), which takes a line of text and hands it to the shell, the part of the system that runs commands. The route was system(), the library call, a detail worth fixing because it is often confused with systemd, the unrelated service-management program through which the backdoor reached the login server in the first place. The trigger had its own elaborate machinery: cryptography, test files, and concealment. For the counterfactual, what matters is the plain fact of the capability. A reachable, backdoored server could be made to run whatever its attacker wanted.
That this was a finished weapon and not a latent flaw is not a matter of inference. A security researcher, Anthony Weems, built and published a working trigger, described in its own documentation as a “cli to trigger the RCE assuming knowledge of the ED448 private key” (Weems 2024). On a vulnerable server he set a watchpoint, connected, and had the backdoor run the harmless command id, which reports the identity of the account under which a command is running. The answer came back “uid=0(root) gid=0(root) groups=0(root)” (Weems 2024). Those eleven characters, uid=0(root), are the stakes in miniature: an arbitrary command, sent across the network to a server that had authenticated no one, executing with the highest privilege the machine has. Ars Technica’s Dan Goodin gave the general-reader version, with the right note of honesty about its limits: “Anyone in possession of a predetermined encryption key could stash any code of their choice in an SSH login certificate, upload it, and execute it on the backdoored device. No one has actually seen code uploaded, so it’s not known what code the attacker planned to run. In theory, the code could allow for just about anything, including stealing encryption keys or installing malware” (Goodin 2024). What we almost lost was not a specific act of sabotage. It was a general-purpose key to the back of the house, and what the holder of the private key would have done with it remains, deliberately and permanently, unknown.
The phrase “predetermined encryption key” points to the design property that makes the counterfactual genuinely frightening and at the same time keeps it honest. The door opened only for whoever held the matching private key. Valsorda spelled this out when asked what he meant by calling the backdoor “gated”: “it takes the attacker’s private key to use the backdoor (it’s NOBUS),” he wrote, and by “unreplayable” he meant that “even if we observe an attack against one host, we can’t reuse it against another host” (Valsorda 2024, post 3koyhqjgcke2k). NOBUS is a term of art from the intelligence world, short for “nobody but us”: a backdoor engineered so that only its intended operator, holding a secret key no one else possesses, can pass through, while everyone else faces what looks like an ordinary, sound lock. The XZ Utils backdoor was NOBUS by construction. It was not a hole anyone could walk through once they knew about it; it was a private entrance keyed to a single party. Roccia made the same point from the other side: “There is a hardcoded certificate used, so that means only the attacker itself can log in to the compromised server, because he already [has] the private key” (Roccia 2024, 29:51).
That property cuts two ways. It bounds the danger: a deployed backdoor would not have been a free-for-all, and a captured attack could not be replayed against the next machine, because the attacker’s signature was bound to one host. But it also explains why ordinary network detection would have struggled once it shipped. The backdoor was built to be invisible to everyone but the key-holder. On any input it did not recognize, it behaved like an unmodified server. “Apparently the backdoor reverts back to regular operation if the payload is malformed or the signature from the attacker’s key doesn’t verify,” Valsorda noted, with the consequence for defenders that “unless a bug is found, we can’t write a reliable/reusable over-the-network scanner” (Valsorda 2024, post 3kowkezwz6g2q). It did not announce itself on the network: “No evidence of it calling home so far,” he reported in the first days (Valsorda 2024, post 3koyopvgoga2n). It left nothing in the records an administrator would consult; in Sam James’s flat summary, “Successful exploitation does not generate any log entries” (James 2024). It did not even slow a victim’s machine in any way an operator would notice. Kevin Beaumont, testing a deliberately infected box, found nothing to see: “I had to double check it was actually vulnerable as I wouldn’t even see a speed issue. For me, it was a completely transparent backdoor — where sshd was running from disk as usual, with the usual file hash and no extra network activity” (Beaumont 2024).
The open-source response, once the alarm was public, worked: distributions reverted, researchers dissected, the thing was contained. But that immune system never engages with a backdoor that, in operation, is indistinguishable from healthy tissue. There were no exploitation logs to raise an alarm, no network chatter to flag, no performance hit to notice on a production server, and, absent a bug, no reliable over-the-network behavioral signature for a scanner to match. The one cost the backdoor could not fully suppress was a faint delay in the milliseconds before login, on a machine watched closely enough, by someone curious enough, to ask why. That, and not any detection system, is what surfaced it. The capability was designed to defeat exactly the apparatus that later performed so well. It was caught through the one channel the operation had not quite closed.
If the capability was catastrophic, the reach was specific, and the credibility depends on the difference. The backdoor did not threaten “every Linux machine,” and saying so is not a hedge; it is the truth, and the more carefully it is told the heavier the real danger lands. Begin with the question a general reader will rightly ask: how could a flaw in a compression library possibly reach the program that guards remote logins? The two have nothing to do with each other. The answer is a quirk of how the major distributions assemble their software. As Przymus and Durieux put it in their 2025 MSR paper, “OpenSSH itself does not depend on liblzma, but this dependency is indirectly enforced by Linux distributions that patch OpenSSH to support systemd. . . . OpenSSH is patched and linked against libsystemd to support sd_notify” (Przymus and Durieux 2025, 93). The chain is worth slowing down on. OpenSSH is the standard suite that provides sshd, the server that handles remote logins. Several large distributions modify their copy of sshd so it can report its status to systemd, the program that manages services on most modern Linux systems, and that modification links sshd to libsystemd, which in turn pulls in liblzma. A compression library ended up loaded inside the login server, through a side door no upstream developer had built and few had reason to notice.
Cox drew the boundary that follows from this, and it is the single most disciplining sentence in the record: liblzma was “a dependency of OpenSSH sshd on Debian, Ubuntu, and Fedora, and other systemd-based Linux systems that patched sshd to link libsystemd. (Note that this does not include systems like Arch Linux, Gentoo, and NixOS, which do not patch sshd.)” (Cox 2024). The danger lived in a particular assembly choice, not in xz itself, and distributions that had not made that choice were never exposed through this path at all. Red Hat’s own investigation drew the perimeter tighter still. “Over the course of the investigation, we found that this is a kind of dormant malware targeting a very specific set of systems,” the company reported: “Specifically the x86_64 architecture, requiring both systemd and sshd. It was not targeting BSDs, Android cell phones, wi-fi routers, IoT devices, Raspberry Pis, or anything else. Essentially, it had to be a standard Linux server” (Freire 2024).
The remaining conditions narrow it further, and they come from the discovery side. Freund’s email enumerated the filters the malicious code checked before it would arm itself: it was “targeting only x86-64 linux,” it ran only “as part of a debian or RPM package build,” the specific packaging machinery of the two big distribution families, and, “due to the working of the injected code,” it was “likely” able to “only work on glibc based systems,” glibc being the foundational C library those distributions are built on (Freund 2024). James’s community FAQ turned the filters into a profile of who was actually at risk: a system had to run a distribution using glibc, and had to “have versions 5.6.0 or 5.6.1 of xz or liblzma installed,” which was “likely only true if running a rolling-release distro and updating religiously” (James 2024). That last clause is the crux of the bound. The poisoned versions were so new that, in the window before the catch, they had reached mostly the users who live on the bleeding edge, not the stable servers that make up the bulk of the installed base. And none of it, James was careful to say, was a defect in the components the backdoor strung together: “This is not a fault of sshd, systemd, or glibc, that is just how it was made exploitable” (James 2024). The attack composed legitimate machinery; it did not break any of it.
So how large is “global,” told honestly? The exposed population, had the versions reached stable, would have been the internet-facing Linux servers running the vulnerable configuration, which is a very large number without being “everything.” SSH is the standard way administrators reach servers remotely, and many of the machines that answer it are the machines that keep web infrastructure running. Beaumont offered the one concrete exposure figure in the early record, and it should be read as what it was, a first-week estimate from internet-wide scanning rather than a settled census: “OpenSSH runs on almost 20 million IPs as of today,” he wrote, “and is almost 10 times more prevalent than RDP.” His own counterfactual was characteristically understated: “Had somebody successfully introduced a widely deployed backdoor, it would have been bad later” (Beaumont 2024).1
The danger scaled with something the 2024 internet has and the early internet did not: homogeneity. Clifford Stoll, chasing a different intruder through a more various computing world in the late 1980s, noticed the protection that variety afforded. “If everyone used the same version of the same operating system, a single security hole would let hackers into all the computers,” he wrote; the networks he knew ran a dozen incompatible systems, so “no single attack could succeed against all systems,” and “diversity in software is a good thing” (Stoll 1989). The world of 2024 is closer to the one Stoll warned about. A handful of systemd-based distributions, built on glibc, running OpenSSH, sit beneath a large population of internet-facing servers, and a single dependency reached into that population. The reach of the counterfactual is a direct function of that homogeneity: the less varied the substrate, the more one poisoned library is worth.
For a sense of magnitude that is measured rather than asserted, the closest precedent is Heartbleed, the 2014 flaw in another piece of ubiquitous open-source plumbing, the OpenSSL cryptographic library. A peer-reviewed study put numbers on how far a single vulnerability in shared infrastructure can reach: “we can reasonably bound the proportion of vulnerable Alexa Top 1 Million HTTPS-enabled websites as lying between 24–55% at the time of the Heartbleed disclosure” (Durumeric et al. 2014, 478). Between a quarter and a half of the most popular secure websites, vulnerable at once. That is the Heartbleed measurement, not an XZ Utils figure, and the distinction is load-bearing: it is offered as the sourced analogue for how large a single shared dependency can loom, not as an estimate of the backdoor’s reach, which stays bounded by version, distribution, build path, and release channel. The comparison the responders themselves reached for was bigger still. Goodin reported that Roccia’s widely shared infographic visualized “the sprawling extent of the nearly successful endeavor to spread a backdoor with a reach that would have dwarfed the SolarWinds event from 2020” (Goodin 2024), the realized supply-chain attack in which, as the security firm FireEye disclosed at the time, intruders had trojanized “SolarWinds Orion business software updates in order to distribute malware” (FireEye 2020). SolarWinds is what a supply-chain compromise looks like once it lands; the comparison is the responders’ attributed assessment of what the XZ Utils backdoor might have exceeded, not the narrator’s measurement.
All of which returns to “almost,” and to why the word is the wrong frame. Run the counterfactual one more time and notice what is and is not in it. The mechanism was finished: a working, weaponized remote-code-execution capability, demonstrated on a test bench. The targeting was deliberate and in place: the right architecture, the right build path, the right login server. The reach was real and large: a ubiquitous dependency moving toward the stable channels of the world’s dominant server platform. Nothing in the design of the open-source supply chain stopped any of it. The system did not detect the backdoor, did not reject it, did not slow its progress toward release. Every layer meant to provide assurance, the code review, the trust placed in a maintainer, the release process, passed the backdoor through, because the backdoor came in through those layers rather than around them.
What stopped it was external to all of that, and contingent. The responders, in the moment, reached for the plainest possible word. James, writing for the community in real time, set it in bold: “While not scaremongering, it is important to be clear that at this stage, we got lucky, and there may well be other effects of the infected liblzma” (James 2024). Luca Boccassi, a systemd and Debian developer, put it with a forward-looking edge on disclosure day: “We were pretty much on the brink of disaster, and got saved because someone’s login got slowed down enough that they went ‘mmh hang on a sec’. It seems to me we just got very, very lucky here. Will we be so lucky the next time this happens too?” (Corbet 2024, bluca comment, 2024-03-29). Those are their words: the contemporaneous reading of people who had just watched a catastrophe pass close enough to feel.
But “luck” alone is not quite right. What averted the disaster was not a coin landing the right way. It was contingency meeting competence: a chain of coincidences that surfaced a faint performance anomaly, and an engineer with the rare combination of instinct, expertise, and stubbornness to chase half a second of unexplained latency to its source rather than dismiss it. Remove either half and the backdoor keeps moving toward stable deployment. That is why “almost” misleads. It suggests a system with a margin, something that nearly engaged and did. There was no margin. There was an accident, and the right person standing where the accident surfaced. Valsorda, watching the teardown, named both things at once: “This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library,” and then, “Looks like this got caught by chance. Wonder how long it would have taken otherwise” (Valsorda 2024, post 3kouaom62oi2b). Best executed, and caught by chance. “Authorized upstream” is the whole of it in two words: the attack ran through legitimate trust, not a flaw in the code.
Set beside the other supply-chain disasters of the era, the XZ Utils incident rhymes with each and matches none, and the mismatches are instructive. The closest comparison for ambition is Stuxnet, the sabotage of Iran’s nuclear centrifuges that Kim Zetter reconstructed: its authors spent extravagantly on secret flaws, “four zero-day exploits in a single attack,” where, by the analysts’ account she relays, “one zero day was bad enough” and a single such bug could fetch “$50,000 or more” on the right market (Zetter 2014). The XZ Utils operation spent almost nothing on bugs. It needed no zero-days at all, because it acquired something more valuable than a stockpile of flaws: the legitimate authority to ship code inside a trusted project. The decisive investment was social, not technical.
For what a ubiquitous open-source vulnerability does when it actually lands rather than being caught, the reference is Log4j, the December 2021 flaw the U.S. Cyber Safety Review Board called, in its register and not the narrator’s, the kind of vulnerability that “can create a once-in-a-generation security event” (Cyber Safety Review Board 2022, iv). Within days, the Board recorded, Cloudflare alone “observed 400 exploitation attempts per second” (Cyber Safety Review Board 2022, 4). That is the realized version of the catastrophe the XZ Utils backdoor only approached, and the contrast disciplines the counterfactual in both directions: Log4j was trivially exploitable by anyone who learned of it, while the backdoor was the inverse, gated to a single key-holder and invisible on the wire. “Almost broke the internet” does not mean the catastrophe would have looked like Log4j’s stampede. It would have been quiet, exclusive, and operator-only, which is worse for the question of who was behind it, not better.
One more disanalogy has to be stated plainly, because the most alarming comparisons mislead on exactly this point. The fastest-spreading incidents of the era, WannaCry and NotPetya, were self-propagating worms; WannaCry, Nicole Perlroth records, hit “200,000 organizations in 150 countries” within twenty-four hours (Perlroth 2021). The XZ Utils backdoor was not a worm. It did not spread on its own. It sat and waited for its operator to connect, which makes its likely reach smaller than a worm’s and its stealth far greater.
That is the shape of what was almost lost, told as honestly as the record allows: a finished, general-purpose remote-code-execution capability, invisible in operation; wired into the login servers of the dominant server platform on Earth through a dependency almost no one knew was there, and moving toward release paths that could have carried it deeper into the installed base; and locked to the holder of the private key. Not every machine. Not a worm. Not a certainty. But fully built, and stopped by an accident rather than by anything in the system meant to stop it.
Everything else follows from those three facts. The capability had to be built and hidden inside ordinary test files. The reach depended on the path from an upstream release to a running server. And the lock on the door points to the hardest problem of all. A backdoor that only the holder of the private key can open, leaving no logs and no trace, is not merely a technical achievement; it is a design for never being identified. Someone held the private key that fit that door. Who that was, the world still does not know, and the reasons it does not know are themselves part of the story.
Beaumont’s figure is a same-week count from internet-wide scanning, reported on March 31, 2024; it is carried here as his contemporaneous estimate of how widely OpenSSH is exposed, not as an independent census, and not as a count of vulnerable hosts. The population actually at risk was a subset of internet-facing servers, those running the specific affected configuration described above, not all of the roughly 20 million addresses answering SSH.↩︎