3 Forty-Eight Hours
The message went out on Friday, March 29, 2024, under a subject line that named the catastrophe and softened nothing: “backdoor in upstream xz/liblzma leading to ssh server compromise.” It went to oss-security, the public mailing list where the open-source world airs its security problems (Freund 2024a). A researcher who later reconstructed the affair for a conference audience fixed the moment to the minute: the story “began on Friday, [March] 29, at exactly 8:51,” with Andres Freund’s message arriving under that subject line (Roccia 2024, 2:38). The archived post bears him out: its header is stamped 8:51 in the morning, Pacific time (Freund 2024a). What is not in dispute is the effect. The previous chapter ended with Freund pressing send. This one begins the instant the message arrived everywhere at once.
Freund’s email did more than raise an alarm; it documented, in a few clipped sentences, how he had already tried to route one. “Given the apparent upstream involvement I have not reported an upstream bug,” he wrote. “As I initially thought it was a debian specific issue, I sent a more preliminary report to security@…ian.org. Subsequently I reported the issue to distros@. CISA was notified by a distribution” (Freund 2024a). Three things are folded into that paragraph, and each would shape the hours that followed. The first is restraint: “apparent upstream involvement” is as far as Freund would go toward saying that the project’s own maintainer was implicated, and he went no further toward naming anyone, a discipline the rest of this book inherits. The second is the path the warning took: first to Debian’s security team, then to the closed distros@ list where the major distributions coordinate on embargoed flaws, then onward to the United States government. The third is buried in the passive voice of the last clause. CISA was notified by a distribution. The Cybersecurity and Infrastructure Security Agency (CISA), the civilian arm of American cyber defense, did not detect one of the most sophisticated supply-chain attacks yet found in open-source software. It was told, by a volunteer-run Linux distribution, after the fact.
The orderly look of that paragraph is a thing of hindsight. The decision underneath it had felt nothing like a procedure. “It felt very clear that it needed to go out very broad,” Freund said afterward of the hours before he posted. “Mechanics of that, I was, like, wildly unsure about. . . . who do you inform?” (2024b, 57:31). The what was obvious and the how was not. There was no playbook for an engineer who had just established that a piece of the world’s baseline software had been sabotaged by one of its own maintainers, and no switchboard to call. He improvised the routing, the routing worked, and the fact that it had to be improvised is part of what the next two days revealed.
Because the catch did not close the emergency. It opened it. Freund’s half-second of suspicion had, by the time he posted, hardened into proof that the official releases of xz carried a backdoor. What no one yet knew on Friday morning was how far those releases had traveled, what exactly the hidden code did, or whether the people now scrambling to contain it were themselves already compromised. The disclosure converted a private investigation into a public race, and the race is the subject of this chapter.
To see why the response took the shape it did, it helps to know the shape it was supposed to take. Open-source culture treats openness as close to sacred. “In free software, doing things openly and transparently is normally almost a religious credo,” the developer and writer Karl Fogel observes in his standard handbook on running such projects, before noting the single place the credo yields. “. . . Security bugs are different” (Fogel 2020, 113). The exception is well worn: when a serious vulnerability surfaces, the usual practice is a quiet, time-limited embargo, a small circle of trusted developers and distributors who build and stage a fix in private and then disclose to the public all at once, so that defenders and attackers learn of it in the same instant. Freund’s case fit none of that. There was no good-faith upstream to coordinate with, because the upstream was the problem. The maintainer’s account was the adversary’s account. And the backdoor was not a latent flaw waiting to be exploited; it was a live weapon already shipping in releases.
So the embargo behind the Friday post was unusually short and unusually strange, and most of it has been told only once, by one of its participants. By Red Hat’s account, the coordination began on Wednesday, two days before the public disclosure. “On Wednesday, March 27, Andres contacted the Debian security team via their contact email,” the company wrote in a later retrospective, “and let them know about the oddities he found in a SSH slowdown when using a new XZ library that was shipped by Debian” (Freire 2024).1 Debian was the entry point because Debian was where Freund had first seen the symptom, and from there the alarm climbed into the cross-vendor machinery.
At Red Hat the tip did not stay with one team. “InfoSec involved Red Hat’s Product Security (ProdSec) department, the ones in charge of securing the software that we provide to our customers and community,” the account continues, and what assembled was “a global multidisciplinary team containing many of our finest developers, PenTesters, offensive analysts, managers, directors and program managers” who “started the then confidential efforts to properly establish, understand and assess the findings from Andres” (Freire 2024). The self-congratulation is Red Hat’s own; what matters for the record is the speed and the shape. Within a day of an emailed tip, a major vendor had stood up a confidential incident team and split it along the two questions that would organize the entire response: had the malicious code reached anything already shipped, and what, precisely, did the payload do.
The closed mailing lists could only reach so far. To pull in the vendors who were not on them, Red Hat reached past the volunteer community to the federal government. “[D]ue to the limited reach of this community, we engaged the US Cybersecurity and Infrastructure Security Agency (CISA) to start a collaborative Vulnerability Information and Coordination Environment (VINCE) ticket,” the company wrote (Freire 2024). The VINCE ticket became the hub the volunteer distribution lists could not be: a place where operating-system vendors, hardware companies, and national response teams could see the same facts at once, with Freund himself kept in the loop as the investigation moved. By Red Hat’s count the ticket drew in a remarkable crowd. Collaboration moved “around the clock through 48 participating entities: BSDs, proprietary software vendors, national Security Response teams and the heavyweights of the hardware and software landscape,” all “working together to advance and level the field on the findings . . . all done at speed to make the issue public as soon as possible” (Freire 2024).
The decision those participants reached was to abandon the long embargo and go public almost at once. The logic was the inverse of the usual case. An ordinary embargo buys time to build a fix before attackers learn of the hole; here the fix already existed, because the remedy was simply to stop using the two poisoned versions and fall back to an older one, and every additional day of silence was a day the backdoor kept shipping to anyone who updated. So the circle chose a moment and disclosed in concert. “On Friday, all the ducks (and there were a lot of them) were lined up,” Red Hat wrote, and “the participants in the VINCE ticket, including the finder, agreed to bring this to the public as soon as possible. As soon as Andres’ post went live in OSS-Security, CISA alerted the general public and Red Hat published a post as well” (Freire 2024). The simultaneity that looked, from outside, like the spontaneous reflex of a startled ecosystem had in fact been arranged. March 29 was Good Friday; the timing ruined a fair number of holidays, and the people in the ticket chose to ruin them anyway.
Once the post was public, the distributions moved in something close to unison. Within hours the open-source news site LWN, reporting the disclosure, appended a single-line update that caught the breadth of it: “there are advisories out now from Arch, Debian, Red Hat, and openSUSE” (Corbet 2024). Four major distributions, four advisories, in the span of an afternoon. What that terse sentence compresses is worth slowing down on, because the advisories were not a formality. Each was a distribution telling its own users, in its own voice, to act now.
Debian’s was the spare, official kind. Its security advisory named the discoverer and located the damage precisely: “Andres Freund discovered that the upstream source tarballs for xz-utils, the XZ-format compression utilities, are compromised and inject malicious code, at build time, into the resulting liblzma5 library” (Bonaccorso 2024). The hyphenated xz-utils is Debian’s name for the source package, and liblzma5 the library it builds; the sentence is the tarball-versus-source split, seen from the receiving end, the same split the previous chapter watched Freund uncover. Then the directive, plain enough to function as an alarm: “Users running Debian testing and unstable are urged to update the xz-utils packages” (Bonaccorso 2024).
Red Hat’s was the loud kind. Fedora, the community distribution Red Hat sponsors, had shipped the poisoned versions to users on its fastest-moving branches, and the advisory did not stand on ceremony. “Yesterday, Red Hat Information Risk and Security and Red Hat Product Security learned that the latest versions of the ‘xz’ tools and libraries contain malicious code,” it began, naming the two compromised releases, 5.6.0 and 5.6.1, and the freshly assigned identifier CVE-2024-3094 (Red Hat 2024). Then it raised its voice. “PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES for work or personal activity,” it shouted, in capitals, telling users that the development branch would be rolled back to a safe xz and could be trusted again only afterward (Red Hat 2024). A vendor bellowing at its own users in capital letters, inside a day, is the response arc in a single line.
openSUSE’s was the ingenious kind. Its maintainers had rolled the package back on March 28 and shipped a clean snapshot built from a known-good backup, and the version string they gave the reverted package was a small piece of defensive craft: 5.6.1.revertto5.4, a number engineered to sort as newer than the malicious 5.6.1 while actually carrying the older, safe 5.4 code, so that the ordinary update machinery would pull users back to safety rather than strand them on the backdoor (Meissner 2024). The same advisory carried a quieter, more honest admission. “As of March 29,” it noted, “reverse engineering of the backdoor is still ongoing” (Meissner 2024). The distributions were pulling the package and ordering reinstalls before anyone fully understood what the package did.
Gentoo’s response had begun, quietly, before the emergency was even public. Its terse first notice told users only what they needed: “A serious bug is being investigated. Please downgrade ASAP until we have a fix” (James 2024, comment 2). But the commit that blocked the package was time-stamped the evening of Thursday, March 28, the night before Freund’s post, and the developer Sam James noted that the block had gone up “since yesterday, after Debian, suse, fedora, et. al all did the same” (James 2024, comment 2). The coordinated pull, in other words, was already running under embargo before the public ever heard of it. The discipline of that response shows in how Gentoo rated the danger. A security developer recorded that the team privately judged the “likelyhood of an impact on Gentoo” to be “small,” then rated the bug at maximum severity anyway, reasoning that “with the backdoor not fully analyzed yet” they had to “opt for a A0 severity, assuming a (currently unclear) worst case scenario where the backdoor could still be included and may act with root privileges” (James 2024, comment 9). A0 was the top of Gentoo’s scale, the rating held for the gravest blockers. The responders defaulted to the worst case precisely because they could not yet rule it out, which is what a disciplined immune response looks like and also what an absence of margin looks like.
The consumer-facing distributions spoke in the second person. Kali Linux, the security-testing distribution, told its users plainly that if they had updated “on or after March 26th, but before March 29th, it is crucial to apply the latest updates today” (Kali Linux 2024). The word was today. Even distributions whose stable users were never at risk reacted with a caution that rippled outward for days. Canonical, the company behind Ubuntu, found no exposure in the version of xz headed for its next long-term release, but it had pulled and rebuilt its packages out of caution, and the disruption pushed back a milestone: “the Beta release for Ubuntu 24.04 LTS (Noble Numbat) has been pushed to April 11, 2024 (previously April 4, 2024)” (Zemczak 2024). That slip came on April 3, days into the emergency rather than in its first hours, and it is a useful measure of how far the uncertainty propagated. A well-funded corporate distribution whose stable channel was never in danger still lost a week of its flagship release schedule to a backdoor in a compression library.
While the distributions reverted, the platform at the center of the story did something blunter. The public xz repository was hosted on GitHub, and on the night after the disclosure GitHub took the whole thing down. A participant in the Gentoo bug thread reported the moment to the minute: “Github has disabled the XZ repository entirely as of 1:29 AM UTC” (James 2024, comment 17). The authoritative repository for the code the world was now urgently trying to inspect had vanished mid-investigation.
The takedown was sensible as triage and clumsy as justice, because GitHub did not only suspend the account behind the backdoor. It suspended the project’s longtime maintainer too. The co-maintainer who had introduced the backdoor operated under the name “Jia Tan,” a persona this book takes up later and whose human author has never been identified; the other account belonged to Lasse Collin, who had built and maintained xz for the better part of two decades and had been manipulated into accepting “Jia Tan’s” help. GitHub suspended them both. In Collin’s own later accounting, “GitHub accounts of both me (Larhzu) and Jia Tan were suspended. Mine was reinstated on 2024-04-02” (Collin, n.d.). For four days the maintainer of the compromised project was locked out of it while the rest of the world picked through the wreckage.
Inside the distributions the first hours were less a fix than a triage operation still defining its own scope. In Debian’s bug thread a developer answered the obvious, anxious question, what should we do, with a candor that captures the moment: “Yes, a multi-team task force is working on it and will inform users once it is known how to proceed, inclusing how much to throw away and rebuild” (Hess 2024, msg #97). How much to throw away and rebuild was not rhetorical. Part of what made the scramble disorienting was that the responders could not immediately be sure they were clean themselves. As one Debian contributor pointed out, the people who build and upload the distribution’s packages tend to run exactly the fast-moving branches the backdoor had reached: “many people (and I’d guess that includes DDs/DMs) run their workstations/laptops with testing/unstable,” he wrote, using the project’s shorthand for Debian Developers and Maintainers, so it was “not enough to know that Debian stable is likely not affected” (Hess 2024, msg #82). The channels that carried the backdoor and the channels that carried the response were, to an uncomfortable degree, the same channels.
As the distributions pulled the package, a second response was already underway, faster and more diffuse: the worldwide, self-organizing effort to take the thing apart. It started within hours. Filippo Valsorda, a well-known cryptography engineer, posted his first reaction less than three hours after Freund’s email, and it reads like a man watching a building collapse and cataloging the architecture on the way down. “Woah. Backdoor in liblzma targeting ssh servers,” he wrote. “. . . It has everything: malicious upstream, masterful obfuscation, detection due to performance degradation, inclusion in OpenSSH via distro patches for systemd support… Now I’m curious what it does in RSA_public_decrypt” (Valsorda 2024, post 3kotxq46zj223). Two clauses in that breathless inventory reach backward and forward through this book. “Detection due to performance degradation” is Freund’s half-second seen from the outside, the entire catch compressed into four words. And “inclusion in OpenSSH via distro patches for systemd support” is the supply-chain path a later chapter has to explain, the unlikely chain by which a compression library ended up inside the program that guards remote logins. The last line, the curiosity about RSA_public_decrypt, points straight at what the backdoor was for, a question the next chapter takes up.
What followed was collaboration in real time, conducted in public and governed by the community’s own etiquette. “I’m watching some folks reverse engineer the xz backdoor, sharing some *preliminary* analysis with permission,” Valsorda posted, the asterisks his own and the “with permission” a small courtesy in the middle of an emergency (Valsorda 2024, post 3kowjkx2njy2b). This is the distributed, public system that had not made the catch, now doing the next job: not one institution dissecting the payload behind closed doors, but a scattered crowd of specialists narrating the analysis to one another as it happened, each deciding what to share and when.
The work produced practical checks almost as fast as findings. Freund’s own disclosure email, written before any of the reverse engineering had finished, already carried a practical attachment: “Vegard Nossum wrote a script to detect if it’s likely that the ssh binary on a system is vulnerable, attached here. Thanks!” (Freund 2024a). The point is the clock: the public triage began before the payload was fully understood. openSUSE had said as much on the day (Meissner 2024), and the tooling and the triage advanced alongside the analysis rather than after it.
As the analysis spread, the incident was also being entered into the formal machinery of vulnerability tracking. It received an identifier, CVE-2024-3094, the standard catalog number by which the world’s security systems would refer to it ever after, assigned by Red Hat in its role as a numbering authority. The catalog entry carries a credits field, and the field is worth reading for what it says about everything that came before. The record of a patient, multi-year operation names exactly one human being, and it is the engineer who tripped over it by accident: “Red Hat would like to thank Andres Freund for reporting this issue” (CVE Program 2024). The attribution of the operator stays open, as it does throughout this book. The attribution of the discovery is fixed, in the permanent record, to a man who had been running a database benchmark.
The federal government’s own statement arrived the same day, in the spare register of an agency that had been told rather than tipped off. “CISA and the open source community are responding to reports of malicious code being embedded in XZ Utils versions 5.6.0 and 5.6.1,” the alert read, before glossing, for a general audience, what the obscure package even was: “XZ Utils is data compression software and may be present in Linux distributions” (Cybersecurity and Infrastructure Security Agency 2024). The phrasing is quietly telling. CISA placed itself beside the open-source community, “CISA and the open source community are responding,” not above it, which is an accurate description of where the government stood in this story. Its instruction was a single sentence asking for three things: “CISA recommends developers and users to downgrade XZ Utils to an uncompromised version—such as XZ Utils 5.4.6 Stable—hunt for any malicious activity and report any positive findings to CISA” (Cybersecurity and Infrastructure Security Agency 2024). Downgrade, hunt, report. A private discovery had become, within a day, a government-issued call to act.
That CISA had a role to play at all rested on groundwork laid before the incident. Six months earlier, in September 2023, the agency had published a roadmap for open-source security that staked out exactly this kind of coordinating function: “CISA will continue to coordinate vulnerability disclosure and response for OSS vulnerabilities by leveraging relationships with the OSS community,” it had written, anticipating even the pattern the XZ Utils response would follow, processes to “specifically look for upstream issues in open source packages that critical infrastructure organizations depend on and quickly notify affected users of the identified vulnerabilities” (Cybersecurity and Infrastructure Security Agency 2023, 8). The roadmap described coordination after a vulnerability surfaces, not detection of a slow trust compromise before release, which is the limit worth keeping in view. But it is why a federal alert about a compression library was part of the machinery at all.
The first operational question that machinery faced was brutally concrete: who was running the poisoned versions, and who could prove they were not. That is the question a software bill of materials, an itemized inventory of the components inside a piece of software, is meant to answer. As a federal report on the format put it, some uses “require complete or mostly complete graphs, such as the ability to ‘prove the negative’ that a given component is not on an organization’s network” (National Telecommunications and Information Administration 2021, 12). An inventory like that does nothing to catch a hidden backdoor in advance; what it speeds, once the alarm is public, is exactly the frantic accounting the response now demanded.
Step back from the forty-eight hours and the speed of all this cuts two ways. The response was fast, broad, and competent, and its very competence throws into relief how little of it had anything to do with the catch. Every distribution that reverted, every researcher who dissected the payload, every practical check that followed, all of it was triggered by the disclosure. None of it found the backdoor. That was the burden of a separate observation, made that week by the security researcher Kevin Beaumont, and it is the uncomfortable core of the chapter. “Nobody else had raised concerns,” Beaumont wrote, “and I don’t believe any existing security tooling or processes would have caught this (I realise there will be a torrent of vendors claiming they detect this… but they will detect this now that somebody told them)” (Beaumont 2024). The jab at after-the-fact detection marketing is earned. No public alarm and no automated control had surfaced the thing before Freund; the entire apparatus that moved so impressively on Friday had been inert on Thursday.
What that apparatus did accomplish, once it moved, was real, and Beaumont named it precisely: “Because Andres privately researched the issue and got the Linux distributions to take it seriously, he averted this reaching any kind of wide (or even small) deployment in the real world” (Beaumont 2024). The backdoor was evicted before it reached the stable releases that most of the world actually runs. Red Hat, looking back, allowed itself the understatement that the episode “had the potential to really hurt the open source community” (Freire 2024). The potential was the point, and how narrowly it was contained is the subject of the chapter that follows.
It is tempting to read the clean disclosure as evidence that the system simply works well, and a comparison cuts against that comfort. A decade earlier, in 2014, a flaw called Heartbleed had exposed a different weakness in the same broad ecosystem, and its disclosure had gone badly. A peer-reviewed study of that episode found that “patching was delayed because the Heartbleed disclosure process unfolded in a hasty and poorly coordinated fashion,” with several major vendors “not notified in advance of public disclosure” (Durumeric et al. 2014, 486). Against that, Freund’s single deliberate email, handing the distribution security teams the initiative before any public scramble, looks like a model of coordination. But Heartbleed also supplied a measurement that should sober anyone inclined to relax. Even when disclosure went well enough that “the most prominent websites patched within 24 hours,” the same study “observed vulnerability scans from potential attackers within 22 hours” (Durumeric et al. 2014, 486). The window between a flaw going public and attackers moving on it is counted in hours, not weeks. Heartbleed was an accidental bug, found in good faith, not a deliberate backdoor, and its numbers are its own; but it measures the clock the XZ Utils responders were racing, and explains why a circle of forty-eight entities chose to ruin their holidays rather than wait.
So the forty-eight hours resolve into a single, double-edged fact, and the book’s argument lives in the hinge between its halves. The cure was systemic. The catch was not. Once the alarm was public, the open, distributed mechanism functioned about as well as anyone could ask: an outsider had found the thing, a coordinated disclosure had been arranged across nearly fifty organizations, every major distribution had reverted within hours, a scattered crowd of researchers had reverse-engineered the payload while practical detection began, and the backdoor had been contained before it reached a single stable release. That is not the picture of a fragile system failing. It is the picture of an immune response working. What the same forty-eight hours also show is that none of that capacity was ever pointed at finding the backdoor in the first place. The response was a cure the system could mount only because an accident had bought it the time to do so: a chain of coincidences, set out in the previous chapters, that surfaced the anomaly, met by an engineer with the rare instincts to chase it down. The system had no built-in margin to catch what Freund caught. It had a formidable capacity to react once he had.
The clearest verdict on how the disclosure was handled came, in the end, from the person it had hurt most. When Collin returned to his project and tore the backdoor out, he used the commit message to thank the man who had exposed the betrayal of his life’s work: “Thanks to Andres Freund for finding and reporting it and making it public quickly so others could act without a delay” (Collin 2024). The maintainer whose account had been suspended, whose project had been hijacked, and whose trust had been the attack’s actual target endorsed the very choice that broke the security world’s usual quiet, going public at once, because the only thing that made the response possible was that nobody waited.
That endorsement should not be mistaken for a clean triumph, and the responders themselves were the first to say so. Three weeks after the disclosure, when the acute emergency had passed, the same Debian developer who had worried about compromised maintainers noted what the fast, public response had quietly left undone. “It feels as if there are still many discussions about how to prevent such things in the future,” he wrote, “but less so about the concrete fallout of the particular backdoor, where it seems most people were lead to conclude from media reports, that an attack was only possible if sshd was actually running an reachable” (Hess 2024, msg #152). The immune response had been public and quick, and it had also turned to the comfortable question, how do we stop the next one, while the unglamorous one a worried user actually had, am I safe, went underanswered. That is the honest shape of it: neither the triumph Red Hat described nor the doom the headlines reached for, but a real capacity that worked because, this once, it was given the time. The harder question is the one the responders were racing, mostly without stopping to articulate it. What, exactly, had almost happened? What would the backdoor have done if Freund had moved a few days slower, or never run the benchmark at all? That is the counterfactual the next chapter takes up.
Red Hat dates Freund’s first contact with the Debian security team to Wednesday, March 27 (Freire 2024). The widely cited public timelines place it a day later, on March 28 (Boehs 2024; Cox 2024). The discrepancy is a single day and is left unresolved here; the public disclosure date, Friday, March 29, is not in dispute.↩︎