10 The Release Train
By March 9, 2024, the operation’s whole material achievement amounted to two compressed archives on a download server. Lasse Collin would later fix their identity in two flat sentences: “XZ Utils 5.6.0 and 5.6.1 release tarballs contain a backdoor. These tarballs were created and signed by Jia Tan” (Collin, n.d.). A backdoor in a release file compromises no one by existing. Between those archives and the millions of servers they were aimed at stood machinery that almost nobody outside the trade ever has reason to look at: the system that turns an upstream release into the operating systems the world installs. The operation’s final phase was a campaign against that machinery, and the near-miss at the center of this story happened inside it. The danger was never only that malicious code existed upstream. The danger was that trust, packaging automation, and release calendars were in the process of converting it into infrastructure.
The machinery rewards a plain description. An operating system like Debian or Fedora is not written by anyone. It is assembled. A distribution gathers thousands of independent projects (the kernel, the login server, the compression libraries), builds each into an installable package, and ships the whole as one coherent, vouched-for system; the projects that write the code are upstream, the distributions that package and deliver it are downstream, and the words simply describe the direction the software flows.
The unit that moves between them is old. In the years before there was any standard platform for hosting code, Nadia Eghbal observes, “most open source code was published as a ‘tarball’ . . . on some self-hosted, stand-alone website” (Eghbal 2020, 22), and the release tarball, the bundled snapshot at the center of the concealment story, is still the thing an upstream hands to the world. Two sociologists of infrastructure, Geoffrey Bowker and Susan Leigh Star, argued that understanding systems like this means foregrounding their “normally invisible Lilliputian threads” and granting them “causal prominence in many areas usually attributed to heroic actors” (Bowker and Star 1999, 34). The people recede here, and the plumbing decides.
The plumbing is also old in a second sense. Infrastructure, Star wrote, “wrestles with the inertia of the installed base and inherits strengths and limitations from that base. Optical fibers run along old railroad lines” (Star 1999, 382). xz is an old railroad line: wired into the foundations years ago, inherited by every new build because the base beneath already carried it, and inherited along with its limitations, which by 2024 included a thinly supported maintainer and a release process nobody else checked.
What a distribution ships is not one product but a graded series of them. The Linux kernel’s own release model sorts its output into four categories, “Prepatch (or ‘-rc’) kernels, Mainline, Stable, and Long Term Stable” (Linux Foundation 2020, 17), and the stable stream is the trunk the whole industry grows from: “These stable updates are the base from which most distributor kernels are made” (Corbet and Kroah-Hartman 2016, 7). Distributions extend the same gradient to everything they ship. At the fast end are the rolling and unstable lanes, where a new upstream release can land within days and the users are developers and enthusiasts who accept the risk. Behind them sit testing lanes, where the next numbered release is staged and shaken out. At the slow end are the stable and enterprise releases, the ones running banks and hospitals and the machines nobody wants surprised, which change reluctantly and on a published calendar. New code enters at the fast end and earns its way toward the slow end. Which lane a piece of code has reached is, for most practical purposes, what its danger means, and the entire XZ Utils incident is a story about lanes.
Seen from the user’s side, the apparatus is a convenience so dependable it disappears. Seen from an attacker’s side, it is something else. A 2020 study of malicious packages stated the inversion in one sentence: “From an attacker’s point of view, package repositories represent a reliable and scalable malware distribution channel” (Ohm et al. 2020, 16). The industry’s own self-descriptions make the same point without meaning to, calling artifact repositories the “App Stores” of modern software development and “critical points of trust in every developer’s workflow” (Alpha-Omega 2025, 15). Institutional doctrine even has a name for the move. A supply chain attack, in the definition of the European Union Agency for Cybersecurity (ENISA), “is a combination of at least two attacks. The first attack is on a supplier that is then used to attack the target” (European Union Agency for Cybersecurity (ENISA) 2021, 6). The definition imagines a supplier compromised from outside. What happened here was stranger and cheaper. There was no break-in. The operator had spent two years becoming the supplier, and the campaign began already inside the trust boundary the definition assumes must be breached.
The power of a trusted delivery channel had a recent demonstration. NotPetya, the self-propagating catastrophe an earlier chapter set beside this one, entered the world through an update mechanism: the attackers, in Andy Greenberg’s reconstruction, “hijacked the company’s update servers” of M.E.Doc, a Ukrainian tax-accounting package, so that machines installing a routine update received the payload with the vendor’s own blessing (Greenberg 2019). Ben Buchanan drew the structural lesson: “It was the software’s reach, rather than its function, that made MeDoc an ideal vehicle for their attack” (Buchanan 2020). The comparison must be held in proportion: M.E.Doc was a commercial product whose servers were seized from outside, and the XZ Utils payload never detonated anywhere. But the geometry rhymes. An unglamorous, single-purpose piece of software becomes strategically decisive because of how many systems sit downstream of it, and a compression library wired beneath the world’s Linux systems offered reach of exactly that kind.
The reach had a precise and almost accidental shape, traced in an earlier chapter. Upstream OpenSSH, the project that writes the login server, does not use the compression library at all. Andres Freund put the chain in three clauses in the disclosure email: “openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma” (Freund 2024). In plain terms: several large distributions modify the login server they ship so that systemd, the program that supervises services on most modern Linux systems, can be told when the server has finished starting; that modification links sshd against the libsystemd helper library; and libsystemd, for its own unrelated reasons, can read compressed data, so it pulls in liblzma. No upstream developer designed a path from a compression library into the login machinery. The path was assembled downstream, by packaging decisions, which is to say that the attack surface itself was a product of the release train. Freund, retelling the discovery later, kept the point and its limit together: “So only that indirect loading of the library would actually cause the backdoor to be active. So it’s very indirect” (Freund and Roccia 2024).
The indirection was a bound as well as a route, and one distribution proves it. Gentoo, which builds packages from source on the user’s own machine, never carried the patch: “In Gentoo, we don’t patch net-misc/openssh with systemd-notify support which means liblzma, at least in the normal case, doesn’t get loaded into the sshd process,” the Gentoo developer Sam James explained in the project’s incident bug (James 2024, comment 3). Same upstream code, same poisoned releases, different train: no path to the login server at all. Exposure was a property of distribution mechanics, not of the code alone.
A release that nobody packages reaches nobody, and the operation did not leave the packaging to chance. The operator tagged and released 5.6.0 in late February (Cox 2024, 2024-02-24), then updated the backdoor files and released 5.6.1 in early March (Cox 2024, 2024-03-09). “In the following weeks,” Dan Goodin summarized in the week of the disclosure, “Tan or others appealed to developers of Ubuntu, Red Hat, and Debian to merge the updates into their OSes” (Goodin 2024). The “or others” is the record’s honesty: who sat at which keyboard remains unresolved. The timing, by the reading of the software-engineering study that later reconstructed the campaign, was synchronized to the calendar: the operator “waited until a month before the official RedHat release and close to Debian release to push his attack into the repository during a period of high project activity” (Przymus and Durieux 2025, 95), a reconstruction rather than a documented intention, but one consistent with the dated record of what happened next.
The main stage was Debian, and the position was undefended in a specific, institutional way. Every Debian package has a named maintainer responsible for shepherding upstream releases into the archive, and the xz package’s maintainer had drifted away from it years earlier; the package survived on the goodwill of a developer who was not its maintainer, Sebastian Andrzej Siewior, working through a procedure whose name says it plainly: the Non-Maintainer Upload, or NMU. “I NMU maintained it for the last few years,” Siewior wrote that March, adding that he was considering taking the package over officially (Jansen 2024, msg #32, 2024-03-27). The condition should sound familiar. Upstream, one volunteer in Finland with little backup; downstream, an absent maintainer and a stand-in doing unowned work. The same vacancy, stacked twice in the same supply chain, and the campaign had already exploited the first instance.
Into that vacancy, in late March, came a request. A Debian bug titled “xz-utils: New upstream version available,” asking the distribution to import the backdoored 5.6.1, was filed under the name “Hans Jansen” (Jansen 2024), a name that had surfaced once before, in mid-2023, contributing groundwork for the very optimization that later hid the backdoor’s hook (Boehs 2024). Evan Boehs, tracing the accounts afterward, found the request had been opened the same week the “Hans Jansen” Debian account was created, and that the account had seeded “a few similar ‘update’ requests in various low-traffic repositories to build credibility, after asking for this one” (Boehs 2024). A supporting chorus materialized: “Several other, suspicious, anonymous name+number accounts with little former activity also push for its inclusion, including misoeater91 and krygorin4545. krygorin4545’s PGP key was made 2 days before joining the discussion” (Boehs 2024), a cryptographic identity minted for the occasion. It was the sock-puppet method of the pressure campaign two years earlier, pointed now at a distribution instead of a maintainer.
The push met friction, and the friction is the immune system, caught in the record. Thorsten Glaser, a Debian developer, objected on procedural instinct: “Very much not a fan of NMUs doing large changes such as new upstream versions” (Jansen 2024, msg #27, 2024-03-26). In the same message he asked the question underneath the whole affair, addressed to the absent maintainer: “what’s up with the maintenance of xz-utils? . . . are you active? Are you well?” (Jansen 2024, msg #27, 2024-03-26). Ordinary suspicion of an irregular upload, and ordinary concern for a colleague: nothing in the objection required knowing about the backdoor. The procedure itself was generating resistance, which is what procedures are for.
The train was already moving. A non-maintainer upload importing 5.6.1 had been prepared the day before Glaser wrote, its changelog announcing “Non-maintainer upload” and “Import 5.6.1” (Jansen 2024, msg #5, 2024-03-25), and on March 27, 2024, the bug closed with the version’s arrival in unstable, Debian’s fast lane: “We believe that the bug you reported is fixed in the latest version of xz-utils, which is due to be installed in the Debian FTP archive” (Jansen 2024, msg #42, 2024-03-27). The upload that carried it was not the work of any sock puppet. It was Siewior’s, and its changelog announced, alongside the import, “Takeover maintenance of the package” (Jansen 2024, msg #42, 2024-03-27): a conscientious developer formally adopting an orphaned package, fixing real reported issues, doing exactly what the project would have wanted done, and carrying the backdoor into the archive in the same gesture. Forty-eight hours later Freund’s email reversed it. The release train converts trust into infrastructure, and it cannot tell, because it was never built to tell, whether the trust has been captured.
Debian was not the only channel being worked. The Intercept reported a Red Hat employee describing how “Jia Tan” had personally lobbied him to help get the compromised xz releases into Fedora (Mazurov 2024); a Fedora contributor, in Boehs’s reconstruction, remembered the sales pitch as “great new features” (Boehs 2024). “Jia Tan also attempted to get it into Ubuntu days before the beta freeze,” Boehs records (Boehs 2024), the deadline after which Ubuntu’s next release stops accepting new versions. Kevin Beaumont noted a further bid pointing at the kernel itself: “a request was opened to make the threat actor a Linux kernel module maintainer for XZ Embedded” (Beaumont 2024). And the quiet result of all of it, before the catch: “These changes were committed to Github back in February, and made their way into test releases of Debian, Fedora and Kali Linux. Nobody noticed the problem” (Beaumont 2024).
When the catch came, the first descriptions reached for totality. “With a library this widely used, the severity of this vulnerability poses a threat to the entire Linux ecosystem,” the Kali Linux project wrote (Kali Linux 2024), and, as a statement about the library’s reach, that was not wrong. But the dated record the distributions published over the following days draws a much more precise shape. The phrase “every Linux system” has to be bounded by version, by distribution, by build path, and above all by lane.
Debian’s security advisory did the bounding in its first breath: “Right now no Debian stable versions are known to be affected. Compromised packages were part of the Debian testing, unstable and experimental distributions, with versions ranging from 5.5.1alpha-0.1 (uploaded on 2024-02-01), up to and including 5.6.1-1” (Bonaccorso 2024). Two facts sit in that sentence. The compromise had been flowing into Debian’s fast lanes for nearly two months, since the first of February: the earlier uploads had entered by the ordinary route, and the late-March campaign was a race to put the updated 5.6.1 in behind them, while 5.6.0’s valgrind noise was drawing exactly the attention the operation could not afford. “The race is on,” Russ Cox’s timeline notes on March 4, “to fix this before the Linux distributions dig too deeply” (Cox 2024, 2024-03-04). And the compromise never touched the stable releases that run production machines. The fix was a reversion to the known-good upstream 5.4.5, shipped under the version string 5.6.1+really5.4.5-1 (Bonaccorso 2024): Debian’s house idiom for the same defensive craft openSUSE used in the same week, a number that sorts as an upgrade so the update machinery will accept it, with the +really confessing what is actually inside.
Red Hat’s advisory drew the same line at its own boundary: “No versions of Red Hat Enterprise Linux (RHEL) are affected by this CVE” (Red Hat 2024). On the community side, “the packages are only present in Fedora 40 and Fedora Rawhide within the Red Hat community ecosystem” (Red Hat 2024), and the advisory paused to explain what Rawhide is, “the development distribution of Fedora Linux,” the always-moving stream that becomes the next numbered release (Red Hat 2024). Red Hat’s own retrospective made the channel boundary explicit: “While no Fedora stable builds were affected, Fedora 40 beta nightly builds, the leading edge of Linux innovation, were affected” (Freire 2024). The payload had reached the edge and only the edge.
The rest of the map repeats the pattern with different dates. openSUSE: “Our rolling release distribution openSUSE Tumbleweed and openSUSE MicroOS included this version between March 7 and March 28” (Meissner 2024), a three-week window in the fast lane, while the same vendor’s fixed releases never inherited it because “SUSE Linux Enterprise and openSUSE Leap are built in isolation from openSUSE Tumbleweed” (Meissner 2024), a statement that is at once an accurate technical fact and a vendor reassuring its customers. Kali Linux, the security-testing distribution built on Debian: “The impact of this vulnerability affected Kali between March 26th to March 29th” (Kali Linux 2024), roughly seventy-two hours, and its remedy arrived down the same rails as the compromise had: “It has already been patched in Debian, and therefore, Kali Linux” (Kali Linux 2024). The train that delivered the backdoor delivered the fix. Even on Docker Hub, the public registry of prebuilt system images from which containerized deployments are assembled, the payload arrived only from Debian’s fast lanes: it “was only found in development tracks of Debian, not any releases,” as a maintainer of Debian’s official images later put it (Haruyama 2025, comment 2025-08-13). And the formal record bounds the upstream side to exactly two releases: in the CVE catalog, xz is affected at 5.6.0 and 5.6.1, and everything else is unaffected by default (CVE Program 2024).
Two things were true at once. “The backdoor . . . affected mostly a development version, so it was not widely deployed when Andres found the backdoor,” the security researcher Thomas Roccia told the DEF CON audience (Roccia 2024, 6:24); “Do I need to panic? No,” Beaumont advised the enterprises checking their estates (Beaumont 2024). And: “Malicious updates made to a ubiquitous tool were a few weeks away from going mainstream,” ran the subheading on Goodin’s report (Goodin 2024), a judgment Luca Boccassi, a Debian and systemd maintainer, made specific on disclosure day: “This was caught before it got in any stable release of any distribution, it’s only in development/testing. The only exception is SUSE Tumbleweed, because it’s rolling. Just a few weeks of delay and it would have been part of the new Fedora 40 release and the new Ubuntu LTS 24.04 release” (Corbet 2024, bluca comment, 2024-03-29). For calibration, the going rate for detection in one 2015–2019 dataset of malicious packages was an average of 209 days from publication to public report, with a worst case of 1,216 days for a package quietly taken over after abandonment (Ohm et al. 2020, 11). This one was caught in weeks, before it had reached the slowest lanes. A genuine emergency, and a bounded one: the urgency and the bound come from the same machinery.
The map of where the packages went still overstates the danger, because having the affected version present was not the same as the exploit taking effect. The payload was choosy about where it would even build itself. “The attack appears to specifically target amd64 systems running glibc on Debian or Red Hat derived distributions, although other systems may also be vulnerable,” the Przymus and Durieux study records (Przymus and Durieux 2025, 93), and James had described the mechanism in the first days: the injection script “looks for .deb and .rpm specific files/environment variables in the build environment” (James 2024, comment 3), arming itself only while being assembled into the package formats Debian- and Red Hat-style distributions ship. After that it still needed the systemd-patched sshd at runtime. Three conditions, stacked.
The stacking produced cases of presence without fire. Red Hat’s advisory, in a correction issued a day after its first read, determined that Fedora 40 beta “does contain two affected versions of xz libraries” yet “does not appear to be affected by the actual malware exploit” (Red Hat 2024): the poisoned bottle on the shelf, the cork never drawn, and the day-later correction itself a small instance of the rolling, self-revising response of those weeks. Gentoo shipped the affected xz-utils inside a stage3 archive, the base-system bundle from which a Gentoo machine is first installed, built on March 24 (James 2024, comment 12), and was still not impacted, because its build path never armed the payload and its sshd never loaded the library (James 2024, comment 3). The version’s presence is a fact about distribution. The exploit’s firing is a fact about build conditions. The two came apart everywhere outside the targeted path.
Canonical, Ubuntu’s publisher, drew the deepest version of that distinction, and no other vendor went so far. Rather than revert a package, it distrusted its own factory: it decided “to remove and rebuild all binary packages that had been built for Noble Numbat after the CVE-2024-3094 code was committed to xz-utils (February 26th), on newly provisioned build environments” (Zemczak 2024), the February date being Canonical’s own fixing of the poisoned commit. The reasoning: any binary compiled in an environment that might have contained the backdoored xz at build time was suspect, whether or not that binary contained the library, because a poisoned build tool can taint what it builds. The stated aim was affirmative: “confidence that no binary in our builds could have been affected by this emerging threat” (Zemczak 2024). Noble Numbat, the working name of the Ubuntu 24.04 LTS release then weeks away, was itself still a pre-release; the stable channel was never in question; uncertainty about build integrity alone justified throwing away the builder and starting clean.
Getting the thing out again was harder than letting it in had been, and the difficulty lived in the same machinery. The Debian bug demanding a response was titled, plainly, “revert to version that does not contain changes by bad actor,” and its author, the developer Joey Hess, stated in it the problem that made a simple downgrade insufficient: “Reverting the backdoored version to a previous version is not sufficient to know that Jia Tan has not hidden other backdoors in it. Version 5.4.5 still contains the majority of those commits” (Hess 2024, msg #5). The damage was epistemic, not just technical. Once a trusted contributor stands exposed as hostile, no version number restores confidence, because every commit that passed through his hands, two and a half years of them, is now a question, a contamination problem a later chapter returns to.
Even the mechanical half was fraught. Reverting that far back, the Debian developer Aurélien Jarno warned, “will break packages that use new symbols introduced since then”; his quick survey of what would break named dpkg, erofs-utils, and kmod, and he added the operational understatement: “Having dpkg in that list means that such downgrade has to be planned carefully” (Hess 2024, msg #10). Symbols, here, are the named entry points a shared library offers; programs built against newer ones will not run with an older library. And dpkg is the tool that unpacks and installs every Debian package. Rip out xz carelessly and the system could lose the ability to install anything, including the fix. That is the inertia of the installed base, rendered as a bug comment: the library could not simply be removed, because the means of removing things depended on it. The condition is general and it compounds, as the Census II report observed of aging packages whose support thins while their deployment does not: they “become more likely to break with each passing day without the guarantee of support on-hand to provide fixes” (Nagle et al. 2020, 30).
The same thread holds the response’s most concrete act of repair. Within a day, Hess had built an exit: “I have prepared a git repository that is a fork of xz from the point I identified before the attacker(s) did anything to it. In my fork, I have renamed liblzma to liblzmaunscathed. That allows it to be installed alongside current dpkg without breaking dpkg with an old version of liblzma. . . . The goal is not to take over from xz upstream, but to get the possibly backdoored code off of production systems ASAP” (Hess 2024, msg #62). One volunteer, overnight, reconstructed a clean line of the library from before the operator’s first touch and renamed it so it could stand beside the installed base without breaking it: a stopgap, offered as a stopgap, and a measure of what the distributed immune response could improvise while the institutions were still drafting advisories.
The full revert lost anyway, and the reasons it lost are the machinery’s gravity made visible. Siewior, weighing how far back to go, named what a deep rollback would cost: the 5.4.x series had threaded decompression he wanted to keep, the 5.6.x series had a faster decompressor, and, decisively, “I want to stay on an official upstream release which is also used by other distros” (Hess 2024, msg #142). Features, performance, and alignment with what everyone else ships: the gravity that converts upstream code into infrastructure is exactly the gravity that resists un-converting it. Meanwhile every distribution encoded its remedy differently. Debian shipped 5.6.1+really5.4.5-1; Alpine kept the 5.6.1 label and switched to building from git archives instead of release tarballs; Gentoo deleted its masked 5.6.1 package outright; and users comparing notes got lost in the strings, as one admitted in the Gentoo bug: “First, I saw 5.6.1 and did not pay attention to the next part…” (James 2024, comment 36). The United States government’s own advisory pointed at yet another target, recommending a downgrade to 5.4.6, a version uncoordinated with the fixes the distributions actually shipped (Cybersecurity and Infrastructure Security Agency 2024). After the train had run, a version number alone could no longer tell anyone whether they were safe.
Two further facts bound the story, one about the attack and one about the cure. The first: the path was fragile, and ordinary maintenance nearly closed it. On February 29, 2024, Cox’s timeline records, a contributor known as teknoraver “sends pull request to stop linking liblzma into libsystemd. It appears that this would have defeated the attack” (Cox 2024, 2024-02-29): a routine dependency-slimming change, nothing to do with security, that would have severed the only road from the compression library into the login server. Beaumont, who examined the same sequence, noted that “the fix for this was already in train before the XZ issue was highlighted” but “hadn’t yet rolled out into a release of systemd” (Beaumont 2024), and he ventured a reading of the operator’s late hurry: “I believe there’s a good chance the threat actor realised this, and began rapidly accelerated development and deployment” (Beaumont 2024), the public bug reports and distribution lobbying of March as a race against a closing window. Cox passes the hypothesis along explicitly as speculation, and speculation is what it remains: a plausible reconstruction of tempo, unproven, and a question that returns with the hunt for the operator. What is documented is the contingency itself. The corridor the operation needed was being walled up by accident, by maintenance, while the campaign to ship through it accelerated.
The second fact runs the other way. The friction that makes the stable lanes slow is real, and it is usually told as a defect. Nicole Perlroth, reporting on critical-infrastructure patching, recorded that “Automated patches were still big no-nos inside critical infrastructure networks,” updates needing high-level approval and narrow maintenance windows, with even urgent fixes deferred for fear of disruption (Perlroth 2021). She was explaining why known holes stay open for months. But the same gating ran the other way in March 2024: the conservatism that slows every fix is also what had kept the backdoor out of the lanes that matter. The friction that slows the cure slowed the disease. The train’s slowness is not a virtue or a vice; it is a property, and that month it cut in the defenders’ favor.
What, then, does the machinery actually verify? At every coupling of the train there are signatures: the maintainer signs the release tarball, the distribution signs its packages, the mirrors serve signed indexes. Signing, as a large-scale study of the practice defines it, uses public-key cryptography to “bind an identity (e.g., a package maintainer’s private key) to an artifact (e.g., a version of a package),” so that anyone “can verify whether the artifact was indeed produced by the maintainer” (Schorlemmer et al. 2024). The backdoored tarballs passed that test perfectly. They were produced by the authorized signer; the signer was the operator. And the chain checks in one direction only, a structural habit a study of industry practitioners summarized as “a unidirectional use of Software signing, establishing trust for outputs while placing lower expectations on the provenance of inputs” (Kalu et al. 2025, 93). Each link signs what it ships and trusts what it receives: a supply chain of signed outputs that are nobody’s verified inputs.
The closed-source world had already demonstrated the consequence at scale. The SolarWinds implant, an earlier chapter’s comparison, rode inside what its discoverers described as “a SolarWinds digitally-signed component of the Orion software framework that contains a backdoor” (FireEye 2020): the malicious update verified flawlessly, because the compromise sat upstream of the signature. The disanalogy is real, a vendor’s compiled binary in a commercial update channel against a build-time injection into an open-source release tarball, but the lesson is identical. A signature authenticates an artifact and the identity behind it. It says nothing about the good faith of that identity. Where the signer is the attacker, the signature is the attack’s passport.
The law has since internalized the threat with striking precision. The European Union’s Cyber Resilience Act, defining the severe incidents manufacturers must report, gives as its example “a situation where an attacker has successfully introduced malicious code into the release channel via which the manufacturer releases security updates to users” (European Parliament and Council of the European Union 2024, rec. 68), which is very nearly a definition of the XZ Utils operation; the regulation was not written about this incident, but it names exactly this seam. The same act mandates the software bill of materials, or SBOM, an ingredients list for software, “in a commonly used and machine-readable format covering at the very least the top-level dependencies of the products” (European Parliament and Council of the European Union 2024, Annex I, Part II(1)). The floor in that phrase is the limit: top-level dependencies are the ones a product declares directly. The link that mattered was not sshd declaring liblzma; it was a transitive, distribution-specific path through a patch and libsystemd, below the floor the mandate names.
The U.S. guidance that defined the SBOM grasped the deeper seam too: the data can be collected from source or from the build, and the two diverge, since “a compiler may pull in a slightly different version of a component than what was expected from the source” (National Telecommunications and Information Administration 2021, 14), so consumers “should try to obtain it from the instance of the build” (National Telecommunications and Information Administration 2021, 18). That recommendation points at precisely the gap the operation exploited, the distance between the inspectable source and the shipped artifact. But the honest middle ground runs in both directions: a build-instance inventory of the poisoned release would have faithfully recorded a correctly named component, from a trusted project, signed by an authorized maintainer. An SBOM is an inventory, not a verdict. It would not have caught this, and it is not therefore worthless; it answers “what is in the box,” not “should the box have been trusted.”
And the train, it turns out, does not fully un-ship. In 2025, more than a year after the reverts, researchers at the firm Binarly went looking for the backdoor on Docker Hub. The backdoored Debian packages of March 2024 had been captured into base images, and the base images had been built upon: “other images have been built on top of these infected base images, making them transitively infected” (Binarly REsearch 2025). They counted “more than 35 images that ship with the backdoor,” and bounded their own number in the same breath: they had scanned only a small portion, only Debian-based images, only to the second order (Binarly REsearch 2025). With “nearly 12 million repositories” on the registry, an exhaustive scan “would be infeasible” (Binarly REsearch 2025). The figure is a floor, not a census, and the affected images sit in development tracks, not in anything a stable system pulls by default (Haruyama 2025, comment 2025-08-13). But the installed base does not run backward. What the release machinery ships, it cannot fully recall; it can only ship something newer on top.
That is the problem the catch preempted, and the era’s other great dependency fire shows what it looks like realized. When the Log4j vulnerability detonated in 2021, the U.S. Cyber Safety Review Board found that “there is no comprehensive ‘customer list’ for Log4j, or even a list of where it is integrated as a sub-system” (Cyber Safety Review Board 2022, iv); the library was “routinely embedded in other software components, often unknown at later levels of integration or system operation” (Cyber Safety Review Board 2022, 11). The two cases sit at different levels: Log4j was an accidental flaw whose crisis was a post-disclosure inventory problem, while the XZ Utils backdoor was an authored operation concealed before disclosure. But had 5.6.1 ridden the spring release calendar into Fedora 40 and Ubuntu 24.04 LTS, then sat there quietly for a year, the world would have faced Log4j’s question, where is it, with a payload built by an adversary instead of a bug, and the honest answer would have been the same: no one keeps that list.
Step back from the dates and the version strings and the machinery resolves into a single image. Every gate the payload passed was working as designed. The maintainer’s signature on the tarball was valid. The bug asking Debian for the new version followed the form. The upload that imported it was a model act of package stewardship. The lanes carried the code at exactly their rated speeds, fast at the edge, slow toward the center, and the slowness bought the time in which one engineer’s curiosity could matter. The train neither failed nor saved anyone; it moved what the upstream gave it, as it does every day, for thousands of projects, almost always to everyone’s benefit. Its one assumption, repeated at every coupling, is that the upstream is sound: that behind every release tarball stands a project, and behind every project stands someone fit to be trusted with all of it. For XZ Utils, the someone had been one unpaid volunteer in Finland, and then, by handover, his attacker. How many of the other thousands of upstreams look like that, what the cathedral under the train actually is and who keeps it standing, is the next question, and it has been waiting since 1991.