16 What Holds It Up
Two years after Andres Freund’s disclosure, the response to the XZ Utils backdoor had hardened into a stack of instruments a reader can recognize: a European regulation whose main obligations were still phasing in, a set of American secure-development frameworks whose procurement mandate had been rescinded, a push to standardize the list of every ingredient in a piece of software, and a handful of funds pointed at the people who keep the code alive. Each addresses something real. None, on its own, reaches the condition that produced the backdoor, which lived in the social layer of trust: the goodwill extended to a stranger, the authority handed to a maintainer, the release no one audited. A serious response would have to reach that layer, and the labor layer beneath it, without destroying the openness that makes the commons work in the first place. There are no clean fixes. The security researcher Kevin Beaumont said as much in the first week: “There are no easy fixes.. we should just try to reduce the risk and calmly work some solutions” (Beaumont 2024). What is left is to judge the responses on the table by a single question: which layer does each one actually touch, the law, the code and its metadata, market incentives, foundation capacity, maintainer labor, or social trust?
That judgment has to be made before the evidence to settle it exists. The responses are arriving faster than anyone can measure whether they work; the impacts of public funding on open source, one survey of the field concedes, “remain poorly understood, with a lack of consensus on how to meaningfully measure them” (Osborne et al. 2024, 1). And each instrument is an ethical choice that carries risk rather than a villain to be unmasked. Geoffrey Bowker and Susan Leigh Star, writing about the standards buried in every working infrastructure, put the discipline plainly: a standard “is an ethical choice, and as such it is dangerous—not bad, but dangerous” (Bowker and Star 1999, 5–6). Openness is dangerous in exactly that sense, and so is every proposal to constrain it. The temptation the political economist Elinor Ostrom warned against, after a career spent watching communities govern shared resources, is to mistake a model for the world and address “proposals to governments that are conceived in their models as omnicompetent powers able to rectify the imperfections that exist in all field settings” (Ostrom 1990, 215). The omniscient fix is the move to decline. What remains is to name the criteria a real response would have to meet, and to say, instrument by instrument, what reaches the under-defended layers and what does not.
The American response is best told as an arc, because it rose and fell inside those two years. It began with the right instinct. Executive Order 14028, signed in May 2021, located precisely the category of software that matters, “software that performs functions critical to trust,” the kind with “direct access to networking and computing resources” (Executive Office of the President 2021, sec. 4(a)), which describes liblzma wired into sshd almost exactly. But the same order diagnosed the problem as a property of “the development of commercial software,” which it said “often lacks transparency” (Executive Office of the President 2021, sec. 4(a)). XZ Utils was the harder inverse case: public in the ordinary open-source sense, yet compromised through a release path whose most dangerous contents did not match the public git history. The order did reach for open source by name, requiring producers to attest to “the integrity and provenance of open source software used within any portion of a product” (Executive Office of the President 2021, sec. 4(e)(x)), the two properties the operation corrupted. But it placed that duty on the firm that bundles the code, never on the unpaid maintainer who produces it, and the maintainer is where the operation happened. From the order came the Secure Software Development Framework, a catalog of practices that NIST was careful to call non-prescriptive, “not . . . a checklist to follow, but . . . a basis for planning and implementing a risk-based approach” (National Institute of Standards and Technology 2022, 3). A voluntary vocabulary, in other words, addressed to “software producers.”
Then procurement gave the voluntary vocabulary teeth. A 2022 federal memorandum ruled that agencies “must only use software provided by software producers who can attest to complying with” the framework’s practices (Office of Management and Budget 2022, sec. II): no signed self-attestation, no sale to the federal government. For three years that made the framework a condition of federal software purchasing. In January 2026 it was undone. A new memorandum rescinded the mandate, finding that its predecessor “imposed unproven and burdensome software accounting processes that prioritized compliance over genuine security investments” (Office of Management and Budget 2026, 1). The self-attestation form was not abolished but demoted to optional, a resource agencies “may choose to use” (Office of Management and Budget 2026, 1), and the software bill of materials, or SBOM (a machine-readable list of every component inside a piece of software), became one contractual tool agencies could ask for rather than a general federal requirement. The arc is the policy landscape’s advances and reversals in miniature, and the rescission’s own diagnosis, compliance bought at the expense of security, is the judgment the other instruments keep inviting, conceded by the policy that built the mandate in the first place.
The most recognizable remedy of all is the SBOM, which the order also introduced. The order reached for a homely metaphor, calling it “analogous to a list of ingredients on food packaging” (Executive Office of the President 2021, sec. 10(j)), and CISA’s 2025 public-comment draft kept the same handle, an “ingredients list” for software (Cybersecurity and Infrastructure Security Agency 2025, 4). On the response side it earns the praise. Once a vulnerability is public, an inventory of what is installed where genuinely speeds the cleanup, “significantly improving response time compared to organizations that did not have SBOMs” (Cybersecurity and Infrastructure Security Agency 2025, 12). But an ingredients list would not have caught this. Its mechanism is keyed to “a newly reported vulnerability” in a component that is “listed in the SBOM” (Cybersecurity and Infrastructure Security Agency 2025, 10), and before Freund’s disclosure there was no reported vulnerability for an SBOM tool to match: the backdoor rode inside liblzma, a normal, trusted, correctly named dependency. A conformant inventory would have listed xz versions 5.6.0 and 5.6.1 as ordinary components, with valid hashes, and flagged nothing.
The standard says so itself. “SBOM will not resolve all software security and supply chain concerns” (Cybersecurity and Infrastructure Security Agency 2025, 4), the draft concedes; an earlier report had already granted that “there are no cybersecurity panaceas, and SBOM is no exception” (National Telecommunications and Information Administration 2021, 7). The design assumes good-faith error. It asks consumers to tolerate the occasional honest mistake, then draws one explicit line: “this tolerance should not apply to intentional obfuscation or willful ignorance” (National Telecommunications and Information Administration 2021, 13). Intentional obfuscation, a backdoor concealed in test files and a release tarball, is exactly the XZ Utils case, and the framework excludes it by fiat rather than by mechanism. The empirical record is blunter still. When the U.S. Cyber Safety Review Board examined the Log4j emergency, the canonical open-source catastrophe, it spoke with organizations already using SBOMs and reported that “none reported having leveraged them to identify vulnerable deployments of Log4j” (Cyber Safety Review Board 2022, 13). Knowing your ingredients is not the same as trusting the cook.
The European Union’s Cyber Resilience Act is the one instrument that recognizes the structural fact the whole story turns on, that critical software is published rather than sold, kept alive by people and entities outside the market, and it answers that fact with a deliberately light hand. The regulation invents a legal category for it, the “open-source software steward,” and subjects stewards to “a light-touch and tailor-made regulatory regime” (European Parliament and Council of the European Union 2024, rec. 19), reaching the foundation layer the OpenSSF and the Linux Foundation occupy. But the steward’s entire substantive duty is to “put in place and document in a verifiable manner a cybersecurity policy” (European Parliament and Council of the European Union 2024, art. 24(1)): a documentation duty, not a resourcing one, a requirement to have a policy rather than to fund a maintainer. The enforcement that can reach a commercial manufacturer, fines into the tens of millions of euros, is switched off entirely for stewards, by an explicit derogation under which “the administrative fines . . . shall not apply to . . . any infringement of this Regulation by open-source software stewards” (European Parliament and Council of the European Union 2024, art. 64(10)). The lone, unpaid contributor falls outside the regime altogether (European Parliament and Council of the European Union 2024, rec. 18).
The two readings of that softness, protective and toothless, are both available, and the choice between them is deliberate, because coercion applied to volunteers would break the openness the law is trying to defend. In force since December 2024, with reporting obligations beginning in September 2026 and full application in December 2027, the Act is still at the stage where its effect can only be guessed at. One developer caught the register exactly in the disclosure-day discussion: “I think the CRA is a step in the development of the business models that will improve the funding situation in the future but I don’t think we yet know how this will work out” (Corbet 2024, kleptog comment, 2024-04-01). The law does one quiet thing worth marking. It opens a voluntary channel for reporting not only incidents but “near misses that could have resulted in such an incident” (European Parliament and Council of the European Union 2024, art. 15(2)). For an operation that was caught with half a second to spare, a regulatory category for the thing that almost happened is a small, apt grace note.
Money is the response that reaches furthest down. The funds are real and some of them are aimed squarely at the labor layer. Germany’s Sovereign Tech Agency, the public financier whose 2024 study of open-source maintenance an earlier chapter drew on, runs a Fellowship that pays “the typical work of maintainers up to €78,000 per year per individual” (Osborne et al. 2024, 7), a salary-scale grant to a named person rather than a bounty or a mandate; the OpenSSF-linked Alpha-Omega fund makes grants to critical projects; a 2026 Linux Foundation round named maintainer labor as its explicit target (Linux Foundation 2026). Where a project has an organization able to receive the money, it demonstrably works. The Rust programming language’s foundation moved, on Alpha-Omega funding, from “three part-time volunteers . . . limited to reactive support” to staffed security and software-engineering roles (Alpha-Omega 2025, 20). Money is not nothing.
But the funders name two failure modes themselves. The first is that the money routes to the legible core and misses the long tail, and the sharpest illustration is an irony the record hands over without comment: Alpha-Omega’s stated mission is to protect “the most critical open source software projects and ecosystems” (Alpha-Omega 2024, 16), and its 2023 grant slate, published on February 16, 2024, six weeks before the disclosure, named Eclipse, Node.js, Rust, OpenSSL, the Linux kernel, and several others, and did not include xz (Alpha-Omega 2024, 14). The fund whose word was “critical” missed a library that, six weeks later, would become one of the year’s defining examples of critical open-source risk. The second failure mode is durability. Grants end, and a one-time injection can leave a project more exposed than before; an evaluation of one public fund found that “over half of their funded projects struggled to secure follow-up funding” (Osborne et al. 2024, 15), a funding cliff a fellowship walks toward from its first day. Bounties, the market-shaped version of the remedy, can be worse than nothing, providing for organizations that skip the underlying work “an illusion of security” (Ellis and Bollampalli 2024, 31). The most disciplined judgment in the funding literature is also the simplest: invest in maintenance, not just in the market for bugs, because buttressing maintenance is not charity but, the report insists, an investment “in security” (Ellis and Bollampalli 2024, 37).
That is the layer every instrument keeps circling without landing on. Beaumont named the proportion in the same breath as his praise for the metadata work: “Let’s just keep doing the good SBOM work at CISA, and stop doing stunts around Huawei and such — Huawei is a speck of dust compared to the issues around tens of thousands of unpaid developers writing the core of the world’s most critical infrastructure nowadays” (Beaumont 2024). The endorsement and the limitation arrive together, from the same person, in the same sentence. The metadata is worth doing; it is also a rounding error against the human problem underneath it.
And the human layer cannot simply be told to do more. In the 2020 FOSS Contributor Survey, respondents reported that security already takes “an average of 2.27% of their total contribution time,” and that contributors “do not desire to increase this significantly” (Nagle et al. 2020, 5). The disinclination is cultural as much as economic. “I find the enterprise of security a soul-withering chore and a subject best left for the lawyers and process freaks,” one respondent to that survey wrote; “I am an application developer” (Nagle et al. 2020, 31). A separate study four years later found a core contributor to Ruby on Rails giving the same answer in three words about why volunteers avoid the work: “Because it sucks!” (Ellis and Bollampalli 2024, 24). Onto that disposition the new standards land badly. A majority of maintainers, 52%, said they “weren’t aware of any of these new government and industry standards” at all (Tidelift 2023, 10), and asked why they would not comply, they ranked the two reasons in order: 38% had no time, 37% were not paid for it (Tidelift 2023, 14).
A compliance obligation dropped onto unpaid, time-starved maintainers does not add security; it adds what the same survey called “unfunded mandates” (Tidelift 2023, 8), work nobody is paying for. The cost is already measurable: “asked to comply with requirements I don’t have time for” has become one of the named reasons maintainers cite for quitting (Tidelift 2023, 26). The mechanism by which a security regime exhausts the people it depends on is visible in the daily grind of vulnerability reports, where, one maintainer wrote, skewed incentives leave the maintainer “to do essentially all the work validating/disputing findings” (Tidelift 2023, 36), much of it for alerts that turn out to be empty. The tools that generate those alerts are part of the problem: one study of a widely used dependency scanner found that 88.8% of its findings, among the vulnerabilities the authors could independently check, were false positives (Ponta et al. 2020, 3175). The reframe that the institutions themselves have begun to reach is the criterion any serious response has to meet. “Ensuring our maintainers are well supported,” the OpenSSF and the OpenJS Foundation wrote after fending off a copycat attack, “is the primary deterrent we have against these social engineering attacks” (Bender Ginn and Arasaratnam 2024). Maintainer support is not welfare for the commons. It is a security control.
One finding runs underneath all of it: compliance is not security, and the policy wave largely does not reach the practice it names. Software is signed, but the signatures go unchecked. Engineers in one 2025 industry study described signing as a “check the box” technique performed to satisfy “regulatory, framework, customer or organizational requirements,” not as something they believed made them safer (Kalu et al. 2025, 92). The hardest single piece of evidence is a measurement: a study of public software registries found that the SolarWinds compromise, “the subsequent executive order and NIST guidance” included, “had no discernible effect on signing adoption” (Schorlemmer et al. 2024). That compromise, plus the American policy apparatus built in its wake, moved the practice by an amount too small to see. What the frameworks did move was SBOM collection, not verification (Kalu et al. 2025, 92): the layers cheap to demonstrate advanced, the layers that require sustained checking did not. And the one control aimed squarely at this kind of attack is inert against it by construction. A 2021 European best-practice guide advised buyers to “receive assurance of suppliers and service providers that no hidden features or backdoors are knowingly included” (European Union Agency for Cybersecurity (ENISA) 2021, 27). The word doing the work is “knowingly.” A supplier who is the adversary, an operator who has become the maintainer, can give that assurance in perfectly good-faith-looking form.
The technical fixes that do reach the artifact layer are real, and they close a real seam. The disclosure provoked an immediate push to stop shipping the hand-curated release tarballs the backdoor had hidden in: “we need to stop using curated tarballs, only auto-generated from tags” (Corbet 2024, bluca comment, 2024-03-29), one developer proposed, building releases automatically from the public source instead. The rebuttal came within a day, and it is the reason the fix is partial: stop curating tarballs, another answered, and “developers will just add generated files into the SCM” (Corbet 2024, jengelh comment, 2024-03-30), relocating the unreviewed content rather than abolishing it. A third sketched the durable version, a build pipeline that takes “the tagged source code commit, generate[s] artifacts and sign[s] them so you do have the provenance” (Corbet 2024, gdamjan comment, 2024-03-30), an unbroken, checkable record from source to shipped file. The distro responders had already done it by hand. Restoring xz in Gentoo, Sam James “reproduced [Collin’s] tarball with only minor differences in dates” and urged everyone downstream to do the same (James 2024, comment 46), rebuilding the release from source to confirm the two matched. Dependency surgery followed in the same spirit. systemd worked to reduce what libsystemd pulls in (Przymus and Durieux 2025, 98), and in version 9.8, released in July 2024, OpenSSH added support for notifying systemd through a standalone implementation that no longer needed libsystemd at all (OpenSSH 2024), removing the reason distributions had patched sshd to pull the library in, and liblzma with it.
Every one of these, though, makes the same assumption as commit signing: that the party who signs is the party to trust. That assumption is precisely the position the operation spent two years acquiring. The remedies reach the code, the metadata, and the procurement contract; they do not reach the layer of social trust where the backdoor was planted. Linus Torvalds, asked about the lesson, put the limit in one line: “when you have rules in place, the bad actors, they don’t follow the rules” (Mastery Learning 2024, 3:25). Rules bind the honest and route around the adversary. The controls that might actually reach the trust layer, verifying the human behind a commit, flagging the newcomer whose behavior looks unusual, are the ones that cut against the welcoming openness the commons runs on and the privacy of the ordinary contributor, who pays a cost a disciplined operator simply absorbs. A commons that verified everyone continuously would stop being a commons. That, and not any missing piece of metadata, is why the menu of responses leaves the structural condition standing. Clifford Stoll named the deeper hazard decades ago. A community like this one could be destroyed outright by an attack, he wrote, or, “worse, it could consume itself with mutual suspicion, tangle itself up in locks, security checkpoints, and surveillance; wither away by becoming so inaccessible and bureaucratic that nobody would want it anymore” (Stoll 1989). The remedy that misses the trust layer is one kind of failure. The remedy that reaches it by force is the other.
The double edge cuts even where the response was fastest. Within hours of the disclosure, GitHub suspended the operator’s account and took down the XZ Utils repository. That removed the attacker, and with him the public record of every change the world’s reverse engineers were at that moment racing to read; for a stretch the platform also suspended Collin’s own account, restoring it only days later (Boehs 2024). Excising the actor and foreclosing the investigation were, briefly, the same act. Platform moderation reaches the hosting layer, decisively and at once, and a hosting layer is not where the trust failure lived either.
It also leaves a residue, which is the first reason the story refuses a clean ending. The distributions reverted within days; the standing advice from CISA, relayed by Akamai, was simply to “downgrade to an uncompromised version, such as 5.4.6” (Akamai Security Intelligence Group 2024), and most of the world did. But the backdoor did not leave when the patch arrived. More than a year after the disclosure, twelve official Debian base images on Docker Hub still contained it, and they remained, the researchers who found them noted, because the artifacts “persist in container registries for a very long time” (Binarly REsearch 2025). They stayed not by oversight but by a decision. Asked to pull the images, a maintainer of the Docker library explained that the team had made “an intentional choice to leave these artifacts available as a historical curiosity” (Haruyama 2025, comment 2025-08-08), reasoning that the affected builds were old, never meant for production, and exploitable only by whoever held the backdoor’s key. The judgment is defensible and the risk is narrow; it is also a deliberate human choice to keep a state-grade backdoor downloadable into 2026. A removed backdoor can stay published as long as an installed vulnerability stays installed. The Cyber Safety Review Board had used the same word for Log4j, calling it an “endemic vulnerability” likely to persist “perhaps a decade or longer” (Cyber Safety Review Board 2022, v). The true tempo of the aftermath is not a clean closure but a slow administrative wind-down, long after public attention has moved on.
The second reason is the catch itself. It hung on half a second of latency on a test machine, and the contingency was real. “It was found randomly,” Torvalds observed; “random ends up being good” (Mastery Learning 2024, 3:01). That is the line to interrogate, not to adopt, because a defense that depends on luck is not a defense. But the randomness was not quite pure. The slowness Freund noticed existed partly because someone, for unrelated reasons, had been working to reduce sshd’s dependencies and shrink its exposed surface, an effort that plausibly forced the operator to rush and produce the very latency that gave the operation away. Freund drew the lesson himself: the episode “shows the value of doing some prospective security work to reduce your exposed code areas” (Freund and Roccia 2024). And there were, as Torvalds noted, no gates designed to catch this, and yet “they were actually really caught fairly quickly” (Mastery Learning 2024, 2:30). What worked was not a control but the distributed, public system functioning as it is built to: many hands, an open record, a stranger’s unrelated diligence, and one engineer who chased an anomaly nobody had asked him to chase. Jonathan Zittrain had described the only defense a system like this really has, years before the operation began. Security in such a system, he wrote, “requires the continuing ingenuity of a few experts who want it to work well, and the broader participation of others with the goodwill to outweigh the actions of a minority determined to abuse it” (Zittrain 2008, 166). Freund was the expert; the distributions and the reverse engineers were the goodwill; “Jia Tan” was the minority. Contingency met competence. But that is a description of an immune response, not a guarantee, and an immune response can fail; it is not the same as a system designed to stop the next one, and the honesty of that “almost” depends on keeping the two apart.
Ten days after the disclosure, Collin was reverting the operator’s changes one commit at a time, with the same unhurried care that had run the project for years. One of those commits removed a piece of the backdoor and listed, among its reasons, simply: “Backdoors are bad for security” (Collin 2024). He put his project back in order, and he maintains it still.
The question that should hang over all of it is not Collin’s but one a security researcher left ringing at the end of a conference talk, after walking an audience through every move of the operation. “How many other Jia Tans,” Thomas Roccia asked, “do you think are paid to insert backdoors in open-source projects?” (Roccia 2024, 41:27). These instruments are the world’s first attempt at an answer, and none of them reaches the place the question comes from. The half-second held. There is no reason yet to believe the next one will.