9 The Payload
What the operator shipped, once the analysts finally took it apart, provoked a reaction that complicates everything else about it: the people best equipped to despise the thing admired it. That response has a respectable pedigree. Donald Knuth, the computer scientist whose multivolume The Art of Computer Programming gave the field its working sense of its own craft, used his 1974 Turing Award lecture to insist that the word “art” in that title was exact and not decorative: “When I speak about computer programming as an art,” he said, “I am thinking primarily of it as an art form, in an aesthetic sense” (Knuth 1974, 670). The claim he built on it was about reading, not writing. “When we read other people’s programs,” Knuth went on, “we can recognize some of them as genuine works of art” (Knuth 1974, 670). The recognition happens in the act of dissection, which is exactly where it happened here: not in the operator’s composition of the backdoor but in the analysts’ taking it apart, the same way Lasse Collin read his own poisoned repository afterward and marked, commit by commit, what had been done to it.
The vocabulary the reverse engineers reached for was not loose. Thomas Roccia, a security researcher at Microsoft, said it plainly on a podcast in the first days: “This is super sophisticated,” and unlike anything he had seen before (Freund and Roccia 2024). Kaspersky’s analysts, closing a teardown dense enough to satisfy any specialist, wrote that “several highlights make this threat unique” (Leite 2024). Naming that artistry is part of taking the threat seriously; flinching from it would understate what the world nearly absorbed. But the admiration has to be held at arm’s length, and the limit set in the same breath as the praise: technical admiration must not become glamour. The craft mattered only because it exploited real blind spots, the places where humans and their tooling do not look. Knuth’s lecture supplies its own escalating palette for skilled work, “elegant,” “exquisite,” “sparkling,” and one more besides, “noble” (Knuth 1974, 670). That last word is the one this program cannot earn.
The first move was the quietest. Collin, reviewing the wreckage line by line after the disclosure, put his finger on the exact commit, in January 2024, where more than two years of genuinely useful contribution turned into staging for an attack: “This is the first commit preparing for the backdoor,” he wrote against the addition of a pair of innocuous-looking test files (Collin 2024b). Test files are an unglamorous category. A compression library has to prove it can compress and decompress real data, including malformed or “corrupt” data it should reject cleanly, so a directory of binary sample inputs is ordinary and expected. The payload rode inside that ordinariness. As Collin described it once he had cut it out, “the executable payloads were embedded as binary blobs in the test files” (Collin 2024a), opaque chunks of data that no reviewer reads because there is nothing in them a human can read.
The directory was effectively unreviewable, and not by accident. The project’s own README had explained, long before “Jia Tan” arrived, that many of its test files were built by hand and could not be regenerated from anything more legible: “Many of the files have been created by hand with a hex editor,” it noted, so “there is no better ‘source code’ than the files themselves” (Cox 2024, 2024-02-23). Russ Cox, whose timeline became one of the standard references, flagged that the note predated the operator: the concealment exploited a legitimate, standing practice rather than a flaw introduced for the purpose. The test harness made the trick cheaper still. As Collin observed, the test runner selected its inputs by wildcard, so dropping a file into the directory was enough to wire it into the build’s own checks: “Backdoor files. . . . Note that tests/test_files.sh uses globs to pick the files. So just adding files means that a decompression test will be done with them” (Collin 2024b). The camouflage was layered even inside the fixtures. The malicious files carried real test content, genuine coverage for the library’s RISC-V filter, but padded far past what that content needed; the surplus was the payload. “The RISC-V test files also have real content that tests the filter,” Collin wrote, “but the real content would fit into much smaller files” (Collin 2024a). And the fixtures were not even doing the job their category implied. Andres Freund, in the disclosure email, noted that in the first poisoned release “the files were not even used for any ‘tests’ in 5.6.0” (Freund 2024b). Calling them test fixtures was itself the disguise.
The hiding place was chosen for what it is. Test infrastructure is the least-read, most-trusted code in a project. Nadia Eghbal, surveying maintenance at large, cataloged the genre’s neglect from the inside: there are “tests that are no longer useful, tests that are slow, tests that are flaky, tests that don’t do what the developer thought it would, and tests that are orphaned by their authors” (Eghbal 2020, 133–34). When the institutions later wrote up the attack for other maintainers, they named the same blind spot precisely. The joint OpenJS and OpenSSF alert listed, among the warning signs of a social-engineering takeover, “PRs containing blobs as artifacts,” and gave the case at hand as its example: “the XZ backdoor was a cleverly crafted file as part of the test suite that wasn’t human readable, as opposed to source code” (Bender Ginn and Arasaratnam 2024). Not human readable, as opposed to source code: the payload lived in the one corner of an open project where openness stops meaning anything, because there is nothing legible there to be open about.
How a blob in a test file becomes a running hook inside a login server is a staged chain, and the staging is where the craft is most visible. The trigger that started it has already appeared in this book: the single malicious line in build-to-host.m4, the build-system macro that an earlier chapter watched Freund trace, present in the released tarball and absent from the source repository. What that line set in motion is the rest of the chain. At the moment a packager prepared the software, the macro decoded the first test fixture, bad-3-corrupt_lzma2.xz, into a shell script; that script performed a more elaborate decode on the second fixture, good-large_compressed.lzma, producing a further script; and that script extracted a compiled object file, liblzma_la-crc64-fast.o, and slipped it into the compilation of liblzma, the compression library at the center of the whole story (Akamai Security Intelligence Group 2024). The formal record compresses the same sequence into a sentence: “the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code” (CVE Program 2024). There is a small, dark elegance in the means. To unpack each stage, the chain used xz itself, the very tool it was poisoning, as the instrument of its own corruption (Kaspersky GReAT 2024b).
Linking a hostile object file into a library is only useful if something calls into it, and the mechanism that arranged that is the most quietly clever part of the build. The extraction script made a one-character edit to a line of the library’s source, changing a call to a function named __get_cpuid into a call to _get_cpuid, deleting a single underscore (Kaspersky GReAT 2024b). That edit pointed the call at the attacker’s object file instead of the system’s. The redirection then exploited a legitimate optimization feature called an indirect function, or ifunc: a mechanism that lets a program choose, at the moment it loads, the fastest version of a routine for the particular processor it finds itself running on. The groundwork for that feature had been laid in the repository months earlier, in mid-2023, and Cox marked the interpretive limit honestly when he reconstructed it: the change “could be an innocent performance optimization by itself” (Cox 2024, 2023-06-22), and standing alone it was indistinguishable from one. Turned to the operator’s purpose, the ifunc machinery became the doorway through which the smuggled code seized the program’s flow of execution at startup.
From that foothold the payload reached forward into the login server it had no ordinary business touching. At startup it installed what is called an audit hook into the dynamic linker, the part of the operating system that resolves a program’s references to library functions into actual addresses in memory as the program loads (Freund 2024b). The hook let the backdoor watch that resolution happen and wait for one function in particular. “It appears to wait for ‘RSA_public_decrypt@....plt’ to be resolved,” Freund wrote, preserving his own uncertainty in the verb. “When called for that symbol, the backdoor changes the value of RSA_public_decrypt@....plt to point to its own code” (Freund 2024b). In plain terms, the program keeps a table of pointers to the library functions it calls, and the backdoor rewrote one entry in that table so that a call meant for the system’s code arrived instead at the attacker’s. The rewrite was possible only inside a narrow window. The table is made read-only once loading is finished, a routine hardening step, and the backdoor struck before it closed: “It is possible to change the got.plt contents at this stage because it has not (and can’t yet) been remapped to be read-only” (Freund 2024b). The whole construction is a sequence of legitimate mechanisms, each load-bearing in ordinary software, assembled into a path that none of them was built to permit.
What the hijacked function then did is the covert channel, and an earlier chapter already established the capability: pre-authentication remote code execution, the power to run an arbitrary command on a reachable server before anyone has logged in, gated so that only the holder of a secret key could use it. The mechanism beneath that capability turns on a fact that the hooked function’s own name obscures: RSA_public_decrypt does not decrypt traffic; it verifies signatures. Filippo Valsorda supplied the gloss the rest depends on: RSA_public_decrypt “is a (weirdly named) signature verification function” (Valsorda 2024, post 3kowk663sll2p), the misnomer an artifact of the symmetry in RSA’s mathematics, confirmed by the function’s own documentation, which lists it among “low-level signature operations” (OpenSSL, n.d., NAME). The server reaches that function when it checks the cryptographic key in a connecting client’s certificate. That is the moment the backdoor seized.
The command was hidden in the key. A client connecting to a backdoored server presents a certificate, and the backdoor read its command not from any message field but from the raw number at the heart of the certificate’s RSA key, the modulus, conventionally written N. Valsorda laid out the layering: “The payload is extracted from the N value (the public key) passed to RSA_public_decrypt, checked against a simple fingerprint, and decrypted with a fixed ChaCha20 key before the Ed448 signature verification” (Valsorda 2024, post 3kowk2vdagg2c). Each stage in that sentence is a deliberate filter. A cheap fingerprint check rejected ordinary traffic without spending effort; a decryption with a hard-coded key turned the smuggled bytes into a command; and a signature verification, using the elliptic-curve scheme Ed448, proved the command came from the one party holding the matching private key. That last check was bound to the specific machine. The signed data included a hash of the target server’s own host key, so a command captured against one server could not be replayed against another, a precaution Kaspersky read, correctly, as the adversary building the defending researcher into its threat model (Leite 2024). Only after all of it passed did the command run. Of the request types the backdoor recognized, the operative one handed its payload straight to the C library’s system() call, which runs a line of text as a shell command: “Type 2: executes null-terminated payload with system()” (Weems 2024). A certificate, the object a server inspects to decide whom to trust, had been turned into the envelope that carried the instruction to betray it.
Two final touches show how completely the design anticipated inspection. The verification key the backdoor used was never stored anywhere an analyst could simply find it; it was taken apart and scattered across the program’s own machine instructions, one bit at a time, and reassembled at runtime. Kaspersky’s team described being briefly fooled by it, and then the resolution: “the attackers used a custom steganography technique in the x86 code to hide an arbitrary message (in this case, the public key),” recovered only by walking “456 instructions” of the binary to rebuild it (Leite 2024). And the implant did not merely slip past the records a successful intrusion would normally leave; it rewrote them. The backdoor hooked the logging machinery and, in Kaspersky’s flat description of the result, “replaces these successful connection messages with messages about failed authentication attempts” (Leite 2024). A break-in was made to read, in the server’s own logs, as a failed login. Anthony Weems, who built a working trigger to prove the capability was real, confirmed the silence from the other side: “successful exploitation does not generate any INFO or higher log entries” (Weems 2024).
All of that is the true account of how the trigger was hidden, and it invites a conclusion that is tempting and not quite right: that the decisive concealment was locational. The malicious build-to-host.m4 shipped only in the release tarballs and never entered the git repository, so anyone reading the public source would have seen nothing. The fact is real and load-bearing. Red Hat stated it within a day: “the Git distribution lacks the M4 macro that triggers the build of the malicious code” (Red Hat 2024).1 And the gap drew no suspicion because shipping a release tarball that differs from the repository is ordinary practice. As Sam James explained in the FAQ that organized the early response, “the release tarballs upstream publishes don’t have the same code that GitHub has. This is common in C projects so that downstream consumers don’t need to remember how to run autotools and autoconf” (James 2024). A reviewer who noticed the difference would have found in it nothing more than the normal seam between a project’s source and its shipped artifact.
But the people closest to the artifact doubt that the locational trick was what mattered most. Freund, who caught the thing, said as much when pressed on whether openness had forced the operator’s hand: he thought they “could have actually committed it to the repo, and nobody would have noticed” (Freund 2024c, 01:10:11). A Canonical security maintainer, Marc Deslauriers, put the contrary case in the contemporaneous record more sharply still, arguing that the hidden code would likely have escaped notice in git as easily as in the tarball, and that the mismatch was if anything a liability for the attacker: “this was easily spotted because the tarball didn’t match the git repo” (Corbet 2024, mdeslaur comment, 2024-03-29). Another commenter reconstructed the operator’s reasoning the other way, as a deliberate bet that “a commit in git would certainly be more visible” (Corbet 2024, epa comment, 2024-03-30), though that imputes a motive the record cannot confirm: an inference, not a fact.
The tarball-versus-git gap is a real account of how the trigger was placed, and it was at most redundant insurance. The decisive concealment was social, as the previous chapter showed: the things the operator added, test files, a build macro, a release, were the authorized, reviewed, signable output of the very person the project existed to trust. The payload did not have to evade review. It arrived wearing review’s own credentials. The real attack surface was never the code’s location but the standing that decided whose code shipped at all, and once that standing was captured, where the bytes sat was a detail.
The admiration the craft compels is genuine, and Knuth’s lecture is also where it runs out, by the field’s own measure. Knuth did not treat his aesthetic as amoral. The pleasure he described was bound to purpose: he confessed to especially enjoying “writing programs which do the greatest good, in some sense” (Knuth 1974, 671). By that criterion, the standard set by the man who made the case for program-as-art in the first place, a program built to compromise the machines of everyone who installed it fails as art no matter how exquisitely it is made. The skill can be conceded without the word: the work was skilled, and the thing it was for disqualifies it from the only praise that would have counted.
The sophistication was also, in plain fact, fragile, and the fragility is part of what got it caught. Freund’s own assessment of the build-time machinery was that it was “over-engineered” (Freund 2024a), elaborate past the point of usefulness. The elaboration made it unreliable. “Sometimes when I would rebuild,” he recounted, “the backdoor would vanish. And then it would rebuild again, and the backdoor would reappear. And there’s also some concurrency bug in the make file” (Freund and Roccia 2024). The same surplus effort produced the tell that exposed the whole operation. The payload looked up far more symbols than its job required, and that needless work is what cost the roughly half a second of latency the book opened on, the anomaly Freund chased to its source. He came away unimpressed by exactly the part that should have been most careful: it read to him like “less experienced people using the techniques that other people built for them” (Freund 2024c, 41:15). Kaspersky reached the same double verdict, reading the campaign as a “patient, multi-year approach” that was nonetheless “somewhat clumsily executed” (Kaspersky GReAT 2024a). Sophistication and fragility are not opposites here. The over-engineering and the craft were the same thing seen from two sides, and the thing that made the payload remarkable is the thing that made it visible.
None of this was the unforeseeable novelty the early coverage sometimes implied, and the evidence that it was foreseen is specific. The 2020 catalog of malicious-package attacks an earlier chapter drew on, by Marc Ohm and colleagues, mapped the routes into the software supply chain as a tree, and one labeled branch was the concealment of malicious code in test cases (Ohm et al. 2020, 8). The fit is close but not exact: the paper’s category is code that executes when the tests run, striking the maintainer’s own machine, whereas the operation here used its fixtures chiefly as an inert hiding place for a blob assembled later, at build time. The same paper named the deeper structural fact that the operation turned on. Malicious code committed to a version-control repository, it observed, “is more accessible to manual or automated reviews of commits or entire repositories” than code introduced through the build and packaging path (Ohm et al. 2020, 7), which is the negative space of the concealment choice stated four years early. Ohm and his coauthors also drew the distinction that separates this incident from an accidental one, the line between a vulnerable package and a malicious one: the two “may look identical,” they wrote, and “the main difference lies in the intention of the developer” (Ohm et al. 2020, 3). Heartbleed, the catastrophe the book met earlier, was vulnerable: a mistake. This was malicious: deliberate, evasive, authored.
The discipline that would have broken the trick existed too, in writing, before the trigger ever shipped, and went unenforced for the kind of project that needed it most. The 2021 United States executive order on cybersecurity, issued after SolarWinds, listed among the baseline practices of secure development the maintenance of “provenance (i.e., origin) of software code or components” (Executive Office of the President 2021, sec. 4(e)(vi)), which is precisely the property the tarball broke: the shipped artifact no longer matched the source anyone could inspect. The companion framework from the National Institute of Standards and Technology was almost uncanny read against the incident. One of its examples instructed developers that “if the integrity or provenance of acquired binaries cannot be confirmed, build binaries from source code after verifying the source code’s integrity and provenance” (National Institute of Standards and Technology 2022, 12, PW.4.1, Example 8). Build from verified source rather than trust an opaque artifact: the named, published control that the concealment defeated. The framework had not failed: the control was real and it was correct. The economic and social conditions that would have applied it to a one-volunteer compression library simply did not exist, a problem the final chapters return to.
What the artifact establishes, in the end, the formal record states without hedging. The catalog assigned the incident the weakness class CWE-506, “Embedded Malicious Code” (CVE Program 2024), and the precision matters. That category is not a flaw class like a buffer overflow or a memory error, the residue of human mistake. It is the security community’s own label for code that was deliberately placed to do harm. The formal record therefore agrees, in the field’s own taxonomy, that the XZ Utils backdoor was an authored operation and not an accident. An authored operation implies an author, and the author is still unnamed.
That someone was, the evidence shows, a person at a keyboard, debugging his own malware in real time, which is the detail that most complicates the admiration. The payload broke things, the operator fixed them, and the fixing left fingerprints. Between the two poisoned releases the operator worked on his own code under the cover of ordinary maintenance, and in one of the most revealing lines in the entire record, Collin recognized the cover for what it was: a routine review comment of his own, about the dictionary size in the RISC-V test files, had been turned into an alibi. “I had mentioned the dictionary size,” Collin wrote, “and that gave a good excuse to update something in the backdoor code” (Collin 2024b). Welcomed, well-meant feedback had become operational cover. The figure that remains is not a genius in the abstract but a developer maintaining a flawed implant in public, who used at least one ordinary review comment, in Collin’s retrospective reading, as cover for another backdoor change. That figure is harder to romanticize, and closer to the truth.
The craft sits in a lineage, and the lineage is old. Clifford Stoll built his 1989 account around the cuckoo (Stoll 1989), the nesting parasite that leaves its egg in another bird’s nest to be hatched and raised by it: hostile code smuggled into a trusted host that then feeds it and grants it privileges. The metaphor is thirty-five years old and fits exactly. The nearer comparison for the sheer target-awareness is Stuxnet, the centrifuge sabotage Kim Zetter reconstructed, a payload that hunted for one precise industrial configuration and ignored everything else (Zetter 2014), with one instructive difference. Stuxnet was built to destroy; this was built for access, a passive door that did nothing until the operator chose to open it. And the design’s deepest intention is captured by a maxim recorded by Ben Buchanan and credited to the cryptographer Matthew Green, speaking of a different case: “the best backdoor is a backdoor that looks like a bug” (Buchanan 2020). A backdoor engineered to remain attributable to accident keeps its author hidden even after its mechanism is fully understood. The mechanism of this one is now understood in extraordinary detail, reconstructed down to the 456 scattered instructions. The author is not. Who that author was, the world has spent the years since trying to establish, and failing.
The absence is verifiable directly. As of this writing the malicious
build-to-host.m4appears nowhere in thetukaani-project/xzhistory: no commit touches the file, and a request for its contents in the repository returns a “not found” error. The injected macro existed only in the release tarballs “Jia Tan” cut and signed, which is the seam the next chapter follows downstream from upstream release into distribution packaging.↩︎