2 Blood in the Water
Freund had decided the half-second was worth chasing. That decision is the cheapest part of the story and the part on which everything else depends. He had a login path doing work nothing had asked of it, a profiler pointing at a place that made no sense, and a benchmark he had actually sat down to run. He could have filed the anomaly, suppressed the warning, and gone back to measuring database performance. Instead he stayed with it, and the staying took days.
It did not feel like a breakthrough while it was happening. It felt like one strange reading leading to another. “Initially, I didn’t think it was, like, a backdoor or anything,” Freund said afterward. “It was like a weird symptom. . . . but the more time I spent the weirder it got” (2024). That sentence is the chapter in miniature. There was no moment of revelation and no single clue that arrived wearing the answer. There was a symptom that refused to resolve and an engineer who refused to stop pulling on it.
The work fit itself into an ordinary week, which is part of why the legend leaves it out. The first day he gave over to the anomaly almost entirely: he had, as it happened, a day with no meetings on the calendar, and he spent most of it inside the problem. The next day was not his to spend. “I remember sitting in a bunch of meetings,” he said, “and, like, not really being able to concentrate because it feels like . . . I should, I need to continue looking into this” (2024c, 28:41). The person who would catch one of the most patient supply-chain attacks ever found in open-source software was a Microsoft engineer paid to work in open source, and the chase still had the texture of ordinary work: a home machine, a calendar, and meetings through which the anomaly kept pulling at his attention.
He has also been candid that the closer the investigation came to its end, the less precisely he can reconstruct it. The strangeness built slowly at first, “gradually increasing,” and then, once he understood what he was looking at, the experience compressed into something he can no longer time. “After that, once I figured out that it was nefarious,” he said, “such a whirlwind, I kinda don’t have the timing . . . down anymore” (2024c, 25:59). The chronology that follows is his, assembled in retrospect, and he is the first to say its final hours blur. What can be fixed is the shape: a couple of hours to be sure something was genuinely wrong, then, by his own estimate, a few days from that certainty to the warning he would send the world (2024c, 27:36).
The method underneath those days is worth slowing down on, because it is the opposite of the thriller version of discovery. What Freund did, over and over, was profile, record, and refuse the easy explanation. Profiling means running a program under an instrument that samples it thousands of times a second and reports where its time is actually going, rather than where a programmer assumes it goes. A slow login, by itself, only says that something took too long. A profile begins to say where the time went.
“To analyze I primarily used perf record -e intel_pt//ub,” he wrote, naming perf, the standard Linux profiler, driven by Intel Processor Trace (intel_pt), a feature built into the processor that can record the path that execution takes through a program (Freund 2024b). The purpose of the setup was “to observe where execution diverges between the backdoor being active and not,” and where the profile localized that divergence he switched to gdb, a debugger that can freeze a running program in place, “setting breakpoints before the divergence” and stepping through what came next (Freund 2024b). The pairing matters. perf could tell him where ordinary expectations and actual execution began to part company. gdb let him stop at the edge of that parting and walk forward instruction by instruction.
That description, written after the discovery, makes the comparison sound cleaner than it could have felt in the middle of the hunt. A profiler is most useful when the investigator can produce two runs that differ in one important way: the strange behavior appears in one and disappears in the other. Without that contrast, a profile can become a map of ordinary machine noise. Freund’s task was therefore not only to inspect the slow path, but to make the slow path reliably come and go. Every condition he isolated (the way sshd was launched, the build flags in use, the version on the machine) mattered because it sharpened the comparison.
A tool like that does not answer the question for you. It gives you coordinates. The investigator still has to decide whether a hot path is normal, whether a missing name is just a build artifact, whether one run differs from another because the machine was noisy or because the software actually changed. None of this is exotic to a systems engineer, and all of it is specialized, which is exactly the point the previous chapter insisted on: Freund was not a random user tripping over a random delay. Catching the thing required someone who already lived inside these tools.
In a compressed version of the sequence, written while the disclosure was still new, Freund described the work as a chain rather than a leap. He had quieted the machine for micro-benchmarking, saw sshd processes using surprising CPU even though the attempted logins were failing immediately, profiled sshd, found time in liblzma that perf could not attach to a symbol, and remembered an odd valgrind complaint from PostgreSQL testing weeks earlier (Freund 2024d, post 112180406142695845). valgrind is a diagnostic tool that watches a running program’s memory behavior and complains when the program does something that looks invalid. That is the investigation’s real grammar: not “I saw the answer,” but “I saw a symptom, and it connected to another symptom I had almost dismissed.”
The rhythm of the work was repetition, and most of it was negative. A profile would point somewhere; Freund would form the most boring hypothesis that fit, a cache that had not warmed, a scheduling quirk, an artifact of measurement, and then set about killing it. When it died he formed the next one. This is the same instrument-tuning discipline that had made him notice the anomaly to begin with, now turned from a benchmark onto a target: run the operation, watch where the time actually goes, compare it against what the code claims to be doing, and follow the gap. Most of what an investigation like this produces is dead ends, and the dead ends are not failures. They are the mechanism. The space of innocent explanations is emptied one boring possibility at a time, until the only one left standing is the one nobody wants to be true. What separates the person who finds a thing like this from the thousands who run the same software untroubled is rarely a flash of brilliance. It is the willingness to spend several days subtracting comfortable answers.
The cast of mind is older than the tools, and it has a literature. Decades before any of this, the astronomer Clifford Stoll chased a different intruder through the precursors of the internet using nothing more exotic than the same disposition, and he set it down in terms that fit Freund precisely. “Well, I’d invested a couple hours in trying to understand a trivial problem,” Stoll wrote of the moment his own hunt nearly ended. “I got stubborn: dammit, I’d stay there till midnight, if I had to” (1989). Stubbornness, not genius, is the through-line. The work itself Stoll described as a kind of astronomy, in which “we passively observed a phenomenon, and from a few clues tried to explain the event and find the location of the source” (1989). That is the posture Freund was in: a handful of sparse, intermittent readings on one quiet machine, and the slow business of reasoning backward from them to a cause he could not yet see.
The modern equivalent of Stoll’s logbook was mundane. Freund had, he noted with some relief, “a shell history of what I had looked at,” a literal record of the commands he had run, which let him retrace his own path through the investigation without trusting it to memory (2024c, 27:17). Stoll had kept a logbook for the same reason, and offered the astronomer’s rule of thumb behind it: “if you don’t write it down, it didn’t happen” (1989). Freund, without setting out to, had written it down.
The first solid clue was the one that had drawn his eye to begin with: sshd, the program that answers remote logins, was spending processor time it had no business spending. The attempted logins he was watching were not even succeeding. They were failing quickly, as wrong-user attempts should, and still the process was burning time. The previous chapter is named for that cost, seen from one direction as the extra half-second a login took. The new work was localizing it. The profiler did not merely report that sshd was busy; it reported where inside the work the cost was pooling.
Beneath the login machinery, the time was collecting under a routine: in Freund’s own description of the profile, the “high cpu in sshd” sat “below _get_cpuid()” (2024d, post 112187000944648334). The name is worth holding onto. It returns much later in this book, when the question turns from how the backdoor was caught to how it was built. A security researcher who walked an audience through the discovery months afterward paused on exactly this detail and told them to “keep that information in mind, because it will be interesting and important for the next part” (Roccia 2024, 5:35). For now it was simply a coordinate: the expensive thing was happening here, in a place a login had no reason to go.
The symptom was not Freund’s alone, which matters for a chapter built so heavily on one person’s account. Lasse Collin, who had maintained xz for years, would later write, in the commit that tore the backdoor out, that on machines exposed to the ordinary background noise of the internet the cost was visible in aggregate: “the backdoor noticeably increased CPU load, resulting in degraded user experience and thus overwhelmingly negative user feedback” (Collin 2024). The two observations came from opposite ends of the same fact. Freund saw a single login crawl on an idle benchmarking machine; the maintainer was describing server-wide load on busy public hosts being rattled all day by automated bots. One cause, two manifestations, and the symptom stands corroborated from a channel entirely separate from the discoverer’s own retrospective telling.
The second clue was stranger than the first, and harder to explain away. The processor time was not only in the wrong place; it was in a place that, by the rules of the software, should not have existed. When Freund followed the cost down into liblzma, the compression library that sshd had no obvious reason to be running at all, the profiler reported instruction after instruction executing in code it could not name. There was, in his flat description, “no symbol” (2024a).
In compiled open-source software, the human-readable names of functions, called symbols, are how a profiler can tell you what a program is doing; install the debugging information and the names appear. A missing symbol is not automatically sinister. Programs can be stripped of names, generated code can be hard to label, and optimized binaries can frustrate the clean map a reader imagines. The clue was the combination: unexplained time, inside a compression library, during a login, in code the normal naming machinery could not identify. Code with no useful symbol was code running where the machinery of transparency had gone dark at exactly the point Freund needed light.
Freund could see what the cost was even before he could see why. “If the above decides to continue, the code appears to be parsing the symbol tables in memory,” he wrote. “This is the quite slow step that made me look into the issue” (Freund 2024b). The hedge is his own: appears to, the language of contemporaneous observation rather than finished analysis, and he kept that discipline throughout. The expensive thing, the half-second the book is named for, was a program laboriously reading through internal name-tables as it started up, before it had done any of the work a login actually requires. A compression library, at the moment a login began, was rummaging through the symbol tables of the program that had loaded it. What a library was doing there, Freund could not yet say.
The third clue was the one that nearly sent him the wrong way, and it is the clue the chapter’s caution rests on. The anomaly was intermittent. It would not reproduce on command, and a symptom that comes and goes is an investigator’s worst material and an attacker’s best protection. For a stretch, in his own retrospective phrase, “it didn’t reproduce” at all (2024a). The breakthrough, when it came, was a condition rather than a cause. The strangeness appeared only when sshd was started a particular way.
“It got a lot easier after I figured out that I could reproduce this without a running sshd,” Freund recounted. “At some point, I forked my investigation to see why I couldn’t reproduce a problem . . . in running sshd from a console. Then everything was normal. Only when I started it . . . from systemd . . . it was also clearly suspicious” (2024c, 25:12). Here a second piece of plumbing enters the story: systemd, the program that starts and supervises most of the services on a modern Linux system. A server does not normally start its login daemon because a person types the command into a terminal and watches it run. It starts it through a service manager, under a defined environment, with the dependencies and startup path a distribution has arranged for real machines.
That qualification matters because the clue was narrower than “started by systemd means malicious.” Services can behave differently under a supervisor for perfectly ordinary reasons. The suspicious part was that the expensive, unnamed work inside liblzma appeared on the real service-launch path and disappeared on the simplified path Freund used for comparison. Launched by hand from a terminal, sshd behaved. Launched by systemd, the way it actually runs on many servers, it did not. For an investigator, that was valuable twice over: it made the phenomenon reproducible, and it said the behavior depended on context a compression library had no obvious reason to care about.
The other noise pointed the same way. The injected code, Freund found, “caused valgrind errors and crashes in some configurations,” the backdoor’s own assumptions colliding with an environment it had not anticipated and throwing off diagnostic complaints that a quieter system would never have produced (Freund 2024b). The complaint mattered because it was not a signature. No tool recognized the thing as a backdoor. The tool simply objected to memory behavior that looked wrong. The meaning came from the human being who remembered the complaint, connected it to the high CPU use, and kept testing after the first clean run told him nothing was there.
One of the small accidents inside that complaint was technical enough to be easy to lose and important enough not to flatten. Freund later wrote that, as far as he could tell, valgrind would not have complained without -fno-omit-frame-pointer, a compiler setting that preserves a more regular record of function calls on the stack. The hidden code around _get_cpuid() expected that stack frame to look a certain way. With Freund’s build, it did not. “Without having seen the odd complaints in valgrind,” he wrote, “I don’t think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid()” (2024d, post 112187000944648334). The point, for now, is not the full stack-layout mechanics. It is that the attacker’s assumptions met an investigator’s particular debugging environment and made noise. The backdoor did not announce itself. It misbehaved.
This is the seam where the investigation’s discipline shows, and it is worth marking what Freund did not yet claim. Stoll, the older detective, had a rule for exactly this passage of a hunt: “Record your observations. Apply physical principles. Speculate, but only trust proven conclusions” (1989). Freund speculated, as anyone would, and trusted little of it until it held. He had a slow step with no symbol, a cost that depended on systemd, a scatter of valgrind complaints, and a growing certainty that none of it was innocent. What he did not have yet was the shape of the thing. The full anatomy of the payload, what it intercepted and what access it granted, was not visible from where he stood, and he did not pretend otherwise. That anatomy, and the question of what the thing would have done at scale had no one caught it, are taken up in later chapters; from where Freund sat, both were still ahead of him. In the moment he had a set of symptoms and a hardening suspicion, and the suspicion still had to be turned into a proof.
Proof, in this setting, meant something colder than suspicion. A slow login could be argued with. A profile could be disputed as a local build problem, a strange machine, a measurement artifact, a distribution quirk. Even a crash under a diagnostic tool could remain a bug until someone tied it to a mechanism. To warn the open-source security world that a trusted upstream project had been backdoored, Freund needed evidence another maintainer or responder could inspect without living inside his week of frustration. He needed the trail to leave the realm of “this machine is behaving strangely” and enter the realm of files, bytes, and reproducible differences.
The proof began with a wrong turn, which the honest version of the story keeps. For a while the evidence pointed at the wrong culprit. Part of what sent him there was the intermittency itself: when a build of the upstream source did not always reproduce the behavior, the natural suspicion was that the problem lived not in the project’s own code but in something Freund’s particular Debian system did to it on the way to becoming an installed program. “At first I thought this was a compromise of debian’s package,” he wrote, “but it turns out to be upstream” (Freund 2024b).
The distinction is the entire hinge of the catch, and it needs unpacking for anyone who has never had reason to think about how software arrives on a computer. Debian is not the author of xz. A distribution such as Debian takes upstream software, applies its own packaging rules and build settings, and turns the upstream release into installable packages for its users. If the problem had lived only in Debian’s packaging, the failure would have been grave but bounded to that downstream path. If it lived upstream, in the official material published by the project itself, every distribution that trusted and built from that material had to be treated as a possible next link.
When an open-source project publishes a new version, it does not simply hand users the live, public history of its source code, the working text that developers read and edit. That history lives in a git repository, a versioned record of every change, hosted on GitHub for anyone to inspect. What the project often ships as its release is a tarball: a single packaged archive, a snapshot bundled for download and then built into the finished program by a distribution, a package builder, or a user. The two are supposed to correspond. A person who audits the public git source is supposed to be looking at the same code the tarball will build. Freund’s investigation turned on the discovery that, for xz, they did not correspond at all.
The malicious step lived in the tarball and nowhere else. “One portion of the backdoor is solely in the distributed tarballs,” he found (Freund 2024b). He traced it to a single line in a build file named build-to-host.m4, one of the small scripts that prepare a program to be compiled. “That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git,” he wrote. “However, it is present in the tarballs released upstream, except for the ‘source code’ links, which I think github generates directly from the repository contents” (Freund 2024b).
Read slowly, that one sentence is the whole concealment in a single breath: the trigger was absent from the public source, absent from the path the project actually built from in the git repository, present in the tarballs the world downloaded, and absent again from the archives GitHub generates automatically from the repository. This was not a crude mismatch anyone would necessarily see by opening the front page of the project. It was a mismatch between artifacts that most users reasonably treat as equivalent: the visible source history and the official release archive. The operator had not needed to hide the whole attack from public view. The operator needed only to split it across places that almost no one checked against each other.
The choice of a build file was part of what made the split work. Build systems are the backstage of software: the scripts and macros that turn source text into the executable files a machine can run. They are essential, but they are not where most readers of source code linger, and in many projects they are partly generated, partly inherited, and partly treated as plumbing. A small extra line in a file like build-to-host.m4 did not look like a new feature, a new algorithm, or an obvious change to the compression library’s public behavior. It looked like release machinery. That was why the tarball mattered. The malicious code did not need to live in the place where everyone expected to review the program. It could live in the machinery that prepared the program to exist.
The implication takes a moment to absorb, and it is the reason the discovery reached past one machine. The promise of open-source software is that anyone can read it: the source is in the open, and enough eyes on it will catch what is wrong. The attack had been built to defeat exactly that promise without seeming to. The trigger was missing from the repository those eyes would read. The payload files were there, but disguised as test material and inert without the tarball-only step that activated them. The poison sat across the boundary between source and package, on the reasonable assumption that the two matched. Freund had not found a bug in xz. He had found that the relationship between what xz showed the world and what it shipped to the world had been quietly severed, and that the severing was the attack. And that is where the real attack surface of open-source infrastructure turns out to lie: not only in code that too few people read, but in trust that too few people can check.
He could watch it happen as the program built. “The build system would end up building the same file multiple times and replacing it with different content,” Freund said, and when “there is like suddenly an object file that was like 50 KB or something larger than before, then that was clear that there was something bad going on” (2024). A build is allowed to be complicated. It may generate files, compile the same source in different modes, produce intermediate objects, or carry around test data that looks strange to a reader who does not maintain the project. The clue was not strangeness in the abstract. It was the same file being rebuilt and replaced with different contents, and an object file abruptly tens of kilobytes heavier than it had any reason to be.
At that point the investigation changed character. Before, Freund had been trying to account for symptoms: CPU time, missing symbols, launch conditions, valgrind complaints. Now he had a build process producing different bytes. The rebuild that sometimes produced different output was, in his words, the point where there was no longer a “legitimate reason” left to reach for (2024a). The discipline the previous chapter described, suspecting the dull cause first and chasing it until it either accounts for the anomaly or runs out, had finally driven every dull cause out of the room.
The payload itself, it turned out, had been hiding in plain sight the whole time. The bulk of the malicious code, Freund found, was already sitting in the public repository, disguised as test data: “in an obfuscated form in tests/files/bad-3-corrupt_lzma2.xz” and tests/files/good-large_compressed.lzma, files committed upstream like any other (Freund 2024b). The disguise was plausible because a compression library’s test suite is expected to contain malformed and unusual samples. A file with a name like bad-3-corrupt_lzma2.xz announces itself as junk on purpose; its job, in an innocent test suite, would be to prove that the decompressor fails safely on bad input. That is what made it useful cover.
The trigger that armed those files was the line that lived only in the tarball. Keep the trigger out, as it had been kept out of the git repository, and the test files were inert. Include the trigger in the release archive, and the build could extract and assemble what the repository alone did not appear to contain. This division of the attack, the payload committed in the open and the trigger slipped into the package, is the operator’s craft, and the craft is the subject of a later chapter. Here it is enough that Freund had found the seam between the two and understood what it meant.
That finding did not require him to finish reverse-engineering the payload. It required him to show that the official release artifact had been made to do something the public source history did not explain. In a normal bug report, the difference between symptom and cause can remain partly open. In a backdoor disclosure, the difference is existential: if the official tarball differs maliciously from the git source, the trust decision has already changed, even before every instruction in the payload has been named. Freund had crossed that threshold.
What he refused to claim is as telling as what he found. In the same message that laid out the clues, he was scrupulous about the limits of his own account. “I am not a security researcher, nor a reverse engineer,” he wrote. “There’s lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code” (Freund 2024b). By his own framing this was a report of symptoms and one decisive structural finding, not a complete dissection. The full account of how the payload worked, who could use it and to do what, would be assembled over the following days and weeks by people whose actual profession was reverse engineering. The same man who had begun by doubting it was a backdoor at all now had to be the one to write the sentence that said it was. The restraint is not false modesty; it is the same disposition that produced the catch. A refusal to claim more than the evidence proved is precisely what let him recognize, in the end, that the evidence proved something extraordinary.
By the time he was ready to say it, the saying was plain. On March 29, 2024, Freund sent a message to oss-security, the public mailing list where the open-source world reports its security problems, and the message opened without preamble: “The upstream xz repository and the xz tarballs have been backdoored” (Freund 2024b). He had not gone straight to the public list. In the timeline Russ Cox later assembled from the record, Freund first raised the alarm privately on March 28, the day before the public post, to the handful of people who would need to begin moving at once (Cox 2024). The private warning bought a few hours; the public one started everyone else’s clock. What is striking about the disclosure sentence is how little it strains. It reaches for no drama at all. An engineer who had spent days proving that trusted code could no longer be trusted announced it in eleven words and a period. The drama was in the fact, not in the phrasing.
The discovery has since been reconstructed by others, which is its own small check on the shape of the story. A 2025 MSR paper analyzing the operation describes the same opening Freund lived: an investigator who, “aided by a stroke of luck,” began “with his observation of anomalous behavior: unusually high CPU usage by sshd processes during SSH logins,” and followed it to the attack (Przymus and Durieux 2025, 93). The paper is secondary to Freund’s own accounts for the discovery sequence, but its outline agrees with the first-person record on the shape, the luck included. The only thing it places differently is the date. The paper dates the discovery to March 28, the day Freund first raised it privately; the book dates the disclosure to March 29, the day the warning went public and the rest of the world’s clocks began to run. The two are not in conflict. They are the two halves of the same hinge.
When Freund pressed send on the oss-security post, the investigation stopped being his. The half-second that had been a private irritation on a home benchmark was now a public fact, and the question it opened, what exactly had almost happened and how close it had come, belonged from that moment to everyone with a Linux system to worry about, which was very nearly everyone. Answering it would take the world the next forty-eight hours even to begin.