1 Half a Second
It was late, and Andres Freund was running a benchmark. There was nothing unusual in that. Freund is a software engineer who works on databases, one of the longtime contributors to PostgreSQL, the open-source database system that sits underneath a large share of the world’s applications, and in the last days of March 2024 he was doing what he often did in his own time: measuring how small changes affected the software’s speed, chasing the fractions of a percent that accumulate into something a user can feel.
This is unglamorous work, and it is mostly patience. You change one thing, run the same operation thousands of times, compare the numbers against a baseline, and watch for movement the change alone should not explain. Done well, it is less an experiment than the tuning of an instrument: the engineer comes to know the ordinary sound of the machine so thoroughly that a wrong note announces itself. Most wrong notes are nothing. A benchmark that runs slow has usually tripped over something dull: a cache that never warmed up, a background task stealing cycles, the processor quietly throttling itself to keep from overheating. The discipline is to suspect the boring explanation first and chase it until it either accounts for the anomaly or runs out, and almost always it accounts for it. That habit is what makes a genuine find rare, and it is also what gives one weight when it finally comes. The computer Freund was testing on ran Debian, one of the oldest and most widely used Linux distributions, in its perpetually updating development edition, the branch its users call by its code name, sid. It is the sort of machine a careful engineer keeps close at hand precisely because it runs new software before most people do, which makes it a little faster, a little more current, and a little more likely to behave strangely.
He was not hunting for anything, and that is worth holding onto. He was measuring database performance, not looking for an intruder, and what he found he found sideways. What caught his attention was not dramatic. He would later describe the discovery as a “small slowdown,” the kind of reading that sits at the very edge of measurement and is easy to wave off as noise (2024a). On a machine that should have been close to idle, the part of the system that handles remote logins was doing something. Logging in over ssh, the standard encrypted channel engineers use to reach a computer across a network, was taking longer than it should, and the program on the far end that answers those logins, the SSH daemon, sshd, was burning an unusual amount of processor time to do it (Freund 2024b). In his own telling, what he kept seeing were brief, repeating bursts in which the processor lit up and then subsided: “occasionally there would be these 500 milliseconds . . . of high CPU usage that would then go down again,” he said, “and that’s just not what I was expecting” (2024). Something was spending effort that nothing had asked it to spend.
For a while the extra processor use looked like nothing in particular. The internet is a noisy place, and any machine exposed to it spends part of every day shrugging off automated attacks rattling the doors; Freund’s first reading of the sshd activity was that it might be the “typical probing” every public server attracts (2024a). But it did not behave like probing. The effort was coming from the wrong place, at the wrong moment, for no reason he could name, and the longer he looked the less it resolved into anything routine.
Freund does not cast himself as a security researcher, and he has been careful about the distinction. Pressed on it, he has described his own background as only “security adjacent,” someone who had run production systems and handled the occasional security problem without belonging to the institutions whose actual job is to catch attacks like this one (2024a).
He timed it. A login that ought to have finished in about three-tenths of a second was taking closer to eight-tenths, a difference of roughly half a second that had no reason to be there (Freund 2024b). It was the same half-second seen from two directions: the burst of processor activity on each login, and the extra time the login itself took. Around the same moment valgrind, a diagnostic tool that watches a running program for memory errors and other misbehavior, was registering complaints from somewhere inside that login path. None of it amounted to an alarm. It was a small cluster of odd readings on one machine, the sort of thing a noisier setup might have hidden, or the same person in a busier week might have explained away and forgotten.
He checked that it was real. The slowdown appeared even when he ran sshd in a stripped-down way that did not start it as a server at all, only far enough to do its initial work; that bare invocation took about half a second where it should have taken a hundredth of one (Freund 2024b). Whatever this was, it was not a quirk of his particular configuration or a misread number. It was real, and it was repeatable.
That half-second would become the most famous detail of the whole affair, and in the retelling it hardened into legend: Freund saw the delay, the delay tipped him off, and from there he pulled the thread that unraveled everything. The real sequence is more particular, and Freund has gone out of his way to correct it. The half-second was real, he says, but it arrived second. “I think the myth goes that’s what caused me to look into all of this,” he told one interviewer. “But the 500 milliseconds is actually true. . . . But I only got there after I already had seen the CPUs. . . . So it was like a secondary investigative perspective” (2024). What set him looking was the processor use and the misbehaving logins; the timing was the instrument he reached for to pin the thing down, not the spark. A Microsoft security researcher who later recounted the episode at a conference told it the same way: the half-second was “a kind of a legend,” and what actually triggered the investigation was that Freund “found that the SSH login in some of the testing he was doing was actually failing” (Roccia 2024, 3:19). The legend is tidier than the truth, which is partly why it spread: a single number, a single instant of noticing, a clean line drawn from symptom to discovery. The real path was messier and more human, a careful person tugging at several loose threads at once until one of them held.
It matters who was at that keyboard, because the easy version of this story turns the catch into a miracle, and it was not one. Freund was not a random user tripping over a random delay. He had contributed to PostgreSQL for more than a decade and had been trusted to write changes directly into its code for years; in November 2020 the project’s core team, the small group that steers one of the world’s most relied-upon open-source databases, announced that he was joining it, noting that he had been “a contributor to PostgreSQL for over 10 years and a committer for 6,” with his work concentrated on “replication, performance & scalability, storage and patch review” (2020). There is a quiet symmetry in that. The person who would catch an attack buried in critical open-source software was himself a steward of critical open-source software, on the maintaining side of the same fragile arrangement this book is about.
Performance and scalability were not a hobby he kept on the side. They were his specialty, the cast of mind that treats an unexplained half-second not as noise to be tuned out but as a question with an answer somewhere underneath it. A person who spends his working life learning the ordinary sound of a system is exactly the person to hear a wrong note in it. The contingency of the catch is real, and much of this book rests on it, but it was the contingency of the right anomaly reaching the right person, not of a delay so loud that anyone at all would have heard it.
And even then, by his own account, it nearly did not happen. Asked afterward how he had caught it, Freund gave a six-word answer: “Really required a lot of coincidences” (2024d, post 112180406142695845). Laid out, the coincidences pile up unnervingly, and no single one of them was the catch. It took the whole stack. He happened, that week, to be doing performance work at all. In another release cycle he would have been reviewing other people’s code and tidying details rather than profiling, and, as he put it, he “might not have looked at the time” (2024c, 01:17:04).
He happened to be running a version of xz, a compression package whose liblzma library could be called by other programs, old enough to still exhibit the symptom. Had the maintainer of his Debian system already moved to the newer release, 5.6.1, which quieted the noisy valgrind complaint, he was, he says, “not sure” he would have figured it out (2024c, 01:17:25). He happened, too, to have already brushed against the same strangeness once before. Some weeks earlier, automated testing in PostgreSQL had begun throwing a valgrind error, and Freund, busy with other things, had done the reasonable thing: suppressed the warning, assumed a coming update would clear it, and moved on. When the strangeness surfaced a second time, on the benchmark machine, it landed on a mind that had already grazed it. He has said he is not sure he would have caught it on the second pass “if it hadn’t in the back of my mind already been . . . primed for there’s something really weird here” (2024). And he happened to be testing on a server processor with the chip’s turbo boost, the feature that briefly raises its clock speed under load, switched off so his measurements would stay steady. On that slower, more even core the extra work was, in his words, “really exaggerated,” drawn out into a window long enough to notice; on his faster laptop the same burst would have been briefer and easier to overlook (2024c, 01:18:17).
The strangest coincidence of all does not belong to Freund. It belongs to whoever built the thing he was beginning to circle. The operation was visible because it had been over-built. Had its concealed code done its job more cheaply, the extra effort would have cost not half a second but a few hundred microseconds, which is to say millionths of a second, far too little to surface as anything at all; with “less obfuscation,” Freund has said, he “probably would not have noticed that anything was wrong” (2024c, 30:48). The thing was caught, in part, because whoever made it had hidden it too expensively. That irony runs under the entire story, and the craft of what they built is a subject this book comes back to.
None of which makes the catch a matter of luck and nothing else. The accounts written in the first days got the balance right. One early explainer credited “sheer luck and Freund’s careful eye,” the two halves held in a single phrase (Goodin 2024). The researcher who narrated the discovery called it “a mix of luck, coincidence, and expertise,” and said that without Freund’s investigation the thing might never have been found at all (Roccia 2024, 39:43). Another security researcher, writing that same week, observed that Freund “had a unique testing environment and a set of coincidental setup issues which allowed him to discover the issue,” and doubted that anyone else was running quite the same combination (Beaumont 2024). The environment was part of it: a quiet, low-noise benchmarking machine on which a half-second anomaly had nowhere to hide. On an ordinary desktop, Freund has noted, the ambient churn of something as unremarkable as an “idle Chrome” browser could have swallowed the signal whole (2024a). The honest account is neither luck nor brilliance but the two together: a faint signal, and a person equipped, by temperament and by setup, to register it.
There is an older version of this kind of discovery, worth naming because the shape keeps recurring. In 1989 the astronomer Clifford Stoll opened his account of chasing an intruder through the precursors of today’s internet not with a break-in but with a bookkeeping error: a 75-cent shortfall in a university computer’s accounts, a discrepancy small enough that nearly anyone would have written it off, which turned out to be the visible edge of a foreign espionage operation (1989). Stoll’s own explanation of why he bothered doubles as a description of Freund’s instinct. A large error is obvious and easy to find, he wrote, but “errors in the pennies column arise from deeply buried problems” (1989). He nearly let it go all the same. “Other things seemed more pressing, so I dropped the problem,” he wrote of one of several moments when the investigation almost ended before it began (1989). What a story like that turns on is not the size of the anomaly but the choice, made or unmade on an ordinary afternoon, to treat a trivial one as worth pursuing. Freund’s half-second is a discrepancy of that same family, and so is the territory it opens onto.
Set that chain against what was at stake and it turns vertiginous. No institution caught this. No automated scanner raised an alarm, no audit surfaced it, no monitoring system fired. What prevented a working backdoor already moving through the channels that feed widely used Linux systems from remaining hidden was a run of unrelated accidents that converged, in one week, on a single engineer’s test machine. Nothing in the system had arranged a second chance. It is tempting to call that luck and leave it there. The more exact word is contingency: not that the catch was random, but that it was conditional, hung on particulars that could as easily have fallen the other way. The catch was contingent, not systemic, and that is the uncomfortable fact the rest of this book turns on.
What he had found, though it would be days before anyone could say so with any confidence, was not a bug. It was a backdoor: a deliberately built, deliberately concealed way into a system, placed there by someone who wanted quiet access to it. It had been hidden in backdoored releases of xz, the compression software package whose liblzma library sits quietly on a great many Linux systems, the systems that carry much of the traffic of the internet. Tracing it would consume the next several days and draw in distribution maintainers, security responders, and a scattered, self-assembling crowd of reverse engineers. Naming whoever had spent two years putting it there would prove, even now, impossible. All of that was still ahead. For the moment, the whole chain of consequence rested on a smaller and stranger fact: the security of a great deal of the world’s digital infrastructure had just come to hang, with no one having arranged things that way, on whether one engineer at home judged a half-second worth chasing. He judged that it was.