What does a Microsoft researcher do in the evening at home, when he has some free time? Andres Freund, this is the name of the researcher, is a modern hero and has foiled what is perhaps one of the largest and most ambitious plans to spread devastating malware. Freund was trying to optimize his computer’s performance by reducing the fan speed to make it quieter when he noticed something was wrong. “I didn’t notice this when logging in via ssh. I was doing some micro-benchmarking and was trying to make the system quieter when I noticed that the SSHD processes were using a surprising amount of CPU, despite immediately failing due to incorrect usernames. I profiled SSHD and saw that it reconnected as a DSO to Liblzma. I became suspicious. Then I remembered seeing a strange complaint from Valgrind on my automated Postgres tests, a few weeks earlier after installing some package updates. Too many coincidences.

Freund himself wrote this sentence in a newsletter, and just to make this article readable by everyone we clarify some terms that may be incomprehensible. SSH, or Secure Shell, is a system that allows secure access to a remote computer and SSHD is the component on the remote computer that allows access. Compromising the SSHD means compromising the lock that prevents anyone from accessing a server. Freund noticed that following incorrect login attempts, i.e. in the absence of an established session, SSHD continued to put pressure on the computer’s CPU, even though it shouldn’t have done so, and upon further analysis he discovered that this was connected in some way to a DSO, Dynamic Shared Object, a shared library loaded dynamically during the execution of the server. The meaningless relationship between a library that should be used for something else and a remote access server set off the alarm bells and thus led to the discovery of a backdoor inserted right into the library that connected to the SSHD. This backdoor would have allowed someone, not yet identified, to access a huge, almost impossible to quantify number of servers and systems.

The most interesting part of the whole story is not so much the presence of malware in a key component, but how it got there. It all starts from a compromised library, XZ Utils, an opensource utility created by developer Lasse Collin and integrated into most Linux distributions. XZ Utils has been used for years without problems for data compression on servers, and like any open source library its evolution and the closure of any bugs is in the hands of a person, often the creator but not always, who makes his creation evolve and accepts contributions from outside, verifying that they do not create problems and that they are safe. Open source code, by its very nature, lives with contributions from the community and being open it can also be viewed by everyone, so it is really difficult to hide any backdoors. So how was it possible that XZ Utils was compromised? In 2020, among the users who collaborated on the project as a contributor, a certain Jia Tan, username on GitHub JiaT75 (https://github.com/JiaT75), stood out. The arrival of JiaT75 was providential because reports of bugs or improvements that could be implemented continued to accumulate but Collin, the developer who had created XZ Utils, no longer seemed to have much time to deal with them. It was Jia Tan, little by little, who made all the improvements to the library, and fixed the bugs and responded to users who asked for these fixes, and month after month his role became more and more important.

At a certain point, Collin replied to a user who complained about the lack of updates: “I haven’t lost interest in XZ Utils, but I no longer have much time to deal with this project, mainly due to health problems but also because some other things. I’ve been working a bit with Jia Tan recently on XZ Utils and the things that need to be done and maybe Jia will have a bigger role in the future, we’ll see.” XZ Utils was a free library used by practically every Linux distribution, and Collin worked on XZ Utils for free in his spare time: at some point he had to take a break and Jia Tan practically became a “co-maintainer”. In his new role Jia Tan hid the malware inside XZ Utils: he didn’t do it in the code published on freely accessible Github, but put it exclusively inside the “tarballs”, i.e. the already archived file of the library. Tarballs are often used to distribute the source code of a software so that users can download, compile and install it on their system so the same version of the library that appeared harmless on the GitHub repository was hiding the malware inside the tarball. He hid it from everyone’s eyes. To get a broader picture of the situation it is now necessary to look at everything from another angle: almost all the users who asked for improvements to XZ Utils did not exist, they were all users created at different times using anonymous Proton accounts. In the months following the insertion of the backdoor into malware. In practice they were pushing to distribute and spread the backdoor. An incredible work of social engineering based on deception and which required years of preparation time, essential to first secure Collin’s trust and then that of the distribution managers.

Nobody has any idea who is behind it, whether a group or a government body, what is certain is that first an objective was sought that could be present in every system, and XZ Utils managed by a poor programmer in difficulty because he was alone a perfect target, then the main work began. If Andres Freund had not noticed this, or if the backdoor had had less impact on system resources so as not to attract suspicion, in a few months the backdoor would have been inserted into most operating systems, and in a few years it would have spread to oil stain. With consequences that could even be catastrophic. Andres Freund is a hero.