arstechnica.com Open in urlscan Pro
3.132.188.58  Public Scan

Form analysis
1 forms found in the DOM

GET /search/

<form action="/search/" method="GET" id="search_form">
  <input type="hidden" name="ie" value="UTF-8">
  <input type="text" name="q" id="hdr_search_input" value="" aria-label="Search..." placeholder="Search...">
</form>

Text Content

Skip to main content
 * Biz & IT
 * Tech
 * Science
 * Policy
 * Cars
 * Gaming & Culture
 * Store
 * Forums

Subscribe

Close


NAVIGATE

 * Store
 * Subscribe
 * Videos
 * Features
 * Reviews

 * RSS Feeds
 * Mobile Site

 * About Ars
 * Staff Directory
 * Contact Us

 * Advertise with Ars
 * Reprints


FILTER BY TOPIC

 * Biz & IT
 * Tech
 * Science
 * Policy
 * Cars
 * Gaming & Culture
 * Store
 * Forums


SETTINGS

Front page layout


Grid


List


Site theme

light

dark

Sign in

NIGHTMARE SUPPLY CHAIN ATTACK SCENARIO —


WHAT WE KNOW ABOUT THE XZ UTILS BACKDOOR THAT ALMOST INFECTED THE WORLD


MALICIOUS UPDATES MADE TO A UBIQUITOUS TOOL WERE A FEW WEEKS AWAY FROM GOING
MAINSTREAM.

Dan Goodin - 4/1/2024, 8:55 AM

Enlarge
Getty Images

READER COMMENTS

207

On Friday, a lone Microsoft developer rocked the world when he revealed a
backdoor had been intentionally planted in xz Utils, an open source data
compression utility available on almost all installations of Linux and other
Unix-like operating systems. The person or people behind this project likely
spent years on it. They were likely very close to seeing the backdoor update
merged into Debian and Red Hat, the two biggest distributions of Linux, when an
eagle-eyed software developer spotted something fishy.




FURTHER READING

Backdoor found in widely used Linux utility targets encrypted SSH connections
"This might be the best executed supply chain attack we've seen described in the
open, and it's a nightmare scenario: malicious, competent, authorized upstream
in a widely used library," software and cryptography engineer Filippo Valsorda
said of the effort, which came frightfully close to succeeding.

Researchers have spent the weekend gathering clues. Here's what we know so far.

What is xz Utils?

xz Utils is nearly ubiquitous in Linux. It provides lossless data compression on
virtually all Unix-like operating systems, including Linux. xz Utils provides
critical functions for compressing and decompressing data during all kinds of
operations. xz Utils also supports the legacy .lzma format, making this
component even more crucial.

What happened?

Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL
offerings, was recently troubleshooting performance problems a Debian system was
experiencing with SSH, the most widely used protocol for remotely logging in to
devices over the Internet. Specifically, SSH logins were consuming too many CPU
cycles and were generating errors with valgrind, a utility for monitoring
computer memory.

Advertisement


Through sheer luck and Freund’s careful eye, he eventually discovered the
problems were the result of updates that had been made to xz Utils. On Friday,
Freund took to the Open Source Security List to disclose the updates were the
result of someone intentionally planting a backdoor in the compression software.

It's hard to overstate the complexity of the social engineering and the inner
workings of the backdoor. Thomas Roccia, a researcher at Microsoft, published a
graphic on Mastodon that helps visualize the sprawling extent of the nearly
successful endeavor to spread a backdoor with a reach that would have dwarfed
the SolarWinds event from 2020.

Enlarge

What does the backdoor do?

Malicious code added to xz Utils versions 5.6.0 and 5.6.1 modified the way the
software functions. The backdoor manipulated sshd, the executable file used to
make remote SSH connections. Anyone in possession of a predetermined encryption
key could stash any code of their choice in an SSH login certificate, upload it,
and execute it on the backdoored device. No one has actually seen code uploaded,
so it's not known what code the attacker planned to run. In theory, the code
could allow for just about anything, including stealing encryption keys or
installing malware.

Wait, how can a compression utility manipulate a process as security sensitive
as SSH?

Any library can tamper with the inner workings of any executable it is linked
against. Often, the developer of the executable will establish a link to a
library that's needed for it to work properly. OpenSSH, the most popular sshd
implementation, doesn’t link the liblzma library, but Debian and many other
Linux distributions add a patch to link sshd to systemd, a program that loads a
variety of services during the system bootup. Systemd, in turn, links to
liblzma, and this allows xz Utils to exert control over sshd.

Advertisement


How did this backdoor come to be?

It would appear that this backdoor was years in the making. In 2021, someone
with the username JiaT75 made their first known commit to an open source
project. In retrospect, the change to the libarchive project is suspicious,
because it replaced the safe_fprint funcion with a variant that has long been
recognized as less secure. No one noticed at the time.

The following year, JiaT75 submitted a patch over the xz Utils mailing list,
and, almost immediately, a never-before-seen participant named Jigar Kumar
joined the discussion and argued that Lasse Collin, the longtime maintainer of
xz Utils, hadn’t been updating the software often or fast enough. Kumar, with
the support of Dennis Ens and several other people who had never had a presence
on the list, pressured Collin to bring on an additional developer to maintain
the project.

In January 2023, JiaT75 made their first commit to xz Utils. In the months
following, JiaT75, who used the name Jia Tan, became increasingly involved in xz
Utils affairs. For instance, Tan replaced Collins' contact information with
their own on oss-fuzz, a project that scans open source software for
vulnerabilities that can be exploited. Tan also requested that oss-fuzz disable
the ifunc function during testing, a change that prevented it from detecting the
malicious changes Tan would soon make to xz Utils.

In February of this year, Tan issued commits for versions 5.6.0 and 5.6.1 of xz
Utils. The updates implemented the backdoor. In the following weeks, Tan or
others appealed to developers of Ubuntu, Red Hat, and Debian to merge the
updates into their OSes. Eventually, one of the two updates made its way into
the following releases, according to security firm Tenable:

Distribution Advisory Notes Fedora Rawhide
https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users
Fedora Rawhide is the development distribution of Fedora Linux Fedora 41
https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users
Debian testing, unstable and experimental distributions versions 5.5.1alpha-0.1
to 5.6.1-1. https://lists.debian.org/debian-security-announce/2024/msg00057.html
openSUSE Tumbleweed and openSUSE MicroOS
https://news.opensuse.org/2024/03/29/xz-backdoor/ Backdoored version of xz was
included in Tumbleweed and MicroOS between March 7 and March 28 Kali Linux
https://www.kali.org/blog/about-the-xz-backdoor/ Backdoored version of xz was
included in Kali Linux (xz-utils 5.6.0-0.2) between March 26 and March 28

There’s more about Tan and the timeline here.

Page: 1 2 Next →


ARS VIDEO


HOW THE CALLISTO PROTOCOL'S TEAM DESIGNED ITS TERRIFYING, IMMERSIVE AUDIO





READER COMMENTS

207
Dan Goodin Dan Goodin is Senior Security Editor at Ars Technica, where he
oversees coverage of malware, computer espionage, botnets, hardware hacking,
encryption, and passwords. In his spare time, he enjoys gardening, cooking, and
following the independent music scene.

Advertisement



PROMOTED COMMENTS

Christarp
The whole world got lucky that one developer was determined enough to discover
the cause of a minor performance regression. Just makes you wonder what else
hasn't yet been discovered in our open source tooling out there.

Really eye opening, thank you Andres!

Edit: And of course, Dan for the wonderful article.
April 1, 2024 at 7:16 am
BarnSoftEng

> It should be noted that the attack only works because Debian and Redhat added
> functionality to sshd that is not present in it as distributed by its
> developers. The extra functionality adds systemd interaction, which requires
> libsystemd which requires liblzma, a component of the (compromised) xz
> package.
> 
> One should be wary of distributions adding functionality. Often it increases
> the attack surface, not only because of the modifications/additions
> themselves, but also by adding dependencies.

This conclusion ignores the huge value that the patches that all OS apply to
packages provides. Without patches much upstream software will not build, will
not run correctly.
April 1, 2024 at 7:49 am
pseudonomous

> It should be noted that the attack only works because Debian and Redhat added
> functionality to sshd that is not present in it as distributed by its
> developers. The extra functionality adds systemd interaction, which requires
> libsystemd which requires liblzma, a component of the (compromised) xz
> package.
> 
> One should be wary of distributions adding functionality. Often it increases
> the attack surface, not only because of the modifications/additions
> themselves, but also by adding dependencies.


While this is true, to a degree, it's also quite possible that if there were a
lot of interesting targets out there that didn't patch sshd to use systemd and
libxz, that given the level of access they had, the actor here could almost
certainly found some other way to put an obfuscated backdoor into the code.

I mean, certainly the OpenSSH project is obviously in the clear, but having
gained commit access to a critical library, I don't think we could feel safe
that "Jia Tan" couldn't have constructed a different backdoor even if they
couldn't use the particular behavior Debian and Ubuntu added to their version of
SSH.

And I'm not sure anybody feels totally safe until somebody audits the code to
make sure there's nothing else lurking in here that might be dangerous.
April 1, 2024 at 7:50 am
BarnSoftEng

> So a prime reason this became potentially exploitable is libsystemd in
> OpenSSH. Need I say more.

The prime reason is a very well funded and capable attacker looked for a way in.
if not xz or systemd then they would have attacked via the next candidate weak
point.
April 1, 2024 at 7:51 am
om1

> The prime reason is a very well funded and capable attacker looked for a way
> in.
> if not xz or systemd then they would have attacked via the next candidate weak
> point.

I do know that the OpenSSH devs are very strict on not increasing attack
surface. The main reason why they did not want to link to systemd in the
distributed version. You can be sure that OpenSSH as distributed contains less
opportunity for attackers than the modified versions. That's my main point:
modifications by distributions are not always improvements.
April 1, 2024 at 8:20 am
MikeGale
We are lucky that this was detected and that some competent people have moved in
to analyse. I presume more analysis is still being done. (Thanks to them.)

What we don't know is how many other similar attacks have been deployed, and how
many are in preparation.

What can decent people do to reduce our risks?

Thoughts that come to mind:
1. Some of those who do this, often thankless, maintenance work might like more
support. This might be financial contributions, or people (who know what they're
doing) reviewing code submissions. Those who incorporate these libraries into
their own programs (and all users) should maybe think about this. If there were
a "donate to the maintainers" button on a story like this, that would convert
the immediate story into something of greater value, if the maintainer would
like that.
2. Some of the maintainers might appreciate recognition. Some won't, but worth
considering.
3. Some who use the libraries can improve the checking they do.
4. Unpleasant people who harass maintainers should be detected and treated
appropriately.
April 1, 2024 at 9:17 am
TheMongoose

> "This developer persona has touched dozens of other pieces of open-source
> software in the past few years.". Well, I guess the Opensource community have
> some codes to review. Maybe the xz incident is only the tips of the iceberg.

Right? I didn’t realise until I read this article just how far back this
started. Now you’ve got to wonder: if this was a group, how many other so-far
benign identities they have, contributing good code and building up reputations.
And how easy it would be for them to slip a little something into another
fundamental tool.
April 1, 2024 at 9:18 am



CHANNEL ARS TECHNICA

UNSOLVED MYSTERIES OF QUANTUM LEAP WITH DONALD P. BELLISARIO

Today "Quantum Leap" series creator Donald P. Bellisario joins Ars Technica to
answer once and for all the lingering questions we have about his enduringly
popular show. Was Dr. Sam Beckett really leaping between all those time periods
and people or did he simply imagine it all? What do people in the waiting room
do while Sam is in their bodies? What happens to Sam's loyal ally Al? 30 years
following the series finale, answers to these mysteries and more await.

 * UNSOLVED MYSTERIES OF QUANTUM LEAP WITH DONALD P. BELLISARIO

 * UNSOLVED MYSTERIES OF WARHAMMER 40K WITH AUTHOR DAN ABNETT

 * SITREP: F-16 REPLACEMENT SEARCH A SIGNAL OF F-35 FAIL?

 * SITREP: BOEING 707

 * STEVE BURKE OF GAMERSNEXUS REACTS TO THEIR TOP 1000 COMMENTS ON YOUTUBE

 * MODERN VINTAGE GAMER REACTS TO HIS TOP 1000 COMMENTS ON YOUTUBE

 * HOW THE NES CONQUERED A SKEPTICAL AMERICA IN 1985

 * SCOTT MANLEY REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * HOW HORROR WORKS IN AMNESIA: REBIRTH, SOMA AND AMNESIA: THE DARK DESCENT

 * LGR'S CLINT BASINGER REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * THE F-35'S NEXT TECH UPGRADE

 * HOW ONE GAMEPLAY DECISION CHANGED DIABLO FOREVER

 * UNSOLVED MORTAL KOMBAT MYSTERIES WITH DOMINIC CIANCIOLO FROM NETHERREALM
   STUDIOS

 * US NAVY GETS AN ITALIAN ACCENT

 * HOW AMAZON’S “UNDONE” ANIMATES DREAMS WITH ROTOSCOPING AND OIL PAINTS

 * FIGHTER PILOT BREAKS DOWN EVERY BUTTON IN AN F-15 COCKPIT

 * HOW NBA JAM BECAME A BILLION-DOLLAR SLAM DUNK

 * LINUS "TECH TIPS" SEBASTIAN REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * HOW ALAN WAKE WAS REBUILT 3 YEARS INTO DEVELOPMENT

 * HOW PRINCE OF PERSIA DEFEATED APPLE II'S MEMORY LIMITATIONS

 * HOW CRASH BANDICOOT HACKED THE ORIGINAL PLAYSTATION

 * MYST: THE CHALLENGES OF CD-ROM | WAR STORIES

 * MARKIPLIER REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * HOW MIND CONTROL SAVED ODDWORLD: ABE'S ODDYSEE

 * BIOWARE ANSWERS UNSOLVED MYSTERIES OF THE MASS EFFECT UNIVERSE

 * CIVILIZATION: IT'S GOOD TO TAKE TURNS | WAR STORIES

 * SITREP: DOD RESETS BALLISTIC MISSILE INTERCEPTOR PROGRAM

 * WARFRAME'S REBECCA FORD REVIEWS YOUR CHARACTERS

 * SUBNAUTICA: A WORLD WITHOUT GUNS | WAR STORIES

 * HOW SLAY THE SPIRE’S ORIGINAL INTERFACE ALMOST KILLED THE GAME | WAR STORIES

 * AMNESIA: THE DARK DESCENT - THE HORROR FACADE | WAR STORIES

 * COMMAND & CONQUER: TIBERIAN SUN | WAR STORIES

 * BLADE RUNNER: SKINJOBS, VOXELS, AND FUTURE NOIR | WAR STORIES

 * DEAD SPACE: THE DRAG TENTACLE | WAR STORIES

 * TEACH THE CONTROVERSY: FLAT EARTHERS

 * DELTA V: THE BURGEONING WORLD OF SMALL ROCKETS, PAUL ALLEN'S HUGE PLANE, AND
   SPACEX GETS A CRUCIAL GREEN-LIGHT

 * CHRIS HADFIELD EXPLAINS HIS 'SPACE ODDITY' VIDEO

 * THE GREATEST LEAP, EPISODE 1: RISK

 * ULTIMA ONLINE: THE VIRTUAL ECOLOGY | WAR STORIES

More videos
← Previous story Next story →


RELATED STORIES




TODAY ON ARS

 * Store
 * Subscribe
 * About Us
 * RSS Feeds
 * View Mobile Site

 * Contact Us
 * Staff
 * Advertise with us
 * Reprints


NEWSLETTER SIGNUP

Join the Ars Orbital Transmission mailing list to get weekly updates delivered
to your inbox. Sign me up →



CNMN Collection
WIRED Media Group
© 2024 Condé Nast. All rights reserved. Use of and/or registration on any
portion of this site constitutes acceptance of our User Agreement (updated
1/1/20) and Privacy Policy and Cookie Statement (updated 1/1/20) and Ars
Technica Addendum (effective 8/21/2018). Ars may earn compensation on sales from
links on this site. Read our affiliate link policy.
Your California Privacy Rights | Manage Preferences
The material on this site may not be reproduced, distributed, transmitted,
cached or otherwise used, except with the prior written permission of Condé
Nast.
Ad Choices





WE CARE ABOUT YOUR PRIVACY

We and our 167 partners store and/or access information on a device, such as
unique IDs in cookies to process personal data. You may accept or manage your
choices by clicking below or at any time in the privacy policy page. These
choices will be signaled to our partners and will not affect browsing data.More
information about your privacy


WE AND OUR PARTNERS PROCESS DATA TO PROVIDE:

Use precise geolocation data. Actively scan device characteristics for
identification. Store and/or access information on a device. Personalised
advertising and content, advertising and content measurement, audience research
and services development. List of Partners (vendors)

I Accept
Show Purposes