I think the #xz incident is teaching us that our infrastructure is dangerously fragile in the face of well-organized/funded attackers. The response isn’t “try harder” or “donate to your OSS project”, it needs to be institutional, professional, and at scale.
1/2 Looking at one of the #xz writeup, this struck my eye: “The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf.” Ah, GNU AutoHell, I remember it well. Tl;dr: With AutoHell, even if you're building for a 19-bit Multics variant from 1988, it’s got your back. Except for it’s just too hard to understand and use, thus the above.
Who should be software packaging is a tough problem, I can see the value in #linux distros pushing for better changes downstream, encouraging upstream to change (double click in #KDE) but then I see cases like KeepassXC where the Debian package is now by default broken, actively damaging the reputation of upstream but then I remember #XZ where upstream was left unchecked and hid bad code in plain sight and I go back around in a circle.
Ça fait deux jours que je suis fasciné par ce qui se passe dans le monde de la sécurité informatique, autour de la backdoor XZ. Je vais essayer de vous l'expliquer, ça va être technique, mais c'est important.
Pour Internet, c'est l'équivalent d'un gros astéroïde qui serait passé à 5000km de la Terre. Pas d'impact, pas de dégâts directs, mais on aurait pu tous y passer et personne ne l'a vu venir.
Je vais chercher à vulgariser un maximum, tout en donnant des liens vers les sources directes, qui sont souvent très techniques et en anglais. Ça va être un peu long, mais c'est passionnant.
I can't wrap my head around how almost all of the #xz reporting focuses on the failures of #opensource.
Yeah, sure, but ...
Good luck finding such an attack in proprietary code.
Via the cliché paid off/blackmailed employee, hacked dev servers/repos, or via capitalism's favorite cost-cutting measure: a remote "offshored" contracted temporary developer (or nowadays, embedded into some LLM output).
The things I don't like about the discussion on whether this is a state actor behind the #xz backdoor are:
It doesn't change the response for pretty much anyone except a narrow group of professionals. Ultimately I don't know that it matters for most of us if this was a state attacker or some kid who wants a way to get op privileges.
It distracts from next steps.
Would they think that if the actor were named John? Will this increase suspicion of anyone with a "foreign" sounding name?
Again the FOSS world has proven to be vigilant and proactive in finding bugs and backdoors, IMHO. The level of transparency is stellar, especially compared to proprietary software companies. What the FOSS world has accomplished in 24 hours after detection of the backdoor code in #xz deserves a moment of humbleness. Instead we have flamewars and armchair experts shouting that we must change everything NOW. Which would introduce even more risks. Progress is made iteratively. Learn, adapt, repeat.
There simply is no established or easy way to detect backdoors done the #xz way. We give powers and trust to maintainers because that is the development model.
Anyone suggesting there is an easy fix has not understood the issues at hand.
But we are Open Source which allows everyone to dig, check, read code and investigate.
I know the next 3-7 days will be filled with exaggeration and doomsday talk, but IMHO the #xz backdoor, though seemingly meticulously planned for a long time, failed miserably as it was caught at a stage where it wasn't widely deployed but only in testing/prerelease distros. Yes, it made it quite far in the supply chain but it ultimately failed. The mess is being cleaned up, no cases of actual use of the exploit in the wild are known thus far. The immune system of FOSS has worked. Again.
Here's my main takeaway from the #xz crisis: require GitHub contributors to have a verified fediverse account in their profile links, and use it to find out what their actual reputation is.
The abusive behavior that was being used to manipulate Lasse Collin into bringing on more maintainers for #xz went unnoticed because abusive behavior in Open Source communities is so pervasive. In context, we can clearly see it was part of an orchestrated operation. Out of context, it looks like just another asshole complaining about stuff they have no right to complain about. https://robmensching.com/blog/posts/2024/03/30/a-microcosm-of-the-interactions-in-open-source-projects/
If #xz were a Go or Rust dependency, you wouldn’t have a single copy of xz library on your system, but many, #xzbackdoor hidden in every executable that uses it. Distros would have to rebuild all packages using that lib (not just the lib itself), which could take days or weeks, and users would have to update them all, downloading tens or hundreds of megabytes.
If you install binaries directly from vendors/devs, it’s even worse – you wouldn’t even know which ones are affected and you’d (1/3)
Does everyone understand how much luck was involved in this exploit in #xz being discovered so quickly? And, what it tells us about the attacker?
This was a subtle and sophisticated attack implemented over years. The attacker was made a co-maintainer two years ago, and they made numerous innocuous-looking and seemingly unrelated changes over that time, sometimes through a second account, that eventually added up to a backdoor. Along with many innocent commits, too. #Linux
Hey it's totally cool that #Microsoft#GitHub blocked access to one of the repositories in the very center of the #xz backdoor saga. :blobeyes:
It's not like a bunch of people are scrambling to try and make sense of all this right now, or that specific commits got linked to directly from media and blogposts and the like. :blobcatcoffee:
#xz You know what, sod “tracking all the contributions and dependencies”, that way lies shit ideas like “blockchains for paying musician royalties”.
Let’s have a Universal Basic Donation system for open source maintainers. We donate to a central place, and any open source maintainer of a project that’s used by more than X other people/projects can sign up to get a share. Everyone gets the same.
Can I just say that I have created #curl releases "the #xz way" since the 90s: I generate the release tarballs on my machine. It makes the tarball have (generated) files included that are not present in git. It's a feature. But it also makes it harder for observers to figure out if the additional files are fine or not.
Wir sind dieses Wochenende nur durch unglaubliches Glück und extrem knapp an wohl einer der grössten Katastrophen rund um die globale IT-Sicherheit vorbeigeschrammt.
Phuh! Doch — was ist eigentlich passiert? Wie konnte das überhaupt geschehen? Und was können (und müssen) wir tun, um dies zukünftig zu vermeiden?
A lot has been written about the #xz exploit, but nothing seems to have been said about the systemic vulnerability it exposed. The reason #xz got to sshd was via libsystemd. If you ask the question how big a target is this, you get an uncomfortably large answer:
> rpm -e --test libsystemd0 2>&1|wc -l
166
Debian patching sshd to remove libsystemd only reduces this target by 0.6%