An incredibly technically complex #backdoor in xz (potentially also in libarchive and elsewhere) was just discovered. This backdoor has been quietly implemented over years, with the assistance of a wide array of subtly interconnected accounts:
@eb I really hope that this causes an industry-wide reckoning with the common practice of letting your entire goddamn product rest on the shoulders of one overworked person having a slow mental health crisis without financially or operationally supporting them whatsoever. I want everyone who has an open source dependency to read this message https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html
@eb@glyph Very cool! Any chance you might be willing to share that particular snippet - even just the HTML structure - under a less restrictive license than AGPL? It seems like the kind of thing I would love to use or adapt on my own site, but I don't want to (and probably legally can't) share my site and all the services it uses under that license.
@eb "I never thought a sophisticated APT would backdoor my volunteer-maintained infrastructure that I got for free" sobs entire industry who voted for the "volunteer-maintained infrastructure that I get for free with no defense against sophisticated APTs" party
@glyph@eb gee what a shame they didn't - hang on, I need to check my notes - "pay a fair price for, or otherwise support the developers of the tools they use".
@glyph@eb I guess we will get another corporate advisory to check for this version (well, first, that a taskforce has been formed to investigate the problem), and then everyone moves on as if nothing had happened ...
@glyph@eb please note that we are ALSO no fans of the "subsume free software into capitalism" solution that corporate and statist rhetoric has been pushing for a couple years now
@irenes@glyph@eb It's tricky to avoid the challenge that arises from the problem that (1) producing free software is work and (2) the workers live in a capitalist society and (3) the workers therefore need to pay for food and shelter.
Verily, there is no ethical consumption under capitalism.
@glitzersachen No. I think the project was a tasty target because it was barely maintained (due to capitalism) and no-one was actually looking at the code being submitted in detail (also because of capitalism).
We have come a long way since I installed my first opens source OS, and frankly, I sometimes wonder if we haven't been going into the wrong direction. The big corporation have very much profited from our work, rarely gave back and in the end have poisoned the well to the detriment of us all.
@krans@glyph@eb sure. well, so the reason we personally call the thing we do "free software" is precisely to highlight the point that our own goal in publishing stuff without charge is very much to work towards a world without that problem, by creating something that exists as far outside it as we can manage (not all the way - obviously we have the free time to do that because of our other privileges)
@irenes@glyph@eb I thought it was called "free software" because users are allowed to do whatever they want to with it including modifications, not because it's provided free of charge.
The founders of the Free Software movement were Libertarians, not Socialists (unfortunately).
I guess we were talking at cross purposes — sorry.
@krans@glyph@eb we're very proactive-death-of-the-author about this. the FSF has failed to provide ideological leadership due to RMS's top-down style, but many of the ideals are good ones and it's the job of the current generation to renew the movement if we want our children to be able to enjoy its fruits the way we did
@glyph@eb We’re already seeing the end of open source with all the relicensings. Maybe this’ll accelerate the trend and we’ll be in a vastly different place in a few years from now.
@glyph@eb Twenty-something years ago, the nant maintainer got death threats because a feature in the latest C# release of that time didn't get supported fast enough.
Industry-wide reckoning is not the thing to expect.
@glyph@eb I'm frustrated that big tech's efforts to increase core library security are "your project is too popular, you must use 2FA" and "the best reverse engineers in the world will find your bugs and put you on a 90 day disclosure deadline" and not "here is $100K/year and benefits to keep doing what you're doing at your own pace."
@geofft@glyph@eb I'm certainly not disputing that it's a real problem that that doesn't happen more often, but isn't there some precedent for big tech companies hiring people to work on specific open source projects? So it's not totally unheard of
@diazona@geofft@eb there's actually quite a bit of effort trying to address this problem, but it is a big collective action problem and … well, just look at the email, and tell me if that couldn't be just about any maintainer, on any project, anywhere. and xz is extremely core infrastructure, so the fact that this problem was this severe in this context is discouraging for the state of the rest of the industry
@glyph@geofft@eb Oh of course. I guess I just wanted to acknowledge being in a state of "a tiny bit of progress" rather than "zero progress". (I have an optimistic streak that comes out sometimes)
@diazona@geofft@glyph@eb there’s a lot of precedent for hiring maintainers of top-level programs whose brand (for lack of a better term) has reached the level of awareness of a C-level with a hiring budget. Collectively pooling money to help the projects C-levels have never heard of… has a much weaker track record. We’ve been trying to tackle it at Tidelift for a while and suffice to say I’ve definitely had a lot of “but it can’t happen to me” conversations.
@diazona@geofft@glyph@eb I increasingly wonder if we aren’t due for some “defragging” of a lot of core infra, with many projects pooled together, maintained, and funded more collectively, like Ruby Together.
@luis_in_brief@diazona@glyph@eb Yeah that resonates with my experience. People like GvR get hired (which is great!) but there's a whole dependency stack underneath. Their maintainers often have a strong résumé to get hired for a normal big tech job at a company that uses the language/ecosystem/etc. but not necessarily for maintaining the project as their job. Sometimes the job is even "build something similar for an internal non-OSS ecosystem."
@geofft@diazona@glyph@eb yup. Or they get hired with the promise that they’ll get 20% time to work on it, and that goes away for reasons (sometimes good, sometimes bad), or…. Etc etc
@geofft@luis_in_brief@diazona@eb there are layers and layers to this. Famous maintainers get hired more than critical maintainers. And maintenance is important but how do you pay for the commons of new projects? The tidelift model gets us part of the way there, because these costs need to be aggregated and there needs to be some kind of oversight, but even if they were universally adopted (and that is far from true) there are so many missing pieces
I am way overdue in finishing and publishing my negative review of Eghbal's "Working in Public" but one of my critiques is that she basically concludes that maintainers need to become famous and use Substack/Patreon to crowdfund (from individual donors) in order to sustain their work. Which really doesn't fit what we have found in critical FLOSS infrastructure IMO.
@benwis@brainwane@luis_in_brief@glyph@geofft@diazona@djc there’s some website (I forget what it is) that basically you pay x amount of dollars and it audits your entire dependency tree and attempts to pay maintainers proportionally. Unfortunately iirc it was kinda flawed but I think it’s a solid idea
@brainwane@luis_in_brief@glyph@geofft@diazona@eb yep i never published my own review because of that. The Road and Bridges report was great. The book felt like a massive PR piece for GitHub sponsor feature and a way to hide the problem.
@glyph@luis_in_brief@diazona@eb Yes, e.g., what if the current maintainer is genuinely unavailable/uninterested? As may well have happened with xz even with a job offer.
Funding a new maintainer is by itself defensible, but doing so will drastically change both the pressure on the current maintainer and the choice of who becomes maintainer (e.g. there's now a bias in favor of those who have US work authorization).
I'm curious if either Tidelift or the commercial distros have norms for this.
@geofft@glyph@luis_in_brief@diazona@eb A relationship with a critical FOSS dependency maintainer is very clearly classifiable as independent contractor, and SHOULD or even MUST be for the sake of project integrity. There should be no reason to need US work authorization.
@dalias@geofft@glyph@luis_in_brief@eb Legally speaking I think it could be set up either way. Although if an OSS project maintainer is employed (not contracted) by a company to maintain the project, it is kind of as if the company is acting as the maintainer, which certainly raises questions about their motivation....
By my (cursed) maths based on both Tidelift report and the Synopsys one, at least 50% of every single sloc in commercial project (averaged) is foss maintained by a "weekend maintainer".
Only 10% is commercially maintained.
And the rest is mostly "partially paid for maintenance" which means "sometimes i maybe get paid a few hours on a contract to do it".
And that is going to the conservative side. I think it is more than 50%.
And realistically, we are not going to massively move the needle here. This is what makes opensource win. If we change that equation, people will revert to this.
We need to find ways for weekend maintainers to do more with less effort on their part too. We are not going to pay everyone soon enough. We need to work on both angles.
@luis_in_brief@glyph@eb I should try harder to figure out what a Tidelift is and how to convince my employer to sign up. But also... IMO Microsoft or Google (whom I am subtooting) etc. can singlehandedly employ all the maintainers of ldd sshd and that would get results that fractionally paying for the commons never will.
Like this should be the job of a distro, and RH/SUSE/Canonical/Oracle kinda do this, but clearly none of them actually saved their customers (or the world) from this.
@luis_in_brief@glyph@eb Re "I should try harder to convince my employer to sign up for Tidelift": can Tidelift tell us whether .... looks at /proc/$(pgrep sshd)/maps ... libcap-ng or zlib has a sustainable maintenance story and what it would take to give it a sustainable maintenance story?
Looks like libcap-ng's maintainer is at RH and zlib's maintainer is (probably comfortably) retired, but
both seem to be one-person jobs, which as @ehashman points out is a problem.
@geofft@glyph@eb@ehashman sadly we do that for just about everything except C and C++, because our valuation metrics rely on customer usage and lol/sob C/C++ packaging is (relatively) poor and unstandardized.
@geofft@glyph@eb@ehashman but note also that the goal is not "support this one package", it's support as much of the stack as possible. If we focus on "tell you about one package" then we just replicate the problem (noted elsewhere in multiple threads here) that famous packages get support, and others (which data would surface but 'namebrands' don't) get zero.
@geofft@glyph@eb That but also, could they fund a proper work on you know. Replacing shell scripts and Autotools as the way to do jigs? Could we get one build system paid by them targeted at the actual needs of opensource maintainers?
@Di4na@glyph@eb I am interested in this problem! I do think newer languages are better at this, though, and not many C projects are getting created from scratch these days. It feels like a lot of the problem is existing code and their build systems that support e.g. VMS and Windows XP.
(In Bazel's defense it seems like it was the only way for Google to sustainably open-source stuff at all - some things like Borg or GCL only made it to the public in the form of a v2 reimplementation.)
@geofft@glyph@eb sure but that is Google problem and that stuff is not that used. And like. Chrome is infamously known to be unbuildable.
And the problem is not vms and windows xp. It is that all these build systems are barely a build system at all. And that the incantation to please a C compiler (and let's not talk of a linker) especially when you have a dozen vendored stuff are... More than arcane.
@geofft #Autonomi can change this. One of the goals is to democratise by leveling the playing field for developers as well as users. So removing silos controlled by others on the one hand and providing income to those creating value in the system (including but not only developers). For developers who want to scale, there's a liberation from seeking capital, because infrastructure scales without cost. So one dev in a bedroom cold build the next unicorn and stay true to their values. @glyph@eb
@glyph@eb
In this case, abandonning development was the healthy thing to do, and cajoling this person, has only led to further harms.
I Disagree with the notion throwing money at all the open source is viable, or going to fix complex issues like mental health. although I agree companies should pay more. But it's 2000 commits, so probably easier to remove xz and stop using it; than to attribute much ongoing input to xz format support.
But Jia Tan has earlier commits than the one you identified as first. aa75c5563a760aea3aa23d997d519e702e82726b is their first commit.
Note that there are earlier commits that mention they were based on contributions by Jia Tan, before the first commits with them as the author. 20e7a33e2d59c6a814447d3991f21e2702174b20 in Feb 2022 is the first of those.
@eb I wonder when they were first added to the project on github. Any commits after that point are suspicious, even if the commit claims to be authored by someone else.
They merged a pull request for the 1st time in 6fd39664de47801e670a16617863196bfbde4755 on Jan 7 2023.
@joeyh this is a good question. I'll include this information, I briefly was looking for a change in the maintainer list but right now I'm focused on the exploit chain
@eb what's really surprising is the same thing not happening more often. Or we are all just very unaware since the pay off to do these kinds of things is so huge specially for state sponsored attacks or others with enough money and effort to put into it.
@eb thank you for writing this up. I am particularly curious what is going on with the tukaani-project org, since it seems as of this moment that JiaT75 is still a member. Is the whole project / org completely burned or was this one infiltrator who might be expelled?
@glyph my understanding of this is at this point they are the primary contributor. Notably, a website previously under the control of a different contributor was updated to point to the github of this org, which of course Jia has access to. I'm not positive that Jia owns the org, however.
Add comment