'" The data shows that “frozen” vendor #Linux kernels, created by branching off a release point and then using a team of engineers to select specific patches to back-port to that branch, are buggier than the upstream “stable” Linux #kernel created by Greg Kroah-Hartman. '"
@kernellogger I'm afraid I can't support the counting methodology in the paper either. Besides the not applicable because of config issues RH people cite, there's also the fact that not everything that has a cc: stable tag is an exploitable bug. Plus every fix backported carries risk (just look at the number of regressions in stable due to backports) so that risk has to be set against the benefit of the backport. A general rule would be if it's not exploitable don't backport it.
@kernellogger That's to be expected, but it is also not the point of them.
I agree they shouldn't need to exist, but the realities of how many many an organization manages their IT necessitates their existence.
The industry doesn't want to go through the withdrawal phase of building a better world.
@kernellogger as usual, the point is not that these are bug free, but that they are regression free. The kernel upstream releases break userspace on every new release, and kernel maintainers don't care. See https://github.com/torvalds/linux/commit/a1912f712188291f9d7d434fba155461f1ebef66 for example, as Daan just found out, which removed a mount option without caring that it is still being used, so since 6.8 every btrfs device can no longer be mounted by systemd
Well, to claim "kernel maintainers don't care" you have to at least report the bug to them[1]. That afaics has not happened yet (or I could not find it).
"since 6.8 every btrfs device can no longer be mounted by systemd": then why was this only noticed 2+ months after a release with that commit went out? This raises the question: what kind of problem did users actually run into?
[1] yes, sure, ideally they would have done a code search first, but we are all imperfect…
@kernellogger well, the kernel doesn't have a bug tracker - not for real anyway, bugzilla.kernel.org might as well be pointed to /dev/null, so no idea what "reporting" would even mean in this case. I do not use BTRFS so I am not affected, just sharing what was reported to me. It looks like it was reported against the Debian kernel package too now: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071420
@kernellogger@bluca but, following upstream, passthrough patches, and maintain downstream compatibility, is the duty of a middleware, like systemd, from kernel to user interface, no?
and thx for the link to the debian bug tracker; but I want to see more details first what went wrong there, as I'd expect it would be pretty unlikely that this is the first debian btrfs user that updated to 6.8 or higher; so why did it break for that user, but apparently not for the others?
1/ FWIW, I think you owe the kernel developers an apology, as you made a lot of noise and claimed "kernel maintainers don't care", when they clearly do once the problem was properly reported -- and quite quickly even. And yes, sure, in the ideal world they would have cared some more and performed a code-search before removing this option to prevent it in the first place. But we are all imperfect and make mistakes. Same for @pid_eins, who…
2/ …wrote "And my main beef here is that they claim they wouldnt do it ever..."[1], as that is not even true. They often try changes or removals to see if it breaks something – and if it does, it's reverted. Even the removal of the support for the original i386 was handled like that by Linus himself.
@kernellogger@bluca sure, but then the rule is not "we never break userspace" but more "move fast and break things, and sometimes revert where people protest too loudly".
I mean, that's fine by me, but maybe they should communicate it like that then.
The thing is that removing a widely documented mount option is very obviously a compat breakage. You cannot discount that. It's not just a "mistake" to remove something like that, it's an obvious attempt to break compat.
the exact relevant rule is just "no regressions", nothing more. Everything else is just left to the interpretation by people (in typical Linus manner you might say). 🥴
I find the sound of that mail quite different from your much weaker "let's maybe undo the worst shit if people complain too loudly"... And of course "uh, sometimes we fucked up so hard, we cannot fix it anymore, let's add a new api instead" (which is what happened in the block device capabilities/media change api).
(again, I actually find it OK if API is broken from time to time, just be honest about it, and communicate properly, and do a bit of research first. Don't claim that uppercase extremism and then do not even superficially follow through)
@kernellogger@pid_eins My impression, having more of an outside perspective and working with higher level languages: should deprecations perhaps always be gated with a config flag, perhaps even a common one similar to BROKEN?
With Java/Scala, it's always quite clear for me where deprecated methods are used. Also I can have builds fail due to that or not, so that I notice new deprecations when building / in CI.
there are various things that can work and I guess it depends on the situation what reasonable and effective.
For the kernel I something think "add delays (together with a msg in the logs) that grow longer and longer over time when people use deprecated stuff, at some point people get curious and will investigate" might be something that might help, OTOH it's a kind of stupid idea 😂
reg. the "distros want no-regressions, not no-bugs":
from my point of view the whole situation could be a lot better if distros would spend some of the money they currently invest in CI instead invest in working on workflow improvements and some others stuff to ensure regressions do not happen in the first place or are quickly resolved.
@kernellogger@pavel I’d be interested in knowing how you would improve the workflows. What’s missing, what can be improved and what shouldn’t be done. I would love to help with this however I can. :)
There is no easy answer here, as it are lots of details; but there is a decent chance I need to write this up soon anyway; if I do, I'll get back to you!
@kernellogger On the other hand, what's the difference between a distro branching off and backporting stuff from mainline and upstream stable branching off and backporting stuff from mainline? Why can the upstream stable maintainers do this and "a team of engineers" cannot? I think the difference could be better characterized and (if you pardon the expression) makes all the difference.
@vegard@kernellogger "a team of engineers" COULD do that (hint, Android does, by just taking the stable updates), but the paper shows that a specific "team of engineers" currently does NOT do that, which puts the users of those kernels potentially at a greater risk.
Read the paper, it's interesting.
Disclosure, I read it before it was published, but had no influence on it at all.
Add comment