Terrible little gremlin idea of making a distro only secretly rewrite all the load bearing libraries in the OS bootstrap brainstem in rust, and just trick increasingly large chunks of the system to not be C and see how long it takes anyone to notice that glibc looks a bit weird. Like a supply chain attack of kindness
@vathpela But I was thinking less of strcpy and more __vsyslog_internal, or qsort, or iconv, or realpath, or ld.so handling GLIBC_TUNABLES. To just glance at the last couple years of glibc memory safety CVEs.
glibc is also one among many, I was thinking more of the long tail of pervasive libs that nobody looks at too hard.
Also this is not an announcement or grant proposal, the words I used were "terrible little gremlin idea", not "fleshed out plan to solve the supply chain" :)
Will probably get my cover blown by postgres when the rewrite incidentally makes a random micro-op faster and prompts a full scale forensic autopsy to explain the benchmark variance
And also get on my tiny soapbox and be sad that doing this would set the various source bootstrap projects back significantly, because all the focus has been on bootstrapping to a C toolchain, and rust's "bottom turtle" source bootstrap story is it doesn't have one and needs to path through a working C universe, and even if mrustc makes that path shorter it still terminates at C exists and even if you assumed some ability to build rust from some other foundation, surprise asshole LLVM is C++
@stoltz Yup, although depending on your rules of engagement, this doesn't address source bootstrap so much as hide it behind the couch: who compiled the bootloader? And the kernel? Using what toolchain, that came from where? And there's still at least 1 piece of firmware in play (the PSP, which of course has had CVEs 🙃).
But this is why it pays to be a pragmatist not an idealist in this quest. That board is a big step towards making x86 viable on the source bootstrap path :)
@stoltz (also the board in question is very bespoke and not available to the general public, so I have to hope and pray that either that changes, or that others follow the lead)
@stoltz If you are curious about what the most extreme source bootstrap would have to be, I wrote about it a couple years ago: https://blog.dave.tf/post/finding-bottom-turtle/ . The purist's road on this quest takes them to incredibly weird, beautiful places, all the way back to the literal birth of software.
The state of the art has advanced: last year, Guix got down to a 357-byte binary seed[*], which after many many build steps blooms into a complete linux distro.
@stoltz The Guix source bootstrap is an incredible achievement, absolutely staggering. And because of the way the lowest layers of the bootstrap are structured, there is a viable (difficult, incredibly tedious) pathway to continuing the descent past the host OS and firmware. It's wonderful.
It's easy to get really depressed and nihilistic if you start chasing a source bootstrap, of any form, because until you look at it deeply you can't internalize just how much C/C++ is in that path with not even the start of an idea of how to replace it, and that's before you even get to the best bootstrap we have assuming a whole-ass linux kernel just exists somehow and also firmware doesn't exist.
Rust doesn't make that (much) worse, but there's also not a path to making it better, yet.
@danderson I wonder if Antithesis will make it possible to rewrite large chunks of supporting software "We deleted all of our dependencies (including Zookeeper) because they had bugs, and wrote our own Paxos implementation in very little time and it had no bugs. We rewrote the entire transaction processing subsystem of our database to make it faster and more scalable – a bonkers thing to do btw – and the project was shockingly not a debacle, and oh yeah it had no bugs."
@vaurora Maybe, although that's way higher up the stack than I was worrying about here. And while antithesis looks very neat, I struggle to get excited by silver bullets that come with a "Contact sales" label :/
@vaurora For the purist source bootstrap, the two difficulties I see are somehow replacing all the C toolchains and libs at the bottom of the world without picking them all back up again because the replacement language bootstraps through C, and the replacement language toolchain being something that's possible to reach from the bottom turtle step (slow, terrible, but firmware-free computer that bootstraps from a physically realized monitor program)
@vaurora Rust is intriguing as the low level systems language for everything, except it's an eyewateringly intricate and complex language and that forces its bootstrap path to either go through a simpler language like C (as it does now), or... I dunno, some other miracle has to occur that enables the journey up the stack to still start from pure hardware (which these days means very slow and limited machines, and you somehow have to crawl out of that and up to openpower/riscv/x86 from source)
@danderson@vaurora the best attempts I've seen at crawling out of the morass start at the lower end, the "tiny core language" approach, where you can build the core nonsense by hand even if it doesn't run fast, and use that to get up to a runtime that can handle the more complex stuff. But most, including Rust, start very much higher up the stack and that makes a lot of implicit design choices that are harder to retcon later.
@creachadair@vaurora That's what the current bootstrapping efforts do, yeah. They start with a ~400 byte monitor program whose machine code you have to verify by hand, then using that they crawl up through a basic assembler, a slightly better assembler, and then a half dozen intermediates to crawl up to GNU Mes, a mutually self-hosting root pair (a basic scheme written in a subset of C, and a C compiler written in the scheme). Anther couple dozen more steps gets you to gcc.
@creachadair@vaurora It's quite the achievement, although far from complete: the "stage0" bootstrap path is meant to be startable from anywhere, but in the current best bootstrap path they start from... a posix userspace version of the monitor, so you still end up with a secret kernel and core userspace 💀 But, progress!
@danderson we truly believe the long-term future has to involve not only more FFI, but more source-to-source translation
yes, mechanical source translation requires a ton of tuning to produce idiomatic output. we did it on a small scale a long time ago (a million lines of very insular code), we do believe it could be done at a larger scale with enough attention paid to it
@danderson there is never going to be a "final" programming language, nor should there be - it would mean that the creative element of programming is entirely gone and it has become purely rote work
@danderson it's important however that we be able to actually move on, when it becomes clear on the scale of decades that we need to do so. we can't be carrying our baggage around forever.
@irenes Maybe, I dunno. I'm not optimistic about source translation producing inspectable output, but I'd be delighted to be wrong. And even if things will change in the future, or remain a giant stratified mess, I still want a bootstrap path to exist that can make inspecting the source code mean anything 😅
Add comment