whitequark,
@whitequark@mastodon.social avatar

me: "i like it how the entire LLVM project is one monorepo"
her: "when are you going to compile the entirety of LLVM to WebAssembly?"
me: "yes."
her: "... I said that as a joke"

whitequark,
@whitequark@mastodon.social avatar

i am adding so many ifdefs per minute right now

whitequark,
@whitequark@mastodon.social avatar

TableGen... my old nemesis

sen,
@sen@hachyderm.io avatar

@whitequark I am feeling exactly this today.

whitequark,
@whitequark@mastodon.social avatar

@sen oh no... my condolences

whitequark,
@whitequark@mastodon.social avatar

i've ported much of the upstream LLVM (18.1.2) to WASI, enough to link Clang

i'm linking Clang

thus far the linker has consumed 15 minutes of CPU time and 10 GB of RSS

eventually it might actually link Clang?

nervously looks at %MEM

whitequark,
@whitequark@mastodon.social avatar

it took over 20 minutes and about 18 GB. obviously, there are missing symbols!

whitequark,
@whitequark@mastodon.social avatar

suffering! yay!!!!

image/png
image/png
image/png

whitequark,
@whitequark@mastodon.social avatar

tracking progress of the linker by noting that it is on minute 10 out of approximately 22

whitequark,
@whitequark@mastodon.social avatar

how hard can it be to select a few instructions .

whitequark,
@whitequark@mastodon.social avatar

launching the linker, sitting down and reading a chapter of a book

whitequark,
@whitequark@mastodon.social avatar

motherfucker could you not tell me about the missing symbols BEFORE spending 20 minutes linking

whitequark,
@whitequark@mastodon.social avatar

oh wait, this one is on me

i'm building with -flto facepalm

if i recall, linking with -flto=thin is nearly as bad since basically all time is spent in the instruction selector, and linking without lto is still pretty bad, but it's not quite "twenty minutes each time" bad

whitequark,
@whitequark@mastodon.social avatar

[100%] Linking CXX executable ../../../../bin/clang
[100%] Built target clang

whitequark,
@whitequark@mastodon.social avatar

(maniacal laugh)

$ wasmtime -S threads=y llvm-build/bin/clang --version
clang version 18.1.2 (https://github.com/llvm/llvm-project 26a1d6601d727a96f4301d0d8647b5a42760ae0c)
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir:

jhwgh1968,
@jhwgh1968@chaos.social avatar

Now @whitequark just needs to get the patch set merged upstream

laughter stops instantly

whitequark,
@whitequark@mastodon.social avatar

@jhwgh1968 I am obviously aiming to do this

promovicz,
@promovicz@chaos.social avatar

@jhwgh1968 @whitequark wdym - you run an unpatched compiler!? 😅 (of course, I usually do)

whitequark, (edited )
@whitequark@mastodon.social avatar

$ cat test.c
int add(int a, int b) { return a + b; }
$ wasmtime -S threads --dir . ./llvm-build/bin/clang -c test.c
$ file test.o
test.o: WebAssembly (wasm) binary module version 0x1 (MVP)

notgull,
@notgull@hachyderm.io avatar

@whitequark This genuinely might be the most blessed thing I’ve seen all month. Only a lunatic would attempt this, let alone complete it. Congratulations!

whitequark,
@whitequark@mastodon.social avatar

@notgull wait until I get the linker and the compiler driver working too

notgull,
@notgull@hachyderm.io avatar

@whitequark Does this mean I could have an online C compiler test environment entirely in my browser? Hell, could it be ported so that the Compiler Explorer runs entirely in the browser?

whitequark,
@whitequark@mastodon.social avatar

@notgull yeah

dngrs,
@dngrs@chaos.social avatar

@whitequark super exciting! Do you think it's possible to target WASM for an LLVM JIT compiler?

whitequark,
@whitequark@mastodon.social avatar

@dngrs i don't see why not

whitequark,
@whitequark@mastodon.social avatar

$ wasmtime -S threads --dir . ./llvm-build/bin/wasm-ld test1.o -o test.wasm
Error: error while executing at wasm backtrace:
0: 0x7ed5b - <unknown>!<wasm function 583>

Caused by:
0: memory fault at wasm address 0x378268 in linear memory of size 0x340000
1: wasm trap: out of bounds memory access

ahhhh.... here we go. the suffering I expected!

brouhaha,
@brouhaha@mastodon.social avatar

@whitequark Programming leads to anger. Anger leads to hate. Hate leads to suffering.

whitequark,
@whitequark@mastodon.social avatar

@brouhaha but I like suffering

brouhaha,
@brouhaha@mastodon.social avatar

@whitequark leads to the dark side, it does

whitequark,
@whitequark@mastodon.social avatar

@brouhaha yeah I'm okay with that

whitequark,
@whitequark@mastodon.social avatar

(grins and creates another LLVM checkout) time to upstream the LLVMSupport patches

ethantyping,
@ethantyping@hachyderm.io avatar

@whitequark this is excellent! Thank you for working on this. I've been wanting this for a long time to do funky build things :)

whitequark,
@whitequark@mastodon.social avatar

@ethantyping oh hey, nice! what kind of?

ethantyping,
@ethantyping@hachyderm.io avatar

@whitequark First thing I'd want to do is a client-side Rust playground.

I also have hopes to build a Python build backend that would run Python and a C compiler in WASM to have deterministic, safe, builds, but that is a bit of a deep hole :)

whitequark,
@whitequark@mastodon.social avatar

@ethantyping deterministic builds is a part of what I'm building this for!

whitequark,
@whitequark@mastodon.social avatar
whitequark,
@whitequark@mastodon.social avatar

so... that one was on me, right? so i did a non-LTO, RelWithDebInfo build

the linker finished quickly. and then ran wasm-opt, which consumed a downright concerning 22G of RAM and something like 10 minutes of real time

you can't win.

https://mastodon.social/@whitequark/112465582937702577

whitequark,
@whitequark@mastodon.social avatar

afaict the compiler driver calls wasm-opt unconditionally if:

  • it is on PATH
  • you pass -On with n>0 to the linker

so to make it not spend 10 minutes of real time in wasm-opt you need to pass -O0 to the linker. i only figured this out by reading the sources of the compiler driver.

nobody should be figuring out compiler options by reading the sources of the compiler driver. nobody!

whitequark,
@whitequark@mastodon.social avatar

-rwxr-xr-x 1 whitequark whitequark 877M May 19 09:31 clang-19
-rwxr-xr-x 1 whitequark whitequark 537M May 19 09:35 lld

if i combine the two i will have a 1.5 GB wasm binary. (i will combine the two, eventually)

whitequark,
@whitequark@mastodon.social avatar

so now that i've generated half a gig of debug information i can just feed it into wasmtime, have it generate DWARF for me, and be done with it, right?

wrong.

thread 'main' panicked at /home/whitequark/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-cranelift-19.0.1/src/compiler.rs:565:13:

whitequark,
@whitequark@mastodon.social avatar

do you think the debug information was valid? nope! wasm-opt unconditionally launched by wasm-ld (which performs link time optimization despite lack of -flto) also corrupted the DWARF data

did adding -O0 turn it off? yes! did that make the DWARF data seemingly well formed? yes! did it give me line numbers? of course not, it just gave me function names

0: 0x26e700d - lld!__wasi_thread_start_C
1: 0x26e84cb - lld!wasi_thread_start

thanks

also addr2line doesn't work

whitequark,
@whitequark@mastodon.social avatar

why does a single chromium renderer process peg six cores

whitequark,
@whitequark@mastodon.social avatar

i have solved my problem by disassembling the wasm module (i went through three different disassemblers until i found one that is able to finish this in finite time) and correlating it manually with the source code

... i'm pretty sure that wasmtime isn't reporting the trap correctly T__T

there should be one more frame on top of the frame 0

image/png

sanfierro,

@whitequark Oooh, I admire your patience and determination! Does whitequark refer to quantum physics or something, or even have a meaning beyond surface level (although I personally need an explanation even for the surface level)?

whitequark,
@whitequark@mastodon.social avatar

@sanfierro white quarks don't exist

SoniEx2,
@SoniEx2@chaos.social avatar

@whitequark we would just add process management to wasi ngl.

whitequark,
@whitequark@mastodon.social avatar

@SoniEx2 then do it :3

SoniEx2,
@SoniEx2@chaos.social avatar

@whitequark ...do you need more than the ability to run processes synchronously? because if you just need "processes as function calls" we don't even need process management.

whitequark,
@whitequark@mastodon.social avatar

@SoniEx2 i think gcc for example uses pipes extensively, which are a problem

i plan to add a custom hostcall which handles stdout/stderr but no stdin, or perhaps stdin as a fixed string

stdout/stderr can and have to be callbacks

SoniEx2,
@SoniEx2@chaos.social avatar

@whitequark hmm we don't think there's a wasi interface for fifos yet...

but aside from that, posix_spawn can be emulated by pretty much just running the wasi module. as in calling into it. optionally in a separate thread.

whitequark,
@whitequark@mastodon.social avatar

@SoniEx2 yep, that's the plan. I want it to be fully synchronous because SAB is kind of a pain, though I recently discovered that you could use service workers to add CORS headers client-side apparently?

promovicz,
@promovicz@chaos.social avatar

@whitequark it may be NP-hard to prove whether advanced optimizers terminate? I’m pretty sure there are only subset proofs. many optimizers just „hammer things until nothing changes“ - without any guarantee of local/global optimality.

not sure about llvm!

gsuberland,
@gsuberland@chaos.social avatar

@whitequark they call it clang because it's so heavy that it makes a clang sound when you drop it.

promovicz,
@promovicz@chaos.social avatar

@gsuberland so, @whitequark drops the clang like a mic? fitting, imo.

JamesWidman,

@whitequark this seems... unusually long...?

have you tried lld or mold (https://github.com/rui314/mold)?

(admittedly, i haven't linked clang in the past couple of months)

whitequark,
@whitequark@mastodon.social avatar

@JamesWidman that's lld. mold doesn't support wasm

JamesWidman,

@whitequark oh wow...

i'm wondering if the time consumed by the linker is somehow a wasm-specific thing?

how long does linking take if you build a version of clang for the native CPU arch?

whitequark,
@whitequark@mastodon.social avatar

@JamesWidman it's running the relooper, yes

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • DreamBathrooms
  • ngwrru68w68
  • modclub
  • magazineikmin
  • thenastyranch
  • rosin
  • khanakhh
  • InstantRegret
  • Youngstown
  • slotface
  • Durango
  • kavyap
  • mdbf
  • ethstaker
  • megavids
  • osvaldo12
  • tester
  • GTA5RPClips
  • cubers
  • everett
  • tacticalgear
  • cisconetworking
  • normalnudes
  • anitta
  • provamag3
  • Leos
  • JUstTest
  • lostlight
  • All magazines