hywan,
@hywan@fosstodon.org avatar

I’m starting a new project to learn about linkers. It’s called ˋweld`, and it lives here https://github.com/Hywan/weld.

I do this on my free time; understand very sporadically.

This toot is a thread to show progress or to ask help.

hywan,
@hywan@fosstodon.org avatar

So far, I’m writing the Elf64 parser. The goal is to get zero copy, period.

Yesterday I’ve added section’s data and name, still with zero copy, https://github.com/Hywan/weld/blob/bfb9fd55c5b2f9114e8f8ab21c5f49d48f9c3b98/crates/parser/src/elf64/mod.rs#L720.

It relies heavily on Rust lifetimes, and bstr to get bytes-based string-ish. The parser is written with nom, and is manipulating bytes slices only.

bstr: https://blog.burntsushi.net/bstr/
nom: https://github.com/rust-bakery/nom

#linker #parser #elf #RustLang

hywan,
@hywan@fosstodon.org avatar

Yesterday, I’ve also added support for big- and little-endian. All parser combinators can now handle endianness based on a generic type + trait, https://github.com/Hywan/weld/commit/5a1ff9f9643fe6b82e7b789e4c2cca7ee6615024.

It’s magic. Rust is cool.

#linker #parser #elf #RustLang

hywan,
@hywan@fosstodon.org avatar

Why did I start this project… 🤦‍♂️

hywan,
@hywan@fosstodon.org avatar

OK. Things are clearer. Let’s find time to improve the Elf64 parser. Maybe I’ll be able to link a super super basic object file “soon”.

I want weld to be a cross-linker: like you specify a triple target and it links object files whatever the format and the platform you run.

hywan,
@hywan@fosstodon.org avatar

I've added a new SectionIndex type, https://github.com/Hywan/weld/commit/9ecaef702532bc399c12229e58e688aa05cbf3a6.

It helps dealing with the semantics of section index in Elf64, and it also helps having a valid usize, which is helpful when used as a Vec index.

It's finally less error-prone: Using the Rust type system as much as possible.

#elf #parser

hywan,
@hywan@fosstodon.org avatar

Each program header and section header in Elf has a content. It's represented by the new Data type in weld, https://github.com/Hywan/weld/commit/f541364004c99aa7d666687767cb4005da3c9263 and https://github.com/Hywan/weld/commit/4ec23096b64775494226fea2e0e662bb24a410cb.

I'm experimenting this zero-copy API to request the content. Let's see where it goes. But it's very handy for debugging!

#elf #parser

hywan,
@hywan@fosstodon.org avatar

Next stop: Defining a type for representing alignments. It must be a power of two, non zero unsigned integer. Easy. https://github.com/Hywan/weld/commit/f2e9f00dc65634b6b8145ad04c9221c6f478aaf3

With that, still zero copy but more and more semantics by leveraging the type system. Nice :-).

Note: Option<NonZeroU64> in Rust has the same size as u64. Cool! Zero cost abstraction once again.

#elf #parser #RustLang

hywan,
@hywan@fosstodon.org avatar

Debugging is important. I’ve added a std::fmt::Debug implementation for Elf64 header contents/data, https://github.com/Hywan/weld/commit/4ec23096b64775494226fea2e0e662bb24a410cb. It’s really helpful! It shows strings, and later the symbols…

#elf #parser #RustLang

hywan,
@hywan@fosstodon.org avatar

The Elf64 parser now understands Symbol, https://github.com/Hywan/weld/commit/8d14d7574a2d41dc56dcfb047de9a0fc9f3572dd!
A gently iterator is provided: we allocate only when necessary.

I missed few symbol types and bindings, now in https://github.com/Hywan/weld/commit/d6b1d4de6349e1650158f78a04916696c76f7b9a and https://github.com/Hywan/weld/commit/735eb0bf25bc6df0ad976680ad887e6837b0cc15.

Now I see the same information as ˋobjdump` shows me. Everything is well typed, still zero copy and few allocations.

In my test object file, I see zero relocations yet. Perfect. It’s time to link this simple object file for real 😳.

#elf #parser #RustLang

hywan,
@hywan@fosstodon.org avatar

Don’t ask how or why, but I’m implementing an async file reader supporting fs, mmap, and io-uring within Future.

Eh, this project is an excuse to learn new things right? Right?

hywan,
@hywan@fosstodon.org avatar

#weld

Before leaving the elf64, I wanted to write some tests. Fortunately for me, parsers written with nom are really easy to test, they are just functions!

https://github.com/Hywan/weld/compare/13be3d2917f945cd347e9394c07853ac692f32de...9e6d55e2e137df3786b257228c7e35042520c0a7

When tests are easy to write, it's a pleasure to test everything.

With this test session, I've been able to fix one panic when reading a string in a data segment with an out-of-range offset.

#RustLang #test #elf64 #parser

hywan,
@hywan@fosstodon.org avatar

#weld

The weld-parser crate wasn't happy with its name. Now, we must refer to it as weld-object. This crate has ambitions for its life!, like supporting Elf32, MachO, COFF, and more object formats, look at this little cheeky!

https://github.com/Hywan/weld/commit/7556abeb5d80f015e92634d8b9a6c1494e815b9e

#RustLang #parser #object #elf64

hywan,
@hywan@fosstodon.org avatar

To be able to link many object files, we must be able to read them! Of course, there is std::fs, but we want to be fast right? So let's use mmap.

Oh, it's not available everywhere. So let's use a file picker, don't you mind, which will select the best implementation for your platform.

And here is weld-file, https://github.com/Hywan/weld/compare/7556abeb5d80f015e92634d8b9a6c1494e815b9e...f19d8bae61ef61c443d974bcf3d0a3101a69e7b3.

So far, it supports std::fs and mmap, more is coming

There is a FileReader trait, that reads the content async.

hywan,
@hywan@fosstodon.org avatar

#weld

Later on, I've added a new patch to make the FileReader trait thread-safe, https://github.com/Hywan/weld/commit/3078ab77176b333fb9db0820c24f7c0d09ac23f8.

Just a detail, because something funnier is coming :-).

#RustLang #file

hywan,
@hywan@fosstodon.org avatar

#weld

To be fast, weld needs to link objects concurrently & simultaneously. A few days ago, I’ve implemented a simple ThreadPool type, in the new weld-scheduler crate, https://github.com/Hywan/weld/blob/ea792c808887306acc6985bd71910fc35051a530/crates/scheduler/src/lib.rs.

It’s able to send Futures on various threads where async workers execute them.

It’s done with smol, a light & flexible set of crates to implement custom async runtimes, https://github.com/smol-rs/smol.

Next step: “Linker Strategy” to link things for real!

#RustLang #parallelism #concurrency #thread

hywan,
@hywan@fosstodon.org avatar

#weld

A little improvement on the scheduler to make it more system friendly.

ThreadPool no longer takes a ˋpool_sizeˋ but a ˋdesired_pool_sizeˋ. The system might not have all the expected available parallelism resources. Maybe it is busy, maybe it is limited. So now, the pool size is clamped, https://github.com/Hywan/weld/commit/8590fa4f5c80b5c827c0e928226d5b5bbd231f2c. The documentation provides more details.

Hopefully Rust has a standard function for that: std::thread::available_parallelism. How neat!

#RustLang #thread

hywan,
@hywan@fosstodon.org avatar

#weld

ˋweld-object is now able to disassemble x86 instructions when debugging an object file (with the ˋdebug-x86 feature turned on), https://github.com/Hywan/weld/commit/043e16d731108ac1a7cade05de209a18e0591959.

It’s really handy :-)!

It’s based on the excellent iced-x86 crate, https://github.com/icedland/iced.

❓📣 If you know an equivalent for ARM, please let me know!

#RustLang #disassembly #assembly #x86

hywan,
@hywan@fosstodon.org avatar

#weld

weld now takes a --target &lt;triple&gt; argument. weld is designed to be cross-platform entirely, hence it's legit to be able to specify a target triple, https://github.com/Hywan/weld/commit/b7fde2dbf9fb523dccb041063bfd030967f5e814.

Based on the provided target triple, weld will use a particular linker strategy (e.g Elf, MachO, Coff, Wasm etc.).

$ weld --target x86-64-unknown-linux <input> -o <output> # = Elf

$ weld --target aarch64-apple-darwin <input> -o <output> # = MachO

Cool huh?

#RustLang #TargetLexicon #CrossPlatform #linker

hywan,
@hywan@fosstodon.org avatar

#weld

My biggest difficulty with linkers is… understanding their errors! I know I'm not alone.

That's why weld must be exemplary on errors, period.

Please welcome weld-errors, https://github.com/Hywan/weld/compare/b7fde2dbf9fb523dccb041063bfd030967f5e814...4a41bd666c6dc8b76a59bf7c127247312e84127e.

What's new?

  • Any error contains a code, a (formatted) message, and a help message,
  • weld pretty prints those errors,
  • weld --explain &lt;error_code&gt; gives detailed diagnostics,
  • Automatic awesome documentation.

See the screenshots.

#RustLang #error #diagnostics #friendly #linker

Example of how to declare an error with the
Documentation of an error type generated with the
Generated documentation that contains all the error diagnostics!

hywan,
@hywan@fosstodon.org avatar

#weld

After a long hibernation, the project is now awaken. I've updated the dependencies, and I've committed pending changes on the Read and Write traits of the weld-object crate.

Nothing fancy, but I can work on it now, https://github.com/Hywan/weld/commit/62c439fb394ced6699169b78ec43b27d3b757e19.

#RustLang #linker

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • rust
  • ngwrru68w68
  • magazineikmin
  • Durango
  • ethstaker
  • everett
  • osvaldo12
  • Youngstown
  • slotface
  • khanakhh
  • kavyap
  • thenastyranch
  • DreamBathrooms
  • InstantRegret
  • rosin
  • provamag3
  • tester
  • GTA5RPClips
  • modclub
  • tacticalgear
  • cubers
  • mdbf
  • cisconetworking
  • megavids
  • Leos
  • anitta
  • normalnudes
  • JUstTest
  • lostlight
  • All magazines
    •