Stream: wasi

Topic: wasm-ld for components

Randy Reddig (Mar 13 2024 at 18:14):

Is there any intention or desire to directly update wasm-ld to emit components directly? I saw https://github.com/alexcrichton/wasm-component-ld and wondered if some or all of it could be backported to wasm-ld.

Randy Reddig (Mar 13 2024 at 18:15):

Asking because I’m looking at the Go wasm linker, which doesn’t use LLVM, to figure out how to modify it to directly emit a component.

Alex Crichton (Mar 13 2024 at 18:19):

At this time, no, I don't think anyone's looking to port this to wasm-ld. The main complication is the wit-component crate which performs the componentization process. While that can certainly be written in any language it's a pretty tricky piece of code to port so it's not one I think anyone wants to port to C++ yet.

Other than that though it's a trivial wrapper so that's the only hard part

Alex Crichton (Mar 13 2024 at 18:19):

(sorry sent soon so updated previous message with what I fully wanted to say)

Randy Reddig (Mar 13 2024 at 18:20):

We want to port that logic to Go.

Randy Reddig (Mar 13 2024 at 18:21):

Essentially this is a requirement for mainline Go to support WASI Preview 2, which means natively supporting the component model.

Alex Crichton (Mar 13 2024 at 18:22):

Sure! In that case the work item is basically the wit-component crate and what it does internally

Randy Reddig (Mar 13 2024 at 18:22):

How much (if any) of what wit-component does could be moved to wit-bindgen?

Alex Crichton (Mar 13 2024 at 18:23):

nothing, unfortunately, because it all operates after the wasm module has been created, whereas wit-bindgen works on things before a wasm module has been created

Alex Crichton (Mar 13 2024 at 18:23):

I'll caution you that wit-component is a notoriously subtle crate and does quite a lot internally, but that being said the inputs and outputs are pretty clear so it's also ripe for a second implementation

Randy Reddig (Mar 13 2024 at 18:23):

What’s the best way to understand the work that wit-component performs?

Randy Reddig (Mar 13 2024 at 18:24):

Is the process documented outside of the Rust implementation?

Alex Crichton (Mar 13 2024 at 18:24):

Heh unfortunately I don't think I have a great answer other than reading the implementation yeah

Alex Crichton (Mar 13 2024 at 18:24):

little of what it does is "standard" per se, but it's becoming sort of defacto standard nowadays

Randy Reddig (Mar 13 2024 at 18:24):

Related: is there a second standalone runtime that supports the CM?

Alex Crichton (Mar 13 2024 at 18:25):

the closest equivalent I'm aware of is jco running components on node

Randy Reddig (Mar 13 2024 at 18:25):

Jco leans on the Rust tooling (warm-tools, wit-component) right?

Alex Crichton (Mar 13 2024 at 18:26):

It does, yes.

Alex Crichton (Mar 13 2024 at 18:27):

jco compiles all the rust tooling itself to wasm and then runs that in node to generate code that runs in node

Alex Crichton (Mar 13 2024 at 18:27):

(sort of self-hosting I believe)

Randy Reddig (Mar 13 2024 at 18:28):

Oh interesting.

Randy Reddig (Mar 13 2024 at 18:29):

Observation: the name mangling and metadata added in wasm-tools component embed and component new are workarounds for limitations in LLVM tooling for C/C++/Rust?

Randy Reddig (Mar 13 2024 at 18:31):

How much of the subtlety/complexity of wit-component has to do with adapters?

Alex Crichton (Mar 13 2024 at 18:32):

sort of and sort of not, the decision with wit-component was to use a core wasm module as the unit of abstraction for creation of a component, so you feed in a core wasm and then out pops a component. Decisions around mangling/metadata were made within that context, everything was about how do we use a core wasm module to represent what a component ought to be.

On one hand that's all a limitation of LLVM in that it can't output components, but it's also an acknowledgement of reality of we didn't want to create a whole brand new "everything" in LLVM centered around components when it's all more-or-less modules anyway.

So sort of a workaround, but also even if it weren't a workaround I think we'd likely settle on this as well

Alex Crichton (Mar 13 2024 at 18:33):

How much of the subtlety/complexity of wit-component has to do with adapters?

My gut would be 20%, so some, but not all

Alex Crichton (Mar 13 2024 at 18:35):

The major tasks are, off the top of my head:

• Deserializing a component into a WIT document - especially with respect to resources this is not easy
• Merging WIT documents together (the core wasm might have multiple pieces in it that all have component metadata)
• Auto-gc'ing WIT in a sense where if you have metadata for importing a/b/c but your module only imports a/b then you don't emit "c" in the output
• Handling circular dependencies between lowering imports and the core wasm module

Randy Reddig (Mar 13 2024 at 18:36):

OK I have the deserializing of a Resolve back into WIT working today

Alex Crichton (Mar 13 2024 at 18:36):

Ah this is a very different operation, it's basically taking a binary component and deserializing it into a Resolve

Randy Reddig (Mar 13 2024 at 18:36):

Ah, OK.

Randy Reddig (Mar 13 2024 at 18:37):

In the case of mainline Go, we’d be able to have all this metadata around anyway, when outputting the .wasm file.

Randy Reddig (Mar 13 2024 at 18:37):

So we wouldn’t need to stash WIT metadata somewhere for post-processing a module

Alex Crichton (Mar 13 2024 at 18:37):

you can see this with wasm-tools component wit -t ./my-wit-folder -- that'll output a wasm component, in its text format, and that's what you would need to parse to get back into the WIT (you'd parse the binary, which you can see with --wasm)

Alex Crichton (Mar 13 2024 at 18:38):

Yeah the "smuggle through custom sections" workaround sounds like you could remove, which would be nice

Randy Reddig (Mar 13 2024 at 18:39):

Since we’re starting with TinyGo, I do need to do some smuggling as a starting point

Randy Reddig (Mar 13 2024 at 18:39):

Do we have sufficient information in the Resolve to embed metadata sections in the generated Go code?

Randy Reddig (Mar 13 2024 at 18:40):

That the TinyGo compiler could add as specific sections to the emitted .wasm file?

Randy Reddig (Mar 13 2024 at 18:40):

The approach I'm thinking is to emulate the wasm-tools component embed step directly, before trying to emulate warm-tools component new

Alex Crichton (Mar 13 2024 at 18:41):

yes that's this function, where all you need is a Resolve and a WorldId and then you can make the custom section

Randy Reddig (Mar 13 2024 at 18:41):

What’s the format of that custom section?

Alex Crichton (Mar 13 2024 at 18:42):

you can find the creation here as well as documentation in that module

Alex Crichton (Mar 13 2024 at 18:42):

aka it's a component with a custom section where the custom section has I think maybe 2 or 3 bytes and that's it

Alex Crichton (Mar 13 2024 at 18:42):

the component is the "big piece"

Randy Reddig (Mar 13 2024 at 18:42):

So it’s the binary representation of wasm-tools component wit -t?

Alex Crichton (Mar 13 2024 at 18:42):

indeed yeah

Randy Reddig (Mar 13 2024 at 18:43):

OK thanks

Last updated: Dec 23 2024 at 12:05 UTC