Excited about all the recent progress in wit-bindgen
, wit-component
, and wasmtime
, I've been experimenting with building and hosting components recently. The only hitch so far has been WASI support for components, and I'd like to check on the status of that.
I found https://github.com/bytecodealliance/wit-bindgen/commit/fc35377b64a66d3c58280ccf36e74085dff15ad5, and I've seen Dan and Alex collaborating on shims from Preview1 to Preview2 and vice-versa, but AFAICT wasmtime-wasi
does not yet have Preview2 support, so while it might be (almost?) possible to create a component that uses WASI, it's not yet possible to host one. Is that correct? If so, I'd be interested in helping fill in the missing pieces there.
Hi! so, i'm working right now on some foundational stuff for supporting preview 2
I don't really have a complete plan figured out and I'm working on some transformations to wasi-common as it exists today to get things in the right direction, but id be up for collaborating if you want to figure out how to divide and conquer with me.
Sounds great. Would you by chance have time to chat about it after tomorrow's component model meeting? If not, we can figure out another time.
Apologies for bumping this, but I'm looking at updating our wasmtime
/wit-bindgen
(which looks like it'll require us to use the component model for our modules), but the lack of WASI is currently a blocker. This is not a major issue for us as our existing solution works, but it does mean that we can't update and enjoy some of the newer changes (I'm personally keen for the anyhow::Result
return types in the generated traits!)
Is there a rough timeframe for when a componentized wasmtime-wasi
might be available? No rush or high precision required, just a finger in the air estimate if possible!
I haven't contributed to the project, but I just wanted to let you know that I got it all working yesterday with webassembly components, but there was no documentation and I was figuring out each step one by one.
I had to compile some things with some specific Rust flags that I found when digging through the build files
There isn't a specific timeline, but we are working on it. For example, this is a work-in-progress polyfill which implements the preview1 API in terms of wit-based APIs, which is helping us push forward the overall implementation.
Ari Seyhun said:
I haven't contributed to the project, but I just wanted to let you know that I got it all working yesterday with webassembly components, but there was no documentation and I was figuring out each step one by one.
I had to compile some things with some specific Rust flags that I found when digging through the build files
Yeah, I was able to get components working, but couldn't progress any further due to our dependency on WASI (or at least specific features of it.) Is that something you were able to overcome?
Dan Gohman said:
There isn't a specific timeline, but we are working on it. For example, this is a work-in-progress polyfill which implements the preview1 API in terms of wit-based APIs, which is helping us push forward the overall implementation.
Understandable - looks like you're doing great work! What's the easiest way to stay subscribed to updates on this? I've been checking the repos and the Zulip, but I'm not sure if there's a specific place for announcements or such.
What we have there is very experimental, we have a rough plan for what it will take to land all of that in the wasmtime repo as part of wasi-common
like dan says, we don't have any estimate for when that will be production quality. hopefully Q1 but no promises.
it will definitely make it into the wasmtime release notes once it lands, and hopefully we will blog about it and stuff too
Makes sense - thanks for the answer! I'll keep an eye out until then.
If I only need a subset of functionality from WASI, is it possible to stub out the interface and still compile/run against wasm32-wasi
?
there is a really small adapter in the wit-bindgen repo at crates/wasi_snapshot_preview1/ that adapts the bare minimum set of wasi preview 1 import functions used by the tests in that repo to just some basic print-debugging functions
its not designed to be used outside of that repo, its really just there to make sure we can run tests and debug them with println
but if you just want to satisfy the wasi preview1 imports required by std or wasi-libc with something, its in a more usable state than the sunfishcode/preview1.wasm repo
Hi all,
I've been trying to figure out how to instantiate and run a simple component, but feel like I'm going in circles. Are there any examples / example repositories out there that show a minimum example of using wit-component / wasmtime::Component in Rust.
Specifically, I'm having trouble with the bindings generated using wit_bindgen_host_wasmtime_rust::generate -- I'm not exactly sure how to take these bindings and actually run the module. Any pointers would be helpful:)
There are tests in that repo but they lean on a fair amount of macros to work. That the best we have right now, there is some similar stuff going on in the preview1.wasm repo as well. We haven’t spent much time working on docs or examples yet because it’s still actively changing
Pat Hickey said:
There are tests in that repo but they lean on a fair amount of macros to work. That the best we have right now, there is some similar stuff going on in the preview1.wasm repo as well. We haven’t spent much time working on docs or examples yet because it’s still actively changing
Gotcha. I actually did end up getting it working using cargo-expand to figure out what was going on :).
For anyone in the future -- I found this example very helpful: https://github.com/bytecodealliance/wasm-tools/tree/main/crates/wasm-compose/example
just a quick question -- does wasmtime-wasi
support adding to a wasmtime::component::Linker
? Or is this something I would need to do manually?
There’s no component code in wasmtime-wasi yet, all of this is very experimental still, but when it lands as production ready code there will be a way to add it to a component Linker just like preview 1 / witx based wasi adds to a core Linker
Sorry to keep bothering you about this, but I had a question about using the wasi_snapshot_preview1 stub (https://github.com/bytecodealliance/wit-bindgen/tree/main/crates/wasi_snapshot_preview1)
I see in a doc comment for wit-component
that there's meant to be some special way to compile adapter modules. Do you have any examples of doing this?
Ah, I see that I should have RUSTFLAGS="-Clink-args=--import-memory -Clink-args=-zstack-size=0"
, but no dice on trying to pass that into wit-component
hmm...
@Amit Prasad I managed to do this a few days ago, and I wrote some docs for myself. Here's what I wrote down:
After building this module to wasm32-wasi, you need to use the wit-component cli to convert the wasm to a component.
wit-component ./target/wasm32-wasi/release/counter.wasm --adapt ../../../../bytecodealliance/wit-bindgen/target/wasm32-unknown-unknown/release/wasi_snapshot_preview1.wasm -o ./counter.component.wasm
wasi_snapshot_preview1.wasm comes from the wasi_snapshot_preview1
crate, and should be built with the following command:
RUSTFLAGS="-Clink-args=--import-memory -Clink-args=-zstack-size=0" cargo build --release --target=wasm32-unknown-unknown -p wasi_snapshot_preview1
Note, wasi_snapshot_preview1 might need to be modified to have the testwasi.wit import removed, since its not needed.
Hmm. Whenever I try to do as you said, Ari, I get a
failed to encode a component from module `./target/wasm32-wasi/release/guest.wasm`
Caused by:
0: failed to reduce input adapter module to its minimal size
1: unsupported section found in adapter module
Looking at the WebAssembly text for guest.wasm
(which has been compiled to wasm32-wasi), and wasi_snapshot_preview.wasm
(wasm32-unknown-unknown), I see:
guest.wasm
:
(import "wasi_snapshot_preview1" "fd_write" (func $A_VERY_LONG_SIGNATURE))
(import "wasi_snapshot_preview1" "environ_get" (func $A_VERY_LONG_SIGNATURE))
(import "wasi_snapshot_preview1" "environ_sizes_get" (func $A_VERY_LONG_SIGNATURE))
(import "wasi_snapshot_preview1" "proc_exit" (func $A_VERY_LONG_SIGNATURE))
...
(Some exports that I've defined)
wasi_snapshot_preview1.wasm
(import "testwasi" "log" (func $A_VERY_LONG_SIGNATURE))
(import "testwasi" "log-err" (func $A_VERY_LONG_SIGNATURE))
...
(export "args_get" (func $args_get))
(export "args_sizes_get" (func $args_sizes_get))
(export "clock_time_get" (func $clock_time_get))
(export "fd_write" (func $fd_write))
(export "fd_seek" (func $fd_seek))
(export "fd_close" (func $fd_close))
(export "proc_exit" (func $proc_exit))
(export "fd_fdstat_get" (func $fd_fdstat_get))
(export "environ_sizes_get" (func $args_sizes_get))
(export "environ_get" (func $args_get))
(export "cabi_realloc" (func $cabi_realloc))
(export "__data_end" (global 1))
(export "__heap_base" (global 2))
The only "unsatisfied imports" I can see are the testwasi::{log, log-err}
ones, which are provided by the runtime. I've tried running wit-component
with --import
to the testwasi.wit
, but no luck on that front either.
The "unsupported section in adapter module" error leads me to believe I'm compiling the wasi_snapshot_preview1
wrong somehow. Any ideas?
Further inspection: adding the RUSTFLAGS
env to my cargo command doesn't seem to affect the output wasm at all. However, cargo build --verbose
shows me that the rustc
commands include: -Clink-args=--import-memory -Clink-args=-zstack-size=0
the issue here is the adapter module has an unsupported section, probably either a memory or a data section, since adapters right now can't have locally defined memories or data
if you're writing your own adapter you'll need to update the source to where it only uses an imported memory and it doesn't have any of its own data segments
Alex Crichton said:
if you're writing your own adapter you'll need to update the source to where it only uses an imported memory and it doesn't have any of its own data segments
I'm using the wasi_snapshot_preview1
crate from the wit-bindgen
repo. I see two (data)
sections in the compiled WASM, and I also see that it imports (import "env" "memory"
but proceeds to grow that in a function that seems to be labelled malloc..Allocator
Does this mean I need to manually edit my WASM file to remove these data sections?
You shouldn't need to edit the wasm, are you compiling with release mode?
Alex Crichton said:
You shouldn't need to edit the wasm, are you compiling with release mode?
Yep:
RUSTFLAGS="-Clink-args=--import-memory -Clink-args=-zstack-size=0" cargo build --release --target=wasm32-unknown-unknown --verbose
can you inspect the wasm and see what the errant section is?
Alex Crichton said:
can you inspect the wasm and see what the errant section is?
There's 3 sections that are suspect at the bottom of the file, but I'm not familiar enough with the restrictions to be sure:
(elem $e0 (i32.const 1) $_ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u32$GT$3fmt17h88dfc9d9c4113c02E $_ZN3std5alloc24default_alloc_error_hook17hc8ab192ba9aa1aafE $_ZN4core3ptr100drop_in_place$LT$$RF$mut$u20$std..io..Write..write_fmt..Adapter$LT$alloc..vec..Vec$LT$u8$GT$$GT$$GT$17h78793134c92f917cE $_ZN50_$LT$$RF$mut$u20$W$u20$as$u20$core..fmt..Write$GT$9write_str17hd5c4483eeaa72ff3E $_ZN50_$LT$$RF$mut$u20$W$u20$as$u20$core..fmt..Write$GT$10write_char17h1c5b4e9434b25fcfE $_ZN50_$LT$$RF$mut$u20$W$u20$as$u20$core..fmt..Write$GT$9write_fmt17h3222e8521d113317E $_ZN4core3ptr226drop_in_place$LT$std..error..$LT$impl$u20$core..convert..From$LT$alloc..string..String$GT$$u20$for$u20$alloc..boxed..Box$LT$dyn$u20$std..error..Error$u2b$core..marker..Send$u2b$core..marker..Sync$GT$$GT$..from..StringError$GT$17h994e3ac4e313ea83E $_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17hc9db988414a5dba8E $_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17h4500e4d02ebf2b65E $_ZN4core3ptr70drop_in_place$LT$std..panicking..begin_panic_handler..PanicPayload$GT$17h2e7bd6e17b28fee8E $_ZN90_$LT$std..panicking..begin_panic_handler..PanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$8take_box17h5b55a07cbffecc39E $_ZN90_$LT$std..panicking..begin_panic_handler..PanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$3get17h9c66d8b85c21797fE $_ZN93_$LT$std..panicking..begin_panic_handler..StrPanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$8take_box17h35a36835c420c8e2E $_ZN93_$LT$std..panicking..begin_panic_handler..StrPanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$3get17h9bbdbf24fa06448aE $_ZN4core3ops8function6FnOnce9call_once17h2f0f4071af505877E $_ZN4core3ptr102drop_in_place$LT$$RF$core..iter..adapters..copied..Copied$LT$core..slice..iter..Iter$LT$u8$GT$$GT$$GT$17ha30bd7eefcc06f5fE $_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17h892b8673ce75b752E)
(data $d0 (i32.const 0) "\03\00\00\00\04\00\00\00\04\00\00\00\04\00\00\00\05\00\00\00\06\00\00\00called `Option::unwrap()` on a `None` valuememory allocation of bytes failed\0a\00\00C\00\00\00\15\00\00\00X\00\00\00\0e\00\00\00library/std/src/alloc.rsx\00\00\00\18\00\00\00R\01\00\00\09\00\00\00library/std/src/panicking.rs\a0\00\00\00\1c\00\00\00F\02\00\00\1f\00\00\00\a0\00\00\00\1c\00\00\00G\02\00\00\1e\00\00\00\07\00\00\00\0c\00\00\00\04\00\00\00\08\00\00\00\03\00\00\00\08\00\00\00\04\00\00\00\09\00\00\00\0a\00\00\00\10\00\00\00\04\00\00\00\0b\00\00\00\0c\00\00\00\03\00\00\00\08\00\00\00\04\00\00\00\0d\00\00\00\0e\00\00\00library/alloc/src/raw_vec.rscapacity overflow\00\00\00@\01\00\00\11\00\00\00$\01\00\00\1c\00\00\00\06\02\00\00\05\00\00\00\10\00\00\00\00\00\00\00\01\00\00\00\11\00\00\0000010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899")
(data $d1 (i32.const 580) "ONLY REPEATED \00"))
No other data / memory sections that I can see
Oh hmm.... something seems to have fixed itself. Let me see if this works
Ok.. Seems like something to do with my cargo cache and/or some features in the wit-bindgen-guest-rust
being enabled was screwing up the compilation process
Now, just have to figure out why it says that the module doesn't export a cabi_realloc
function and we're golden
Ah, it was gated behind a feature.
Whoo! everything seems to work. Thanks all.
Pat Hickey said:
There’s no component code in wasmtime-wasi yet, all of this is very experimental still, but when it lands as production ready code there will be a way to add it to a component Linker just like preview 1 / witx based wasi adds to a core Linker
Are there any updates on this?
Still a little ways from landing. I hope we can get it in by the end of March?
Last month we ran into a handful of fun problems we never saw coming. The remaining todo list is pretty reasonable but hard to say what’s lurking
Thanks for taking the time for answering.
Do you have any GitHub Issue of sorts that you could point me to track the progress?
I've got to make some decisions for an upcoming project and I would like to stay in the loop.
Last updated: Dec 23 2024 at 12:05 UTC