Joel Dice (Nov 03 2022 at 20:36): Joel Dice (Nov 03 2022 at 20:36):Excited about all the recent progress in wit-bindgen, wit-component, and wasmtime, I've been experimenting with building and hosting components recently. The only hitch so far has been WASI support for components, and I'd like to check on the status of that.
I found https://github.com/bytecodealliance/wit-bindgen/commit/fc35377b64a66d3c58280ccf36e74085dff15ad5, and I've seen Dan and Alex collaborating on shims from Preview1 to Preview2 and vice-versa, but AFAICT wasmtime-wasi does not yet have Preview2 support, so while it might be (almost?) possible to create a component that uses WASI, it's not yet possible to host one. Is that correct? If so, I'd be interested in helping fill in the missing pieces there.
Pat Hickey (Nov 03 2022 at 20:50):Hi! so, i'm working right now on some foundational stuff for supporting preview 2
Pat Hickey (Nov 03 2022 at 20:52):I don't really have a complete plan figured out and I'm working on some transformations to wasi-common as it exists today to get things in the right direction, but id be up for collaborating if you want to figure out how to divide and conquer with me.
Joel Dice (Nov 03 2022 at 21:03):Sounds great. Would you by chance have time to chat about it after tomorrow's component model meeting? If not, we can figure out another time.
Philpax (Nov 11 2022 at 01:45):Apologies for bumping this, but I'm looking at updating our wasmtime/wit-bindgen (which looks like it'll require us to use the component model for our modules), but the lack of WASI is currently a blocker. This is not a major issue for us as our existing solution works, but it does mean that we can't update and enjoy some of the newer changes (I'm personally keen for the anyhow::Result return types in the generated traits!)
Is there a rough timeframe for when a componentized wasmtime-wasi might be available? No rush or high precision required, just a finger in the air estimate if possible!
Ari Seyhun (Nov 11 2022 at 03:30):I haven't contributed to the project, but I just wanted to let you know that I got it all working yesterday with webassembly components, but there was no documentation and I was figuring out each step one by one.
I had to compile some things with some specific Rust flags that I found when digging through the build files
Dan Gohman (Nov 11 2022 at 03:42):There isn't a specific timeline, but we are working on it. For example, this is a work-in-progress polyfill which implements the preview1 API in terms of wit-based APIs, which is helping us push forward the overall implementation.
Philpax (Nov 11 2022 at 14:07):Ari Seyhun said:
I haven't contributed to the project, but I just wanted to let you know that I got it all working yesterday with webassembly components, but there was no documentation and I was figuring out each step one by one.
I had to compile some things with some specific Rust flags that I found when digging through the build files
Yeah, I was able to get components working, but couldn't progress any further due to our dependency on WASI (or at least specific features of it.) Is that something you were able to overcome?
Dan Gohman said:
There isn't a specific timeline, but we are working on it. For example, this is a work-in-progress polyfill which implements the preview1 API in terms of wit-based APIs, which is helping us push forward the overall implementation.
Understandable - looks like you're doing great work! What's the easiest way to stay subscribed to updates on this? I've been checking the repos and the Zulip, but I'm not sure if there's a specific place for announcements or such.
Pat Hickey (Nov 11 2022 at 17:00):What we have there is very experimental, we have a rough plan for what it will take to land all of that in the wasmtime repo as part of wasi-common
Pat Hickey (Nov 11 2022 at 17:01):like dan says, we don't have any estimate for when that will be production quality. hopefully Q1 but no promises.
Pat Hickey (Nov 11 2022 at 17:01):it will definitely make it into the wasmtime release notes once it lands, and hopefully we will blog about it and stuff too
Philpax (Nov 11 2022 at 18:38):Makes sense - thanks for the answer! I'll keep an eye out until then.
If I only need a subset of functionality from WASI, is it possible to stub out the interface and still compile/run against wasm32-wasi?
Pat Hickey (Nov 11 2022 at 19:23):there is a really small adapter in the wit-bindgen repo at crates/wasi_snapshot_preview1/ that adapts the bare minimum set of wasi preview 1 import functions used by the tests in that repo to just some basic print-debugging functions
Pat Hickey (Nov 11 2022 at 19:23):its not designed to be used outside of that repo, its really just there to make sure we can run tests and debug them with println
Pat Hickey (Nov 11 2022 at 19:24):but if you just want to satisfy the wasi preview1 imports required by std or wasi-libc with something, its in a more usable state than the sunfishcode/preview1.wasm repo
Amit Prasad (Nov 12 2022 at 22:36):Hi all,
I've been trying to figure out how to instantiate and run a simple component, but feel like I'm going in circles. Are there any examples / example repositories out there that show a minimum example of using wit-component / wasmtime::Component in Rust.
Specifically, I'm having trouble with the bindings generated using wit_bindgen_host_wasmtime_rust::generate -- I'm not exactly sure how to take these bindings and actually run the module. Any pointers would be helpful:)
Pat Hickey (Nov 13 2022 at 03:50):There are tests in that repo but they lean on a fair amount of macros to work. That the best we have right now, there is some similar stuff going on in the preview1.wasm repo as well. We haven’t spent much time working on docs or examples yet because it’s still actively changing
Amit Prasad (Nov 13 2022 at 07:44):Pat Hickey said:
There are tests in that repo but they lean on a fair amount of macros to work. That the best we have right now, there is some similar stuff going on in the preview1.wasm repo as well. We haven’t spent much time working on docs or examples yet because it’s still actively changing
Gotcha. I actually did end up getting it working using cargo-expand to figure out what was going on :).
For anyone in the future -- I found this example very helpful: https://github.com/bytecodealliance/wasm-tools/tree/main/crates/wasm-compose/example
Amit Prasad (Nov 13 2022 at 07:49):just a quick question -- does wasmtime-wasi support adding to a wasmtime::component::Linker? Or is this something I would need to do manually?
Pat Hickey (Nov 14 2022 at 16:41):There’s no component code in wasmtime-wasi yet, all of this is very experimental still, but when it lands as production ready code there will be a way to add it to a component Linker just like preview 1 / witx based wasi adds to a core Linker
Amit Prasad (Nov 16 2022 at 02:29):Sorry to keep bothering you about this, but I had a question about using the wasi_snapshot_preview1 stub (https://github.com/bytecodealliance/wit-bindgen/tree/main/crates/wasi_snapshot_preview1)
I see in a doc comment for wit-component that there's meant to be some special way to compile adapter modules. Do you have any examples of doing this?
Amit Prasad (Nov 16 2022 at 04:04):Ah, I see that I should have RUSTFLAGS="-Clink-args=--import-memory -Clink-args=-zstack-size=0", but no dice on trying to pass that into wit-component hmm...
Ari Seyhun (Nov 16 2022 at 04:49):@Amit Prasad I managed to do this a few days ago, and I wrote some docs for myself. Here's what I wrote down:
After building this module to wasm32-wasi, you need to use the wit-component cli to convert the wasm to a component.
wit-component ./target/wasm32-wasi/release/counter.wasm --adapt ../../../../bytecodealliance/wit-bindgen/target/wasm32-unknown-unknown/release/wasi_snapshot_preview1.wasm -o ./counter.component.wasm
wasi_snapshot_preview1.wasm comes from the wasi_snapshot_preview1 crate, and should be built with the following command:
RUSTFLAGS="-Clink-args=--import-memory -Clink-args=-zstack-size=0" cargo build --release --target=wasm32-unknown-unknown -p wasi_snapshot_preview1
Note, wasi_snapshot_preview1 might need to be modified to have the testwasi.wit import removed, since its not needed.
Amit Prasad (Nov 16 2022 at 20:41):Hmm. Whenever I try to do as you said, Ari, I get a
failed to encode a component from module `./target/wasm32-wasi/release/guest.wasm`
Caused by:
0: failed to reduce input adapter module to its minimal size
1: unsupported section found in adapter module
Looking at the WebAssembly text for guest.wasm (which has been compiled to wasm32-wasi), and wasi_snapshot_preview.wasm (wasm32-unknown-unknown), I see:
guest.wasm:
(import "wasi_snapshot_preview1" "fd_write" (func $A_VERY_LONG_SIGNATURE))
(import "wasi_snapshot_preview1" "environ_get" (func $A_VERY_LONG_SIGNATURE))
(import "wasi_snapshot_preview1" "environ_sizes_get" (func $A_VERY_LONG_SIGNATURE))
(import "wasi_snapshot_preview1" "proc_exit" (func $A_VERY_LONG_SIGNATURE))
...
(Some exports that I've defined)
wasi_snapshot_preview1.wasm
(import "testwasi" "log" (func $A_VERY_LONG_SIGNATURE))
(import "testwasi" "log-err" (func $A_VERY_LONG_SIGNATURE))
...
(export "args_get" (func $args_get))
(export "args_sizes_get" (func $args_sizes_get))
(export "clock_time_get" (func $clock_time_get))
(export "fd_write" (func $fd_write))
(export "fd_seek" (func $fd_seek))
(export "fd_close" (func $fd_close))
(export "proc_exit" (func $proc_exit))
(export "fd_fdstat_get" (func $fd_fdstat_get))
(export "environ_sizes_get" (func $args_sizes_get))
(export "environ_get" (func $args_get))
(export "cabi_realloc" (func $cabi_realloc))
(export "__data_end" (global 1))
(export "__heap_base" (global 2))
The only "unsatisfied imports" I can see are the testwasi::{log, log-err} ones, which are provided by the runtime. I've tried running wit-component with --import to the testwasi.wit, but no luck on that front either.
The "unsupported section in adapter module" error leads me to believe I'm compiling the wasi_snapshot_preview1 wrong somehow. Any ideas?
Amit Prasad (Nov 16 2022 at 20:54):Further inspection: adding the RUSTFLAGS env to my cargo command doesn't seem to affect the output wasm at all. However, cargo build --verbose shows me that the rustc commands include: -Clink-args=--import-memory -Clink-args=-zstack-size=0
Alex Crichton (Nov 16 2022 at 20:58):the issue here is the adapter module has an unsupported section, probably either a memory or a data section, since adapters right now can't have locally defined memories or data
Alex Crichton (Nov 16 2022 at 20:59):if you're writing your own adapter you'll need to update the source to where it only uses an imported memory and it doesn't have any of its own data segments
Amit Prasad (Nov 16 2022 at 21:12):Alex Crichton said:
if you're writing your own adapter you'll need to update the source to where it only uses an imported memory and it doesn't have any of its own data segments
I'm using the wasi_snapshot_preview1 crate from the wit-bindgen repo. I see two (data) sections in the compiled WASM, and I also see that it imports (import "env" "memory" but proceeds to grow that in a function that seems to be labelled malloc..Allocator
Does this mean I need to manually edit my WASM file to remove these data sections?
Alex Crichton (Nov 16 2022 at 21:14):You shouldn't need to edit the wasm, are you compiling with release mode?
Amit Prasad (Nov 16 2022 at 21:16):Alex Crichton said:
You shouldn't need to edit the wasm, are you compiling with release mode?
Yep:
RUSTFLAGS="-Clink-args=--import-memory -Clink-args=-zstack-size=0" cargo build --release --target=wasm32-unknown-unknown --verbose
can you inspect the wasm and see what the errant section is?
Amit Prasad (Nov 16 2022 at 21:34):Alex Crichton said:
can you inspect the wasm and see what the errant section is?
There's 3 sections that are suspect at the bottom of the file, but I'm not familiar enough with the restrictions to be sure:
(elem $e0 (i32.const 1) $_ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u32$GT$3fmt17h88dfc9d9c4113c02E $_ZN3std5alloc24default_alloc_error_hook17hc8ab192ba9aa1aafE $_ZN4core3ptr100drop_in_place$LT$$RF$mut$u20$std..io..Write..write_fmt..Adapter$LT$alloc..vec..Vec$LT$u8$GT$$GT$$GT$17h78793134c92f917cE $_ZN50_$LT$$RF$mut$u20$W$u20$as$u20$core..fmt..Write$GT$9write_str17hd5c4483eeaa72ff3E $_ZN50_$LT$$RF$mut$u20$W$u20$as$u20$core..fmt..Write$GT$10write_char17h1c5b4e9434b25fcfE $_ZN50_$LT$$RF$mut$u20$W$u20$as$u20$core..fmt..Write$GT$9write_fmt17h3222e8521d113317E $_ZN4core3ptr226drop_in_place$LT$std..error..$LT$impl$u20$core..convert..From$LT$alloc..string..String$GT$$u20$for$u20$alloc..boxed..Box$LT$dyn$u20$std..error..Error$u2b$core..marker..Send$u2b$core..marker..Sync$GT$$GT$..from..StringError$GT$17h994e3ac4e313ea83E $_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17hc9db988414a5dba8E $_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17h4500e4d02ebf2b65E $_ZN4core3ptr70drop_in_place$LT$std..panicking..begin_panic_handler..PanicPayload$GT$17h2e7bd6e17b28fee8E $_ZN90_$LT$std..panicking..begin_panic_handler..PanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$8take_box17h5b55a07cbffecc39E $_ZN90_$LT$std..panicking..begin_panic_handler..PanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$3get17h9c66d8b85c21797fE $_ZN93_$LT$std..panicking..begin_panic_handler..StrPanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$8take_box17h35a36835c420c8e2E $_ZN93_$LT$std..panicking..begin_panic_handler..StrPanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$3get17h9bbdbf24fa06448aE $_ZN4core3ops8function6FnOnce9call_once17h2f0f4071af505877E $_ZN4core3ptr102drop_in_place$LT$$RF$core..iter..adapters..copied..Copied$LT$core..slice..iter..Iter$LT$u8$GT$$GT$$GT$17ha30bd7eefcc06f5fE $_ZN36_$LT$T$u20$as$u20$core..any..Any$GT$7type_id17h892b8673ce75b752E)
(data $d0 (i32.const 0) "\03\00\00\00\04\00\00\00\04\00\00\00\04\00\00\00\05\00\00\00\06\00\00\00called `Option::unwrap()` on a `None` valuememory allocation of bytes failed\0a\00\00C\00\00\00\15\00\00\00X\00\00\00\0e\00\00\00library/std/src/alloc.rsx\00\00\00\18\00\00\00R\01\00\00\09\00\00\00library/std/src/panicking.rs\a0\00\00\00\1c\00\00\00F\02\00\00\1f\00\00\00\a0\00\00\00\1c\00\00\00G\02\00\00\1e\00\00\00\07\00\00\00\0c\00\00\00\04\00\00\00\08\00\00\00\03\00\00\00\08\00\00\00\04\00\00\00\09\00\00\00\0a\00\00\00\10\00\00\00\04\00\00\00\0b\00\00\00\0c\00\00\00\03\00\00\00\08\00\00\00\04\00\00\00\0d\00\00\00\0e\00\00\00library/alloc/src/raw_vec.rscapacity overflow\00\00\00@\01\00\00\11\00\00\00$\01\00\00\1c\00\00\00\06\02\00\00\05\00\00\00\10\00\00\00\00\00\00\00\01\00\00\00\11\00\00\0000010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899")
(data $d1 (i32.const 580) "ONLY REPEATED \00"))
No other data / memory sections that I can see
Amit Prasad (Nov 16 2022 at 21:37):Oh hmm.... something seems to have fixed itself. Let me see if this works
Amit Prasad (Nov 16 2022 at 21:49):Ok.. Seems like something to do with my cargo cache and/or some features in the wit-bindgen-guest-rust being enabled was screwing up the compilation process
Now, just have to figure out why it says that the module doesn't export a cabi_realloc function and we're golden
Amit Prasad (Nov 16 2022 at 21:52):Ah, it was gated behind a feature.
Amit Prasad (Nov 16 2022 at 21:53):Whoo! everything seems to work. Thanks all.
Elia Bieri (Mar 05 2023 at 12:21):Pat Hickey said:
There’s no component code in wasmtime-wasi yet, all of this is very experimental still, but when it lands as production ready code there will be a way to add it to a component Linker just like preview 1 / witx based wasi adds to a core Linker
Are there any updates on this?
Pat Hickey (Mar 06 2023 at 01:18):Still a little ways from landing. I hope we can get it in by the end of March?
Pat Hickey (Mar 06 2023 at 01:20):Last month we ran into a handful of fun problems we never saw coming. The remaining todo list is pretty reasonable but hard to say what’s lurking
Elia Bieri (Mar 06 2023 at 10:47):Thanks for taking the time for answering.
Do you have any GitHub Issue of sorts that you could point me to track the progress?
I've got to make some decisions for an upcoming project and I would like to stay in the loop.
Last updated: Nov 22 2024 at 17:03 UTC