Stream: wasi

Topic: WASIP3 Update

Bailey Hayes (Jan 15 2025 at 16:20):

Late last year we decided that we'd like to start running a standup style update and to use the Ship WASIp3 project board (see topic #wasi > WASIp3 project board @ 💬 for background) to coordinate work.

Let's kick things off with an async style standup for this week!

Bailey Hayes (Jan 15 2025 at 16:22):

Folks that have active work streams, will you post a quick update?
cc @Victor Adossi @Roman Volosatovs @Joel Dice

Bailey Hayes (Jan 15 2025 at 16:32):

Jan 15, 2025

Bailey Hayes: WASI clocks and random 0.3.0-draft WIT definitions have been reviewed and merged. We adjusted the API for wasi-clocks now that fn's are async, see: https://github.com/WebAssembly/wasi-clocks/blob/main/wit-0.3.0-draft/monotonic-clock.wit#L34-L44

We decided to rev the since versions as 0.3.0 will make breaking changes to the 0.2.x series. Expect all 0.3.0-draft WIT's to align to this. Once @Roman Volosatovs PR's for filesystem and sockets merge, wasi-cli is the next target. I believe we're very close to having a draft for WASIP2 interfaces (http draft already exists but will be updated once deps land).

Once the above is complete, adding support for WASIP3 interfaces in wasmtime and JCO can begin

Bailey Hayes (Jan 15 2025 at 16:40):

Items Ready for Work

If you're interested in jumping in, I see a couple items that could begin now:

• Add a non-blocking attribute issue on board. Before starting this one, check-in with @Luke Wagner
• Doc work in component model book

When picking up a new issue on the board, set yourself as the assignee.

Joel Dice (Jan 15 2025 at 16:41):

Async/future/stream/error-context support has been merged into wasm-tools and wit-bindgen and is available in their latest releases. Still iterating on the Wasmtime PR, knocking out todo items one by one. Hoping to move that one to "ready-for-review" by the end of the month. Meanwhile, it is feature-complete (modulo the error-context work Victor is doing, which I'm reviewing now), so feel free to clone the branch, kick the tires, and start adding WASIp3 support to wasmtime-wasi and wasmtime-wasi-http.

Victor Adossi (Jan 15 2025 at 16:41):

Hey all, here's what I'm working on right now:

• P3 Error impl
  • Progress on implementation of Error, working implementation right now, w/ a PR to wit-bindgen-rt and a draft to Joel's wasmtime branch (immense thanks to @Joel Dice who walked me through this)
• Jco async host imports
  • Tests for CLI stuff and HTTP
  • Adding more tests for the other new implementations
  • Doing a bit of exploration on how we can build asyncify faster (unlikely to pan out)

CC @Calvin Prewitt who is doing the actual bulk of the work of jco host async imports

Roman Volosatovs (Jan 16 2025 at 14:50):

wasi-sockets and wasi-filesystem WIT update PRs are ready for review

will work on removing wasi:io/error.error dep from wasi:http and after that start WASI implementation in Wasmtime

Dave Bakker (badeend) (Jan 16 2025 at 14:52):

Thanks. I've got a few remarks on the wasi-sockets part. Hope to get it written down soon

Pat Hickey (Jan 16 2025 at 16:05):

What’s the status of implementing P3 streams and futures in wasmtime::component? That’s a prerequisite to a wasi impl, correct?

Joel Dice (Jan 16 2025 at 16:14):

@Pat Hickey see my update above; that's feature-complete and has decent test coverage. Working to get that PR in a reviewable state (hopefully in the next week or two), but meanwhile you can use my PR branch.

Pat Hickey (Jan 16 2025 at 16:28):

Thanks. I will take a look. I’ve been factoring out wasi-io from wasmtime-wasi so that at least that much can be built nostd, idk how that will intersect with these new streams and futures but hopefully they can plug together

Roman Volosatovs (Jan 17 2025 at 11:59):

Just a heads-up, I've started working on stream types with no <T>, added this ticket to the board: https://github.com/orgs/bytecodealliance/projects/16?pane=issue&itemId=94157967
i.e. implementing the spec change from https://github.com/WebAssembly/component-model/pull/440 in wasm-tools and wit-bindgen

Roman Volosatovs (Jan 17 2025 at 12:01):

Opened a draft PR to wasi:cli for 0.3.0 https://github.com/WebAssembly/wasi-cli/pull/52
a few smaller-scoped wasi:http updates to 0.3.0 https://github.com/WebAssembly/wasi-http/pull/143 https://github.com/WebAssembly/wasi-http/pull/142
wasi:random fix https://github.com/WebAssembly/wasi-random/pull/52

Roman Volosatovs (Jan 17 2025 at 12:50):

Added a draft PR for stream of unit: https://github.com/bytecodealliance/wasm-tools/pull/1978

Roman Volosatovs (Jan 21 2025 at 15:05):

started the wasmtime WASI 0.3 impl https://github.com/bytecodealliance/wasmtime/pull/10061

Roman Volosatovs (Jan 21 2025 at 16:47):

and now with actual async: https://github.com/bytecodealliance/wasmtime/pull/10063/commits/171a532910790379ebbfd85ddbb62b6f60a1fbff

component build fails with:

  thread 'main' panicked at crates/test-programs/artifacts/build.rs:176:10:
  module can be translated to a component: failed to validate component output

  Caused by:
      canonical option `async` requires the component model async feature (at offset 0x82da1a)

Will try basing on async branch

btw, I've noticed that wit-bindgen-rt dependency, which appears to be used by async is missing out-of-the-box.
I believe deps are normally reexported by wit-bindgen, but in this case it appears that macro attempts to import from ::wit-bindgen-rt
cc @Joel Dice idk, if that's a known/fixed issue or do you want me to take a look?

Roman Volosatovs (Jan 21 2025 at 16:59):

filed https://github.com/bytecodealliance/wit-bindgen/issues/1135, working on a fix

Roman Volosatovs (Jan 21 2025 at 17:35):

this seems to have fixed the issue: https://github.com/bytecodealliance/wit-bindgen/pull/1136

Joel Dice (Jan 21 2025 at 23:43):

FYI, I'm starting to work on wit-bindgen C support for async/streams/futures, which will unblock wasip3 support in wasi-libc and the toolchains which depend on it.

Roman Volosatovs (Jan 22 2025 at 13:31):

I've rebased my Wasmtime WASI changes on latest async from @Joel Dice in https://github.com/rvolosatovs/wasmtime/tree/async-wasi

wasi:clocks@0.3.0 test fails for now, but looks like we're pretty close to getting this working end-to-end:

on https://github.com/rvolosatovs/wasmtime/commit/8ea812e04a77ab7a396bd74876d8c8ef6f0b017b

---- async_::preview3_sleep stdout ----
preopen: TempDir { path: "/var/folders/bq/thy1_b7x29l7s2wqw39r62yw0000gn/T/wasi_components_preview3_sleep.component_cp7fbR" }
thread 'async_::preview3_sleep' panicked at /Users/rvolosatovs/src/github.com/bytecodealliance/wasmtime/crates/wasi/src/runtime.rs:108:15:
Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- sync::preview3_sleep stdout ----
preopen: TempDir { path: "/var/folders/bq/thy1_b7x29l7s2wqw39r62yw0000gn/T/wasi_components_preview3_sleep.component_bQriUp" }
thread 'sync::preview3_sleep' panicked at /Users/rvolosatovs/src/github.com/bytecodealliance/wasmtime/crates/wasmtime/src/runtime/component/concurrent.rs:534:54:
called `Option::unwrap()` on a `None` value


failures:
    async_::preview3_sleep
    sync::preview3_sleep

I'll start digging :)

Roman Volosatovs (Jan 22 2025 at 18:44):

I got wasi:clocks@0.3.0 test passing with async runtime in https://github.com/rvolosatovs/wasmtime/commit/c3406b2cb02a6013add16867a8c6debad8bbef39
thanks @Joel Dice for the help!

I noticed the same "current task missing" panic when attempting to run the same test with async_support(false), i.e. using sync host bindings. It makes sense, since in that case none of the async machinery seem to be used
What is the current thinking around that - are we planning to support wasip3 on hosts without async_support at all?

Joel Dice (Jan 22 2025 at 18:48):

@Alex Crichton and I discussed this a while ago. We don't think WASIp3 or the component model async ABI make sense to support in Wasmtime with async_support(false), and we're actually considering deprecating Wasmtime's component model sync APIs (i.e. making async_support(true) the default and, eventually, only option for running components).

Roman Volosatovs (Jan 22 2025 at 18:50):

That sounds good to me and simplifies this effort - I won't be adding the sync bindings to wasip3 interfaces at all then :)

Pat Hickey (Jan 22 2025 at 18:53):

thanks Joel, that sounds right - we'll have to see about deprecating the sync stuff for p2 we already provide (hacky as it is, it does actually meet some users needs) but I agree we shouldn't worry about sync interfaces to p3 or async CM

Juniper Tyree (Jan 23 2025 at 06:42):

Do I understand correctly that the plan is that the CM host must support async, but that all components can still be fully sync (if one wants that)? If wasmtime will always use async, it may be useful to support compiling with a very cheap async runtime so that e.g. embedded usecases don’t need to pull in a costly featureful runtime

Pat Hickey (Jan 23 2025 at 18:03):

Yes, you should basically just need tasks (which can be implemented with a library e.g. https://docs.rs/futures/latest/futures/task/index.html), and whatever your platform needs for IO to make a custom executor

Pat Hickey (Jan 23 2025 at 18:04):

I'm working on exactly that for P2 right now, the need is not limited to P3

Pat Hickey (Jan 23 2025 at 18:04):

the existing P2 wasmtime-wasi always runs inside tokio, with the sync bindings it just hides that tokio in the background.

Last updated: Jan 24 2025 at 00:11 UTC