Stream: wasmtime

Topic: Wasi: Custom files / directories (for async)


view this post on Zulip Léo Gaspard (Jan 11 2021 at 01:12):

Hey! I'm in the process of starting to implement wasi bindings in my wasm project, and am hitting an issue: as I'm going to be in an async world (using wasmtime-async), I'd like to basically wrap all filesystem calls in unblock.
However, I can't figure out a way to do that. I remember a while ago searching (or being told?) that https://github.com/bytecodealliance/wasmtime/pull/1600 would allow that, but it seems that this only did it for stdio, and not for preopened dirs in particular (files can be virtual-based but AFAIU dirs must be explored at context-build time).
Is this me missing something, or an oversight that I should try to write a PR for, adding a preopened_handle that takes as an argument any T: Handle like stdin & co.?

There have been requests to allow more than just raw OS handles to act as stdio in wasi-common (see #939). This PR makes this possible by loosening the requirement of the WasiCtxBuilder to accept a...

view this post on Zulip Léo Gaspard (Jan 12 2021 at 00:08):

Well, I've just sent https://github.com/bytecodealliance/wasmtime/pull/2569 implementing what I want, I'll just start using my fork until someone comments on the PR, though I'd love it if it was approved out of the box :)

This is a follow-up to #1600 ; that also makes it possible to write custom Handles for directories. For my specific case, it is so as to be able to have a wasmtime-async-compatible WASI. I've r...

view this post on Zulip Pat Hickey (Jan 12 2021 at 17:53):

Hey, I guess this is the message I missed. sorry!

view this post on Zulip Pat Hickey (Jan 12 2021 at 17:53):

zulip is sometimes confusing.

view this post on Zulip Pat Hickey (Jan 12 2021 at 17:54):

Anyway, after I land the wasi-c2 rewrite, my next big thing is making it all work with async

view this post on Zulip Pat Hickey (Jan 12 2021 at 17:55):

lucet has async support built-in right now, and @Alex Crichton has an RFC and PR to add async to natively wasmtime.

view this post on Zulip Pat Hickey (Jan 12 2021 at 17:55):

one of our constraints is that we dont want all users of wasi-common (which is, effectively, all users of wasmtime at the moment) to take a dep on tokio (or async-std). so one trick we need is to factor it so that using an async runtime is optional.

view this post on Zulip Pat Hickey (Jan 12 2021 at 17:59):

my rough plan is that i'll use the async-trait proc macro to make the WasiFile and WasiDir traits have async methods. this will probably mean in-lining the parts of the system-interface::FileIoExt that we actually consume into WasiFile and other code motion.

view this post on Zulip Pat Hickey (Jan 12 2021 at 18:01):

and then, we'll have an impl of those traits for the cap-std types that is 100% synchronous/blocking. when you use this purely synchronous mode we'll assert that the future is ready and unwrap it at each import function call site (not 100% sure how this will be shaped yet). and then, we'll provide a separate trait that impls those traits on top of tokio.

view this post on Zulip Pat Hickey (Jan 12 2021 at 18:03):

see also https://github.com/bytecodealliance/cap-std/tree/main/cap-async-std

Capability-oriented version of the Rust standard library - bytecodealliance/cap-std

view this post on Zulip Pat Hickey (Jan 12 2021 at 18:06):

hopefully this explanation of my plans helps and makes sense. its not 100% figured out yet. but long story short, we'll make it possible to use async inside all methods of the file and dir traits, which should make it reasonable to start implementing them using e.g. network file system protocols or whatever you like

view this post on Zulip Pat Hickey (Jan 12 2021 at 18:08):

alex's async work is here: https://github.com/bytecodealliance/wasmtime/pull/2434 https://github.com/bytecodealliance/rfcs/pull/2

This is an implementation of RFC 2 in Wasmtime which is to support async-defined host functions. At a high level support is added by executing WebAssembly code that might invoke an asynchronous hos...
This RFC proposes adding native support to the wasmtime crate for imported functions which are async. Additionally it provides APIs to call wasm functions as an async function. rendered

view this post on Zulip Pat Hickey (Jan 12 2021 at 18:09):

we appreciate your feedback on everything, please let us know what you think of these plans

view this post on Zulip Léo Gaspard (Jan 12 2021 at 21:32):

These plans look awesome, and thank you for asking about my opinion! I had seen @Alex Crichton's RFC and am putting great hope in them, though I'm a tiny bit scared about the futures not being Send IIRC (not sure it could be fixed though, IIRC wasmtime-async futures are Send but it's very possible that I missed a place where it ought to be restricted... and I haven't integrated them in my production code yet so maybe I'll find out that outside of toy examples they aren't).
For wasi, there's a thing I'm wondering: is the overhead of allocating a future per wasi call OK for the blocking option? I'd think probably yes, but if no it'd I think make sense to have a separate preopen_dir_async or similar, that'd use WasiDirAsync, and still provide the WasiDir blocking possibility.
Does that make sense?

view this post on Zulip Léo Gaspard (Jan 12 2021 at 21:33):

(For zulip being hard I can only concur, this is the only zulip I'm using and it's very different from other chat apps)

view this post on Zulip Léo Gaspard (Jan 13 2021 at 05:03):

Hmm, what's the current state of the wasi-c2 rewrite? I'm trying to use it to preopen a dir and failing to, but OTOH I wonder whether I should investigate more or whether it's still too much WIP for the time being and such behavior is expected (anyway, I've submitted a few usability improvements as PR#2577 :))

view this post on Zulip Pat Hickey (Jan 13 2021 at 18:11):

thanks for the PR, i'm taking a look now. i have a file called TEST_FAILURES in wasi-c2 that i try to keep up to date on whats missing. basically, the scheduler is still wip, and a few weird things related to symlink following and fnctl-equivelant are not working, but otherwise it should do everything it needs to do. missing functionality as far as the ctx builder and so on is just because i never got around to it, so thank you for contributing that :)

view this post on Zulip Pat Hickey (Jan 13 2021 at 18:14):

at the moment we are assuming that allocating a pin<box<future>> for every call is acceptable. if we wanted to make a WasiDir / WasiDirAsync split, we'd have to duplicate all of the code that calls those async methods, which would mean copy-pasting the contents of src/snapshots/preview_1.rs, essentially, with the async variant just having async prefixed on fns .await postfixed on calls. i want to avoid that if possible.

view this post on Zulip Pat Hickey (Jan 13 2021 at 18:18):

if someone comes along with a synchronous use case where that really is a perf bottleneck we can try to find a better solution, but for now the most demanding case we have performance-wise is the async embedding (fastly's compute@edge product)

view this post on Zulip Léo Gaspard (Jan 14 2021 at 00:23):

Then the plan sounds great to me! Thank you :D FWIW once Alex Crichton's async PR lands, I might be interested in porting WASI to async (though it'll quite possibly be very little work) :)


Last updated: Dec 23 2024 at 13:07 UTC