Léo Gaspard (Jan 11 2021 at 01:12): Léo Gaspard (Jan 11 2021 at 01:12):Hey! I'm in the process of starting to implement wasi bindings in my wasm project, and am hitting an issue: as I'm going to be in an async world (using wasmtime-async), I'd like to basically wrap all filesystem calls in unblock.
However, I can't figure out a way to do that. I remember a while ago searching (or being told?) that https://github.com/bytecodealliance/wasmtime/pull/1600 would allow that, but it seems that this only did it for stdio, and not for preopened dirs in particular (files can be virtual-based but AFAIU dirs must be explored at context-build time).
Is this me missing something, or an oversight that I should try to write a PR for, adding a preopened_handle that takes as an argument any T: Handle like stdin & co.?
Léo Gaspard (Jan 12 2021 at 00:08):Well, I've just sent https://github.com/bytecodealliance/wasmtime/pull/2569 implementing what I want, I'll just start using my fork until someone comments on the PR, though I'd love it if it was approved out of the box :)
Pat Hickey (Jan 12 2021 at 17:53):Hey, I guess this is the message I missed. sorry!
Pat Hickey (Jan 12 2021 at 17:53):zulip is sometimes confusing.
Pat Hickey (Jan 12 2021 at 17:54):Anyway, after I land the wasi-c2 rewrite, my next big thing is making it all work with async
Pat Hickey (Jan 12 2021 at 17:55):lucet has async support built-in right now, and @Alex Crichton has an RFC and PR to add async to natively wasmtime.
Pat Hickey (Jan 12 2021 at 17:55):one of our constraints is that we dont want all users of wasi-common (which is, effectively, all users of wasmtime at the moment) to take a dep on tokio (or async-std). so one trick we need is to factor it so that using an async runtime is optional.
Pat Hickey (Jan 12 2021 at 17:59):my rough plan is that i'll use the async-trait proc macro to make the WasiFile and WasiDir traits have async methods. this will probably mean in-lining the parts of the system-interface::FileIoExt that we actually consume into WasiFile and other code motion.
Pat Hickey (Jan 12 2021 at 18:01):and then, we'll have an impl of those traits for the cap-std types that is 100% synchronous/blocking. when you use this purely synchronous mode we'll assert that the future is ready and unwrap it at each import function call site (not 100% sure how this will be shaped yet). and then, we'll provide a separate trait that impls those traits on top of tokio.
Pat Hickey (Jan 12 2021 at 18:03):see also https://github.com/bytecodealliance/cap-std/tree/main/cap-async-std
Pat Hickey (Jan 12 2021 at 18:06):hopefully this explanation of my plans helps and makes sense. its not 100% figured out yet. but long story short, we'll make it possible to use async inside all methods of the file and dir traits, which should make it reasonable to start implementing them using e.g. network file system protocols or whatever you like
Pat Hickey (Jan 12 2021 at 18:08):alex's async work is here: https://github.com/bytecodealliance/wasmtime/pull/2434 https://github.com/bytecodealliance/rfcs/pull/2
Pat Hickey (Jan 12 2021 at 18:09):we appreciate your feedback on everything, please let us know what you think of these plans
Léo Gaspard (Jan 12 2021 at 21:32):These plans look awesome, and thank you for asking about my opinion! I had seen @Alex Crichton's RFC and am putting great hope in them, though I'm a tiny bit scared about the futures not being Send IIRC (not sure it could be fixed though, IIRC wasmtime-async futures are Send but it's very possible that I missed a place where it ought to be restricted... and I haven't integrated them in my production code yet so maybe I'll find out that outside of toy examples they aren't).
For wasi, there's a thing I'm wondering: is the overhead of allocating a future per wasi call OK for the blocking option? I'd think probably yes, but if no it'd I think make sense to have a separate preopen_dir_async or similar, that'd use WasiDirAsync, and still provide the WasiDir blocking possibility.
Does that make sense?
Léo Gaspard (Jan 12 2021 at 21:33):(For zulip being hard I can only concur, this is the only zulip I'm using and it's very different from other chat apps)
Léo Gaspard (Jan 13 2021 at 05:03):Hmm, what's the current state of the wasi-c2 rewrite? I'm trying to use it to preopen a dir and failing to, but OTOH I wonder whether I should investigate more or whether it's still too much WIP for the time being and such behavior is expected (anyway, I've submitted a few usability improvements as PR#2577 :))
Pat Hickey (Jan 13 2021 at 18:11):thanks for the PR, i'm taking a look now. i have a file called TEST_FAILURES in wasi-c2 that i try to keep up to date on whats missing. basically, the scheduler is still wip, and a few weird things related to symlink following and fnctl-equivelant are not working, but otherwise it should do everything it needs to do. missing functionality as far as the ctx builder and so on is just because i never got around to it, so thank you for contributing that :)
Pat Hickey (Jan 13 2021 at 18:14):at the moment we are assuming that allocating a pin<box<future>> for every call is acceptable. if we wanted to make a WasiDir / WasiDirAsync split, we'd have to duplicate all of the code that calls those async methods, which would mean copy-pasting the contents of src/snapshots/preview_1.rs, essentially, with the async variant just having async prefixed on fns .await postfixed on calls. i want to avoid that if possible.
Pat Hickey (Jan 13 2021 at 18:18):if someone comes along with a synchronous use case where that really is a perf bottleneck we can try to find a better solution, but for now the most demanding case we have performance-wise is the async embedding (fastly's compute@edge product)
Léo Gaspard (Jan 14 2021 at 00:23):Then the plan sounds great to me! Thank you :D FWIW once Alex Crichton's async PR lands, I might be interested in porting WASI to async (though it'll quite possibly be very little work) :)
Last updated: Nov 22 2024 at 16:03 UTC