wasi-common lifetime · wasmtime · Zulip Chat Archive

How long will it take before wasi-common gets removed? I'm working on migrating a project from Wasmer to Wasmtime. wasi-common has a much closer api to wasmer-wasi than wasmtime-wasi. As such I chose to use wasi-common for now. Can I expect it to stay around for a while or will it be removed soon?

Alex Crichton (May 14 2024 at 14:42):

While I don't think anyone has a concrete plan yet it's very much on life support and it's recommended to switch to wasmtime-wasi. It's effectively a candidate for deletion as soon as anyone finds the motivation/time

bjorn3 (May 14 2024 at 14:43):

The async requirement of wasmtime-wasi is pretty inconvenient. I couldn't easily figure out how to pipe data into stdin and read it out of stdout again.

Alex Crichton (May 14 2024 at 14:45):

That'd probably be a good candidate for adding various adapters/types to work with that, we've already got Memory{Input,Output}Pipe IIRC and we could probably add std::io-based versions similarly or something similar

bjorn3 (May 14 2024 at 14:47):

With wasi-common I can use ReadPipe::from_shared and WritePipe::from_shared wrapping a Arc<RwLock<VecDeque<u8>>> which is trivial to write to by making a clone of the Arc and then using synchronous read/write calls that will never block, but wasmtime-wasi requires async streams and then somehow handling the case where Poll::Pending is returned, but the host program doesn't use async anywhere in the plugin integration.

Alex Crichton (May 14 2024 at 14:56):

Yes that's something we can provide an adapter for

Alex Crichton (May 14 2024 at 14:56):

it's possible to turn an Arc<RwLock> and adapt it to the async interfaces required by wasmtime-wasi

bjorn3 (May 14 2024 at 16:27):

For AsyncWrite it was almost trivial to forward to Write. For AsyncRead however the implementation seems to be a lot more involved.

bjorn3 (May 14 2024 at 16:58):

I got something that compiles, but it is infecting a lot more with async than just those adapters. instantiate and call both need their async version. I tried wasmtime_wasi::runtime::in_tokio, but there is a deadlock somewhere. The last wasi call is a write to stdout.

bjorn3 (May 14 2024 at 16:59):

https://github.com/bjorn3/zellij/commit/091ec7fd5db1661bf44d390bd0ce64188e74d42d is what I got right now.

[WIP] Switch from wasi-common to wasmtime-wasi · bjorn3/zellij@091ec7f

This currently deadlocks

Alex Crichton (May 14 2024 at 17:10):

oh I wasn't imagining that the whole embedding would turn async since wasmtime-wasi has a sync mode too

Alex Crichton (May 14 2024 at 17:10):

I was thinking that the adapters might only work in sync mode or something like that

Pat Hickey (May 14 2024 at 17:10):

Did you try using the sync::add_to_linker? You shouldn’t use in_tokio from outside the wasmtime-wasi crate or child crates such as wasmtime-wasi-http

bjorn3 (May 14 2024 at 17:14):

I can't find any sync variant in wasmtime_wasi::preview1. I'm trying to prevent adding preview2 support for now in case the migration needs to be rolled back.

bjorn3 (May 14 2024 at 17:17):

Is there a different add_to_linker that I can use for sync + preview1-only?

Alex Crichton (May 14 2024 at 17:18):

this one?

Pat Hickey (May 14 2024 at 17:21):

yeah sorry i got the name wrong, the sync/async specifier is a suffix now not a mod

bjorn3 (May 14 2024 at 17:22):

That solves the async infection issue. It doesn't solve the deadlock though.

bjorn3 (May 14 2024 at 17:23):

https://github.com/bjorn3/zellij/commit/9f35c9fda1c9d05ba78e90048e9be962d53558a2 is what I have now.

[WIP] Switch from wasi-common to wasmtime-wasi · bjorn3/zellij@9f35c9f

This currently deadlocks

Alex Crichton (May 14 2024 at 17:28):

I think that you'll still need to bridge the async-ness in the vecdeque adapters perhaps?

Alex Crichton (May 14 2024 at 17:28):

hm no they're always ready

Alex Crichton (May 14 2024 at 17:32):

are you able to capture the stack at deadlock time?

bjorn3 (May 14 2024 at 17:32):

There are a lot of threads. I have no clue which one is deadlocking.

Alex Crichton (May 14 2024 at 17:33):

on that commit is there a command to run to poke around?

bjorn3 (May 14 2024 at 17:33):

cargo xtask run

bjorn3 (May 14 2024 at 17:33):

It gets stuck at Starting.

bjorn3 (May 14 2024 at 17:34):

You can exit with ctrl+q and due to the deadlock you have to manually kill the zellij --server process after exiting the client.

Alex Crichton (May 14 2024 at 17:34):

05809cb955d40053f1159030fe88f5b69e48f717 now, right?

bjorn3 (May 14 2024 at 17:35):

Yeah, that is my current version. The only difference with the last commit I linked is a change to CONTRIBUTING.md though.

Alex Crichton (May 14 2024 at 17:35):

do you know what to do with:

called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "protoc failed: resize.proto:19:12: Explicit 'optional' labels are disallowed in the Proto3 syntax. To define 'optional' fields in Proto3, simply remove the 'optional' label, as fields are 'optional' by default.\n" }

bjorn3 (May 14 2024 at 17:35):

You can find the logs at /tmp/zellij-1000/zellij-log/zellij.log (substitute your own uid if it isn't 1000)

bjorn3 (May 14 2024 at 17:36):

Alex Crichton said:

do you know what to do with:

called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "protoc failed: resize.proto:19:12: Explicit 'optional' labels are disallowed in the Proto3 syntax. To define 'optional' fields in Proto3, simply remove the 'optional' label, as fields are 'optional' by default.\n" }

I didn't get that error locally. Maybe you have a different protobuf compiler version?

Alex Crichton (May 14 2024 at 17:36):

apparently I had some weird install, trying system package manager

bjorn3 (May 14 2024 at 17:36):

I have version libprotoc 3.21.12 as output of protoc --version.

Alex Crichton (May 14 2024 at 17:37):

same now, making more progress

Alex Crichton (May 14 2024 at 17:39):

Thread 108 (Thread 0x716e2bc006c0 (LWP 3558238) "server_listener"):

wow you're not kidding

bjorn3 (May 14 2024 at 17:41):

Commenting out most add_plugin!() calls in zellij-utils/src/consts.rs may help. You need to leave one of them as otherwise no wasm plugin will be loaded and thus the deadlock can't occur.