Stream: git-wasmtime

Topic: wasmtime / issue #5306 Add ability to stop sleeping WASM ...

Wasmtime GitHub notifications bot (Nov 20 2022 at 18:44):

r12f opened issue #5306:

Feature

Currently, when a WASM program calls sleep, it will eventually call into WasiSched::sleep with duration set to the time until next "wake up". However, if we would like to stop this instance in the middle of the sleep, we won't be able to wake this thread up.

<img width="828" alt="image" src="https://user-images.githubusercontent.com/1533278/202919231-cdda688c-0f92-4f75-aa3b-f92231f0c5bc.png">

I have checked the following discussions, and I am using async runtime with epoch interrupt to stop the instance. And my guess is that - if we could not be waken up from the sleep syscall, we won't be able to get a chance to check the epoch and stop. But I am not sure if I am missing anything and this feature is already supported or not.

• https://github.com/bytecodealliance/wasmtime/issues/860
• https://github.com/bytecodealliance/wasmtime/issues/1268

This is the WASM program that I am running:

use std::time::Duration;

#[tokio::main(flavor = "current_thread")]
async fn main() {
    tokio::time::sleep(Duration::from_secs(100)).await;
}

And here are some key functions that shows up on the stack:

https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasi-common/src/snapshots/preview_0.rs

    async fn poll_oneoff<'a>(
        &mut self,
        subs: &GuestPtr<'a, types::Subscription>,
        events: &GuestPtr<'a, types::Event>,
        nsubscriptions: types::Size,
    ) -> Result<types::Size, Error> {
    // ......
                    self.sched
                        .sleep(Duration::from_nanos(clocksub.timeout))
                        .await?;
    // ......

https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasi-common/cap-std-sync/src/sched.rs

    async fn sleep(&self, duration: Duration) -> Result<(), Error> {
        std::thread::sleep(duration);
        Ok(())
    }

Benefit

This allows us to stop the WASM instances that is sleeping for long time.

Implementation

We can probably change sleep to an event wait, and then we can signal the event whenever we got epoch change or other interrupts.

Alternatives

Some other ways might be alertable wait/sleep, but not sure how feasible they are for this case.

Wasmtime GitHub notifications bot (Nov 21 2022 at 16:32):

alexcrichton commented on issue #5306:

Currently the sync-flavor of the default WASI implementation does not have the support to wake up a sleeping thread and cancel it, but adding such a feature with something like a CondVar seems reasonable to do. Would you be up for making such a PR?

Otherwise I can also recommend using the async support for Wasmtime. An asynchronous sleep doesn't actually block any thread and can be trivially woken up and/or canceled by simply dropping the task.

Wasmtime GitHub notifications bot (Nov 21 2022 at 21:57):

r12f commented on issue #5306:

Hi Alex, thanks a lot for checking! Actually, I am using the async runtime as below. And I wonder if this is because I am using wasmtime 0.39.1, and this issue will be gone in later versions. I have also enabled "tokio" feature in wasmtime-wasi in Cargo.toml, so it should have the async sched there.

        let mut config = Config::new();
        config.epoch_interruption(true);
        config.async_support(true);
        config.consume_fuel(true);

        let engine = Engine::new(&config)?;

The reason I am still using the old version is because of 2 reasons, but my understanding might be outdated now:

• I am trying to get socket creation working with wasmtime, so I will need access to things like TcpSockets in cap-std, but IIRC, in later versions, we have blocked the access, because socket in WASI is still in proposal state.
• I am also using wit-bindgen to generate code for async host functions. But in latest wit-bindgen, the async keyword is removed, and I am not sure which one should we use at this moment. Writing the function bindings one by one might be too much fun for me : D. I wonder if you happen to know any doc to get me started, it will be awesome!

And yes, CondVar should work! Let me check a bit deeper and see how to make it work. I am getting pulled into some urgent weird issues today and will be on business trip next week. And in the worst case, I will come back after next week and continue coding.

Wasmtime GitHub notifications bot (Nov 22 2022 at 15:39):

alexcrichton commented on issue #5306:

Using an older version should be fine for this issue, it has the same support for async sleeping and such. You don't need both epoch_interruption and consume_fuel, though, if you're interested.

The async support in wit-bindgen was unrelated to the .async_support(true) option which is what you want.

Wasmtime GitHub notifications bot (Jul 07 2023 at 14:46):

FedorSmirnov89 commented on issue #5306:

Hi :) .

I have a similar use case as described by @r12f : I am using wasmtime in asynchronous mode and with the tokio feature enabled. I would like to execute WASM modules asynchronously, e.g., to run multiple different modules within the same thread. However, I would like to not have to rely on the assumption that the modules were written as asynchronous Rust code. I was hoping that it would be possible to achieve asynchronous behavior from modules compiled from synchronous code by using the "fuel-based" way of instantiating the modules (if I understand correctly, this instruments the modules with yield points during compilation). This works really well for all cases that I have tested, with the exception of the case where the modules use a synchronous sleep function in their code, e.g.

std::thread::sleep(Duration::from_secs(5));

in Rust. If I understand correctly (I think @r12f also pointed this out), this is caused by the fact that synchronous sleep calls from the module are translated into a call to std::thread::sleep on the runtime side. In this case, this leads to a blocking future which circumvents the fuel mechanic, since no instructions are executed until the thread is woken up by the OS.

Would there be any downside to using tokio::time::sleep for both synchronous and asynchronous sleep calls from the module? Is there maybe another possibility to run modules written in a synchronous way asynchronously (making them yield when the module calls a sleep function?). Thanks a lot!

(Sorry if I am getting the things wrong. Very new to wasmtime and WASM in general :) )

Wasmtime GitHub notifications bot (Jul 07 2023 at 15:11):

alexcrichton commented on issue #5306:

@FedorSmirnov89 you'll want to make sure you're using the async implementation of WASI as it sounds like you're instead using the sync version of WASI? Whether or not wasm itself thinks it's blocking is somewhat irrelevant as it's mostly up to the host and how it wants to implement that functionality. The async implementation of WASI is indeed implemented with tokio::time::sleep which would I believe work as you'd expect.

Wasmtime GitHub notifications bot (Jul 07 2023 at 16:45):

FedorSmirnov89 commented on issue #5306:

@alexcrichton Thank you for the super-quick answer!

You are totally right. I was not paying enough attention when looking at the examples in the repository.

After I replaced

1. wasmtime_wasi::add_to_linker(&mut linker, |s| s) by wasmtime_wasi::tokio::add_to_linker(&mut linker, |s| s) during the creation of the linker
2. and use wasmtime_wasi::WasiCtxBuilder by use wasmtime_wasi::tokio::WasiCtxBuilder in the file where I create the wasi context for the module

evth worked as expected :smiley: . Thank you!

One additional question, if I may: When now I try to run my whole program (which is a wrapper around wasmtime which starts and terminates WASM modules when requested) with a tokio runtime with a flavor = "current_thread", I get a panic saying "can call blocking only when running on the multi-threaded runtime" (indeed, with a multi-threaded runtime, evth works). As far as I am aware, I am not calling any blocking operations or spawning any threads anywhere in my code (only using tokio::task::spawn). Do you maybe have an idea where the blocking call could be happening? Thank you again :grinning:

Wasmtime GitHub notifications bot (Jul 07 2023 at 18:36):

alexcrichton commented on issue #5306:

Are you able to get a stack trace perhaps via RUST_BACKTRACE=1? That'd hopefully help narrow it down as the cause is probably within the WASI implementation of Wasmtime

Wasmtime GitHub notifications bot (Jul 07 2023 at 19:06):

FedorSmirnov89 commented on issue #5306:

Ah ya. Good tip (should have thought of it :big_smile: ), I am getting the stack trace below. Seeing that it seems to have a problem with fd_write, I had the guess that the problem may be that the module I was testing it with was printing to the stdout (which is blocking I guess?). Removing the printing from the module solves the problem, so that the module can be run (and stopped) without problems. Is it a problem with the way I am configuring the modules or should modules just not print to run them from within a single-threaded runtime? Thank you!

thread 'main' panicked at 'can call blocking only when running on the multi-threaded runtime', /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasi-tokio-10.0.1/src/lib.rs:123:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
   1: core::panicking::panic_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
   2: core::panicking::panic_display
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:150:5
   3: tokio::runtime::scheduler::multi_thread::worker::block_in_place
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:427:9
   4: tokio::task::blocking::block_in_place
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/task/blocking.rs:78:9
   5: wasi_tokio::block_on_dummy_executor
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasi-tokio-10.0.1/src/lib.rs:123:5
   6: <wasi_tokio::file::Stdout as wasi_common::file::WasiFile>::write_vectored::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasi-tokio-10.0.1/src/file.rs:144:17
   7: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
   8: wasi_common::snapshots::preview_1::<impl wasi_common::snapshots::preview_1::wasi_snapshot_preview1::WasiSnapshotPreview1 for wasi_common::ctx::WasiCtx>::fd_write::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasi-common-10.0.1/src/snapshots/preview_1.rs:466:56
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
  10: wasi_common::snapshots::preview_1::wasi_snapshot_preview1::fd_write::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasi-common-10.0.1/src/snapshots/preview_1.rs:27:1
  11: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  12: wasmtime_wasi::tokio::snapshots::preview_1::add_wasi_snapshot_preview1_to_linker::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-wasi-10.0.1/src/lib.rs:34:5
  13: wasmtime::store::AsyncCx::block_on
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/store.rs:1876:17
  14: wasmtime::linker::Linker<T>::func_wrap4_async::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/linker.rs:165:32
  15: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2,A3,A4),R>>::into_func::native_call_shim::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func.rs:1982:41
  16: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  17: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
  18: std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  19: __rust_try
  20: std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  21: std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  22: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2,A3,A4),R>>::into_func::native_call_shim::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func.rs:1977:29
  23: wasmtime::func::Caller<T>::with::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func.rs:1777:13
  24: wasmtime_runtime::instance::Instance::from_vmctx
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/instance.rs:217:9
  25: wasmtime::func::Caller<T>::with
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func.rs:1775:9
  26: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2,A3,A4),R>>::into_func::native_call_shim
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func.rs:1966:34
  27: <unknown>
  28: <unknown>
  29: <unknown>
  30: <unknown>
  31: <unknown>
  32: <unknown>
  33: <unknown>
  34: <unknown>
  35: <unknown>
  36: <unknown>
  37: <unknown>
  38: <unknown>
  39: <unknown>
  40: <unknown>
  41: <() as wasmtime::func::typed::WasmParams>::invoke::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func/typed.rs:587:21
  42: <(A1,) as wasmtime::func::HostAbi>::call
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func.rs:1684:18
  43: <() as wasmtime::func::typed::WasmParams>::invoke
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func/typed.rs:586:17
  44: wasmtime::func::typed::TypedFunc<Params,Results>::call_raw::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func/typed.rs:181:17
  45: wasmtime_runtime::traphandlers::catch_traps::call_closure
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/traphandlers.rs:281:18
  46: wasmtime_setjmp
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/helpers.c:55:3
  47: wasmtime_runtime::traphandlers::catch_traps::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/traphandlers.rs:263:9
  48: wasmtime_runtime::traphandlers::<impl wasmtime_runtime::traphandlers::call_thread_state::CallThreadState>::with::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/traphandlers.rs:389:44
  49: wasmtime_runtime::traphandlers::tls::set
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/traphandlers.rs:740:13
  50: wasmtime_runtime::traphandlers::<impl wasmtime_runtime::traphandlers::call_thread_state::CallThreadState>::with
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/traphandlers.rs:389:19
  51: wasmtime_runtime::traphandlers::catch_traps
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-10.0.1/src/traphandlers.rs:262:18
  52: wasmtime::func::invoke_wasm_and_catch_traps
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func.rs:1363:22
  53: wasmtime::func::typed::TypedFunc<Params,Results>::call_raw
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func/typed.rs:177:22
  54: wasmtime::func::typed::TypedFunc<Params,Results>::call_async::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/func/typed.rs:126:26
  55: wasmtime::store::<impl wasmtime::store::context::StoreContextMut<T>>::on_fiber::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-10.0.1/src/store.rs:1589:34
  56: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
  57: wasmtime_fiber::Suspend<Resume,Yield,Return>::execute::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-fiber-10.0.1/src/lib.rs:166:62
  58: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
  59: std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  60: __rust_try
  61: std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  62: std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  63: wasmtime_fiber::Suspend<Resume,Yield,Return>::execute
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-fiber-10.0.1/src/lib.rs:166
[message truncated]

Wasmtime GitHub notifications bot (Jul 07 2023 at 20:00):

alexcrichton commented on issue #5306:

Ah ok that makes sense, thanks for the extra information. This restriction is new to me at least and as far as I'm aware the WASI support was not developed with the intention in mind of explicitly not enabling the current thread executor. Put another way this I think is fine to classify as a normal bug, albeit it's separate from this current one so I think it'd be good to split out.

As to the bug itself, I think that may be resolved in the upcoming preview2 work https://github.com/bytecodealliance/wasmtime/pull/6556 with Joel's recent advice of using the tokio-native filesystem constructs. There may still be other issues, though.

For now I'd recommend using the multi-threaded executor.

Wasmtime GitHub notifications bot (Jul 08 2023 at 04:57):

FedorSmirnov89 commented on issue #5306:

Got it, thank you! Would it make sense for me to create an extra issue describing the problem and providing a minimal example which reproduces it?

Wasmtime GitHub notifications bot (Jul 10 2023 at 20:03):

alexcrichton commented on issue #5306:

Yes that'd be appreciated!

Last updated: Dec 23 2024 at 13:07 UTC