wasmtime / issue #5056 epoch_interruption in async fails ... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / issue #5056 epoch_interruption in async fails ...

Wasmtime GitHub notifications bot (Oct 14 2022 at 19:01):

bryanburgers opened issue #5056:

I am running into a situation where epoch_interruption in async calls is not working as I expect.

When trying to reproduce, I saw strange behavior.

Essentially, when using futures::join! to join the running wasm call with a future that will eventually call engine.increment_epoch(), sometimes the running future will never yield, so the epoch future doesn't have a chance to run and perform an increment_epoch().

In the test case below as written, the program never returns. However, several different minor tweaks does cause it to return.

Test Case

// cargo new --bin wasmtime-epoch
// cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time

/// Infinite loop
const WAT: &str = r#"
(module
    (func $run
        (loop $loop
            br $loop))
    (export "run" (func $run)))
"#;

#[tokio::main]
async fn main() {
    // Basic setup
    let mut config = wasmtime::Config::new();
    config.async_support(true);
    config.epoch_interruption(true);
    let engine = wasmtime::Engine::new(&config).unwrap();
    // engine.increment_epoch(); // Position 1
    let mut store = wasmtime::Store::new(&engine, ());
    // engine.increment_epoch(); // Position 2
    store.set_epoch_deadline(1);
    // engine.increment_epoch(); // Position 3
    let module = wasmtime::Module::new(&engine, WAT.as_bytes()).unwrap();
    let instance = wasmtime::Instance::new_async(&mut store, &module, &[])
        .await
        .unwrap();
    let func = instance
        .get_typed_func::<(), (), _>(&mut store, "run")
        .unwrap();
    // engine.increment_epoch(); // Position 4

    let epoch_future = async {
        tokio::time::sleep(std::time::Duration::from_secs(1)).await;
        println!("incrementing epoch");
        engine.increment_epoch();
    };
    let wasm_future = func.call_async(&mut store, ());
    let (result, _) = futures::join!(wasm_future, epoch_future);
    if let Err(err) = result {
        println!("{}", err.display_reason());
    }
}

Steps to Reproduce

cargo new --bin wasmtime-epoch
cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time
Copy the above reproduction into main.rs
cargo run

Expected Results

After 1s, I expect epoch_future to print "incrementing epoch", and then some time later I expect wasm_future to finish with an error because of a trap.

Actual Results

The program never finishes.

But other similar things do work

If I use a tokio::spawn instead of joining the futures, it works.

rust let engine_clone = engine.clone(); let epoch_future = async move { tokio::time::sleep(std::time::Duration::from_secs(1)).await; println!("incrementing epoch"); engine_clone.increment_epoch(); }; let wasm_future = func.call_async(&mut store, ()); tokio::spawn(epoch_future); let result = wasm_future.await;
If I put an engine.increment_epoch() in position 3 or 4 above, before I even start running the wasm function, it works. (But it doesn't work for position 1 or 2.)
If I use the sync version of this code instead of the async, it works. (But that uses a thread::spawn, which is pretty similar to the tokio::spawn test above.)

Versions and Environment

wasmtime version: 1.0.1

<details>
<summary>output of cargo tree</summary>

wasmtime-epoch v0.1.0 (/Users/bryan/personal/wasmtime-epoch)
├── futures v0.3.24
│   ├── futures-channel v0.3.24
│   │   ├── futures-core v0.3.24
│   │   └── futures-sink v0.3.24
│   ├── futures-core v0.3.24
│   ├── futures-executor v0.3.24
│   │   ├── futures-core v0.3.24
│   │   ├── futures-task v0.3.24
│   │   └── futures-util v0.3.24
│   │       ├── futures-channel v0.3.24 (*)
│   │       ├── futures-core v0.3.24
│   │       ├── futures-io v0.3.24
│   │       ├── futures-macro v0.3.24 (proc-macro)
│   │       │   ├── proc-macro2 v1.0.46
│   │       │   │   └── unicode-ident v1.0.5
│   │       │   ├── quote v1.0.21
│   │       │   │   └── proc-macro2 v1.0.46 (*)
│   │       │   └── syn v1.0.102
│   │       │       ├── proc-macro2 v1.0.46 (*)
│   │       │       ├── quote v1.0.21 (*)
│   │       │       └── unicode-ident v1.0.5
│   │       ├── futures-sink v0.3.24
│   │       ├── futures-task v0.3.24
│   │       ├── memchr v2.5.0
│   │       ├── pin-project-lite v0.2.9
│   │       ├── pin-utils v0.1.0
│   │       └── slab v0.4.7
│   │           [build-dependencies]
│   │           └── autocfg v1.1.0
│   ├── futures-io v0.3.24
│   ├── futures-sink v0.3.24
│   ├── futures-task v0.3.24
│   └── futures-util v0.3.24 (*)
├── tokio v1.21.2
│   ├── num_cpus v1.13.1
│   │   └── libc v0.2.135
│   ├── pin-project-lite v0.2.9
│   └── tokio-macros v1.8.0 (proc-macro)
│       ├── proc-macro2 v1.0.46 (*)
│       ├── quote v1.0.21 (*)
│       └── syn v1.0.102 (*)
│   [build-dependencies]
│   └── autocfg v1.1.0
└── wasmtime v1.0.1
    ├── anyhow v1.0.65
    ├── async-trait v0.1.57 (proc-macro)
    │   ├── proc-macro2 v1.0.46 (*)
    │   ├── quote v1.0.21 (*)
    │   └── syn v1.0.102 (*)
    ├── bincode v1.3.3
    │   └── serde v1.0.145
    │       └── serde_derive v1.0.145 (proc-macro)
    │           ├── proc-macro2 v1.0.46 (*)
    │           ├── quote v1.0.21 (*)
    │           └── syn v1.0.102 (*)
    ├── cfg-if v1.0.0
    ├── indexmap v1.9.1
    │   ├── hashbrown v0.12.3
    │   │   └── ahash v0.7.6
    │   │       ├── getrandom v0.2.7
    │   │       │   ├── cfg-if v1.0.0
    │   │       │   └── libc v0.2.135
    │   │       └── once_cell v1.15.0
    │   │       [build-dependencies]
    │   │       └── version_check v0.9.4
    │   └── serde v1.0.145 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── libc v0.2.135
    ├── log v0.4.17
    │   └── cfg-if v1.0.0
    ├── object v0.29.0
    │   ├── crc32fast v1.3.2
    │   │   └── cfg-if v1.0.0
    │   ├── hashbrown v0.12.3 (*)
    │   ├── indexmap v1.9.1 (*)
    │   └── memchr v2.5.0
    ├── once_cell v1.15.0
    ├── paste v1.0.9 (proc-macro)
    ├── psm v0.1.21
    │   [build-dependencies]
    │   └── cc v1.0.73
    │       └── jobserver v0.1.25
    │           └── libc v0.2.135
    ├── rayon v1.5.3
    │   ├── crossbeam-deque v0.8.2
    │   │   ├── cfg-if v1.0.0
    │   │   ├── crossbeam-epoch v0.9.11
    │   │   │   ├── cfg-if v1.0.0
    │   │   │   ├── crossbeam-utils v0.8.12
    │   │   │   │   └── cfg-if v1.0.0
    │   │   │   ├── memoffset v0.6.5
    │   │   │   │   [build-dependencies]
    │   │   │   │   └── autocfg v1.1.0
    │   │   │   └── scopeguard v1.1.0
    │   │   │   [build-dependencies]
    │   │   │   └── autocfg v1.1.0
    │   │   └── crossbeam-utils v0.8.12 (*)
    │   ├── either v1.8.0
    │   └── rayon-core v1.9.3
    │       ├── crossbeam-channel v0.5.6
    │       │   ├── cfg-if v1.0.0
    │       │   └── crossbeam-utils v0.8.12 (*)
    │       ├── crossbeam-deque v0.8.2 (*)
    │       ├── crossbeam-utils v0.8.12 (*)
    │       └── num_cpus v1.13.1 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── serde v1.0.145 (*)
    ├── target-lexicon v0.12.4
    ├── wasmparser v0.89.1
    │   └── indexmap v1.9.1 (*)
    ├── wasmtime-cache v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── base64 v0.13.0
    │   ├── bincode v1.3.3 (*)
    │   ├── directories-next v2.0.0
    │   │   ├── cfg-if v1.0.0
    │   │   └── dirs-sys-next v0.1.2
    │   │       └── libc v0.2.135
    │   ├── file-per-thread-logger v0.1.5
    │   │   ├── env_logger v0.9.1
    │   │   │   ├── atty v0.2.14
    │   │   │   │   └── libc v0.2.135
    │   │   │   ├── humantime v2.1.0
    │   │   │   ├── log v0.4.17 (*)
    │   │   │   ├── regex v1.6.0
    │   │   │   │   ├── aho-corasick v0.7.19
    │   │   │   │   │   └── memchr v2.5.0
    │   │   │   │   ├── memchr v2.5.0
    │   │   │   │   └── regex-syntax v0.6.27
    │   │   │   └── termcolor v1.1.3
    │   │   └── log v0.4.17 (*)
    │   ├── log v0.4.17 (*)
    │   ├── rustix v0.35.11
    │   │   ├── bitflags v1.3.2
    │   │   ├── errno v0.2.8
    │   │   │   └── libc v0.2.135
    │   │   ├── io-lifetimes v0.7.3
    │   │   └── libc v0.2.135
    │   ├── serde v1.0.145 (*)
    │   ├── sha2 v0.9.9
    │   │   ├── block-buffer v0.9.0
    │   │   │   └── generic-array v0.14.6
    │   │   │       └── typenum v1.15.0
    │   │   │       [build-dependencies]
    │   │   │       └── version_check v0.9.4
    │   │   ├── cfg-if v1.0.0
    │   │   ├── cpufeatures v0.2.5
    │   │   ├── digest v0.9.0
    │   │   │   └── generic-array v0.14.6 (*)
    │   │   └── opaque-debug v0.3.0
    │   ├── toml v0.5.9
    │   │   └── serde v1.0.145 (*)
    │   └── zstd v0.11.2+zstd.1.5.2
    │       └── zstd-safe v5.0.2+zstd.1.5.2
    │           ├── libc v0.2.135
    │           └── zstd-sys v2.0.1+zstd.1.5.2
    │               └── libc v0.2.135
    │               [build-dependencies]
    │               └── cc v1.0.73 (*)
    ├── wasmtime-cranelift v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── cranelift-codegen v0.88.1
    │   │   ├── arrayvec v0.7.2
    │   │   ├── bumpalo v3.11.0
    │   │   ├── cranelift-bforest v0.88.1
    │   │   │   └── cranelift-entity v0.88.1
    │   │   │       └── serde v1.0.145 (*)
    │   │   ├── cranelift-codegen-shared v0.88.1
    │   │   ├── cranelift-entity v0.88.1 (*)
    │   │   ├── gimli v0.26.2
    │   │   │   ├── fallible-iterator v0.2.0
    │   │   │   ├── indexmap v1.9.1 (*)
    │   │   │   └── stable_deref_trait v1.2.0
    │   │   ├── log v0.4.17 (*)
    │   │   ├── regalloc2 v0.3.2
    │   │   │   ├── fxhash v0.2.1
    │   │   │   │   └── byteorder v1.4.3
    │   │   │   ├── log v0.4.17
[message truncated]

Wasmtime GitHub notifications bot (Oct 14 2022 at 19:01):

bryanburgers labeled issue #5056:

I am running into a situation where epoch_interruption in async calls is not working as I expect.

When trying to reproduce, I saw strange behavior.

In the test case below as written, the program never returns. However, several different minor tweaks does cause it to return.

Test Case

// cargo new --bin wasmtime-epoch
// cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time

/// Infinite loop
const WAT: &str = r#"
(module
    (func $run
        (loop $loop
            br $loop))
    (export "run" (func $run)))
"#;

#[tokio::main]
async fn main() {
    // Basic setup
    let mut config = wasmtime::Config::new();
    config.async_support(true);
    config.epoch_interruption(true);
    let engine = wasmtime::Engine::new(&config).unwrap();
    // engine.increment_epoch(); // Position 1
    let mut store = wasmtime::Store::new(&engine, ());
    // engine.increment_epoch(); // Position 2
    store.set_epoch_deadline(1);
    // engine.increment_epoch(); // Position 3
    let module = wasmtime::Module::new(&engine, WAT.as_bytes()).unwrap();
    let instance = wasmtime::Instance::new_async(&mut store, &module, &[])
        .await
        .unwrap();
    let func = instance
        .get_typed_func::<(), (), _>(&mut store, "run")
        .unwrap();
    // engine.increment_epoch(); // Position 4

    let epoch_future = async {
        tokio::time::sleep(std::time::Duration::from_secs(1)).await;
        println!("incrementing epoch");
        engine.increment_epoch();
    };
    let wasm_future = func.call_async(&mut store, ());
    let (result, _) = futures::join!(wasm_future, epoch_future);
    if let Err(err) = result {
        println!("{}", err.display_reason());
    }
}

Steps to Reproduce

cargo new --bin wasmtime-epoch
cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time
Copy the above reproduction into main.rs
cargo run

Expected Results

After 1s, I expect epoch_future to print "incrementing epoch", and then some time later I expect wasm_future to finish with an error because of a trap.

Actual Results

The program never finishes.

But other similar things do work

If I use a tokio::spawn instead of joining the futures, it works.

rust let engine_clone = engine.clone(); let epoch_future = async move { tokio::time::sleep(std::time::Duration::from_secs(1)).await; println!("incrementing epoch"); engine_clone.increment_epoch(); }; let wasm_future = func.call_async(&mut store, ()); tokio::spawn(epoch_future); let result = wasm_future.await;
If I put an engine.increment_epoch() in position 3 or 4 above, before I even start running the wasm function, it works. (But it doesn't work for position 1 or 2.)
If I use the sync version of this code instead of the async, it works. (But that uses a thread::spawn, which is pretty similar to the tokio::spawn test above.)

Versions and Environment

wasmtime version: 1.0.1

<details>
<summary>output of cargo tree</summary>

wasmtime-epoch v0.1.0 (/Users/bryan/personal/wasmtime-epoch)
├── futures v0.3.24
│   ├── futures-channel v0.3.24
│   │   ├── futures-core v0.3.24
│   │   └── futures-sink v0.3.24
│   ├── futures-core v0.3.24
│   ├── futures-executor v0.3.24
│   │   ├── futures-core v0.3.24
│   │   ├── futures-task v0.3.24
│   │   └── futures-util v0.3.24
│   │       ├── futures-channel v0.3.24 (*)
│   │       ├── futures-core v0.3.24
│   │       ├── futures-io v0.3.24
│   │       ├── futures-macro v0.3.24 (proc-macro)
│   │       │   ├── proc-macro2 v1.0.46
│   │       │   │   └── unicode-ident v1.0.5
│   │       │   ├── quote v1.0.21
│   │       │   │   └── proc-macro2 v1.0.46 (*)
│   │       │   └── syn v1.0.102
│   │       │       ├── proc-macro2 v1.0.46 (*)
│   │       │       ├── quote v1.0.21 (*)
│   │       │       └── unicode-ident v1.0.5
│   │       ├── futures-sink v0.3.24
│   │       ├── futures-task v0.3.24
│   │       ├── memchr v2.5.0
│   │       ├── pin-project-lite v0.2.9
│   │       ├── pin-utils v0.1.0
│   │       └── slab v0.4.7
│   │           [build-dependencies]
│   │           └── autocfg v1.1.0
│   ├── futures-io v0.3.24
│   ├── futures-sink v0.3.24
│   ├── futures-task v0.3.24
│   └── futures-util v0.3.24 (*)
├── tokio v1.21.2
│   ├── num_cpus v1.13.1
│   │   └── libc v0.2.135
│   ├── pin-project-lite v0.2.9
│   └── tokio-macros v1.8.0 (proc-macro)
│       ├── proc-macro2 v1.0.46 (*)
│       ├── quote v1.0.21 (*)
│       └── syn v1.0.102 (*)
│   [build-dependencies]
│   └── autocfg v1.1.0
└── wasmtime v1.0.1
    ├── anyhow v1.0.65
    ├── async-trait v0.1.57 (proc-macro)
    │   ├── proc-macro2 v1.0.46 (*)
    │   ├── quote v1.0.21 (*)
    │   └── syn v1.0.102 (*)
    ├── bincode v1.3.3
    │   └── serde v1.0.145
    │       └── serde_derive v1.0.145 (proc-macro)
    │           ├── proc-macro2 v1.0.46 (*)
    │           ├── quote v1.0.21 (*)
    │           └── syn v1.0.102 (*)
    ├── cfg-if v1.0.0
    ├── indexmap v1.9.1
    │   ├── hashbrown v0.12.3
    │   │   └── ahash v0.7.6
    │   │       ├── getrandom v0.2.7
    │   │       │   ├── cfg-if v1.0.0
    │   │       │   └── libc v0.2.135
    │   │       └── once_cell v1.15.0
    │   │       [build-dependencies]
    │   │       └── version_check v0.9.4
    │   └── serde v1.0.145 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── libc v0.2.135
    ├── log v0.4.17
    │   └── cfg-if v1.0.0
    ├── object v0.29.0
    │   ├── crc32fast v1.3.2
    │   │   └── cfg-if v1.0.0
    │   ├── hashbrown v0.12.3 (*)
    │   ├── indexmap v1.9.1 (*)
    │   └── memchr v2.5.0
    ├── once_cell v1.15.0
    ├── paste v1.0.9 (proc-macro)
    ├── psm v0.1.21
    │   [build-dependencies]
    │   └── cc v1.0.73
    │       └── jobserver v0.1.25
    │           └── libc v0.2.135
    ├── rayon v1.5.3
    │   ├── crossbeam-deque v0.8.2
    │   │   ├── cfg-if v1.0.0
    │   │   ├── crossbeam-epoch v0.9.11
    │   │   │   ├── cfg-if v1.0.0
    │   │   │   ├── crossbeam-utils v0.8.12
    │   │   │   │   └── cfg-if v1.0.0
    │   │   │   ├── memoffset v0.6.5
    │   │   │   │   [build-dependencies]
    │   │   │   │   └── autocfg v1.1.0
    │   │   │   └── scopeguard v1.1.0
    │   │   │   [build-dependencies]
    │   │   │   └── autocfg v1.1.0
    │   │   └── crossbeam-utils v0.8.12 (*)
    │   ├── either v1.8.0
    │   └── rayon-core v1.9.3
    │       ├── crossbeam-channel v0.5.6
    │       │   ├── cfg-if v1.0.0
    │       │   └── crossbeam-utils v0.8.12 (*)
    │       ├── crossbeam-deque v0.8.2 (*)
    │       ├── crossbeam-utils v0.8.12 (*)
    │       └── num_cpus v1.13.1 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── serde v1.0.145 (*)
    ├── target-lexicon v0.12.4
    ├── wasmparser v0.89.1
    │   └── indexmap v1.9.1 (*)
    ├── wasmtime-cache v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── base64 v0.13.0
    │   ├── bincode v1.3.3 (*)
    │   ├── directories-next v2.0.0
    │   │   ├── cfg-if v1.0.0
    │   │   └── dirs-sys-next v0.1.2
    │   │       └── libc v0.2.135
    │   ├── file-per-thread-logger v0.1.5
    │   │   ├── env_logger v0.9.1
    │   │   │   ├── atty v0.2.14
    │   │   │   │   └── libc v0.2.135
    │   │   │   ├── humantime v2.1.0
    │   │   │   ├── log v0.4.17 (*)
    │   │   │   ├── regex v1.6.0
    │   │   │   │   ├── aho-corasick v0.7.19
    │   │   │   │   │   └── memchr v2.5.0
    │   │   │   │   ├── memchr v2.5.0
    │   │   │   │   └── regex-syntax v0.6.27
    │   │   │   └── termcolor v1.1.3
    │   │   └── log v0.4.17 (*)
    │   ├── log v0.4.17 (*)
    │   ├── rustix v0.35.11
    │   │   ├── bitflags v1.3.2
    │   │   ├── errno v0.2.8
    │   │   │   └── libc v0.2.135
    │   │   ├── io-lifetimes v0.7.3
    │   │   └── libc v0.2.135
    │   ├── serde v1.0.145 (*)
    │   ├── sha2 v0.9.9
    │   │   ├── block-buffer v0.9.0
    │   │   │   └── generic-array v0.14.6
    │   │   │       └── typenum v1.15.0
    │   │   │       [build-dependencies]
    │   │   │       └── version_check v0.9.4
    │   │   ├── cfg-if v1.0.0
    │   │   ├── cpufeatures v0.2.5
    │   │   ├── digest v0.9.0
    │   │   │   └── generic-array v0.14.6 (*)
    │   │   └── opaque-debug v0.3.0
    │   ├── toml v0.5.9
    │   │   └── serde v1.0.145 (*)
    │   └── zstd v0.11.2+zstd.1.5.2
    │       └── zstd-safe v5.0.2+zstd.1.5.2
    │           ├── libc v0.2.135
    │           └── zstd-sys v2.0.1+zstd.1.5.2
    │               └── libc v0.2.135
    │               [build-dependencies]
    │               └── cc v1.0.73 (*)
    ├── wasmtime-cranelift v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── cranelift-codegen v0.88.1
    │   │   ├── arrayvec v0.7.2
    │   │   ├── bumpalo v3.11.0
    │   │   ├── cranelift-bforest v0.88.1
    │   │   │   └── cranelift-entity v0.88.1
    │   │   │       └── serde v1.0.145 (*)
    │   │   ├── cranelift-codegen-shared v0.88.1
    │   │   ├── cranelift-entity v0.88.1 (*)
    │   │   ├── gimli v0.26.2
    │   │   │   ├── fallible-iterator v0.2.0
    │   │   │   ├── indexmap v1.9.1 (*)
    │   │   │   └── stable_deref_trait v1.2.0
    │   │   ├── log v0.4.17 (*)
    │   │   ├── regalloc2 v0.3.2
    │   │   │   ├── fxhash v0.2.1
    │   │   │   │   └── byteorder v1.4.3
    │   │   │   ├── log v0.4.17
[message truncated]

Wasmtime GitHub notifications bot (Oct 14 2022 at 19:06):

bryanburgers edited issue #5056:

I am running into a situation where epoch_interruption in async calls is not working as I expect.

When trying to reproduce, I saw strange behavior.

In the test case below as written, the program never returns. However, several different minor tweaks does cause it to return.

Test Case

// cargo new --bin wasmtime-epoch
// cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time

/// Infinite loop
const WAT: &str = r#"
(module
    (func $run
        (loop $loop
            br $loop))
    (export "run" (func $run)))
"#;

#[tokio::main]
async fn main() {
    // Basic setup
    let mut config = wasmtime::Config::new();
    config.async_support(true);
    config.epoch_interruption(true);
    let engine = wasmtime::Engine::new(&config).unwrap();
    // engine.increment_epoch(); // Position 1
    let mut store = wasmtime::Store::new(&engine, ());
    // engine.increment_epoch(); // Position 2
    store.set_epoch_deadline(1);
    // engine.increment_epoch(); // Position 3
    let module = wasmtime::Module::new(&engine, WAT.as_bytes()).unwrap();
    let instance = wasmtime::Instance::new_async(&mut store, &module, &[])
        .await
        .unwrap();
    let func = instance
        .get_typed_func::<(), (), _>(&mut store, "run")
        .unwrap();
    // engine.increment_epoch(); // Position 4

    let epoch_future = async {
        tokio::time::sleep(std::time::Duration::from_secs(1)).await;
        println!("incrementing epoch");
        engine.increment_epoch();
    };
    let wasm_future = func.call_async(&mut store, ());
    let (result, _) = futures::join!(wasm_future, epoch_future);
    if let Err(err) = result {
        println!("{}", err.display_reason());
    }
}

Steps to Reproduce

cargo new --bin wasmtime-epoch
cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time
Copy the above reproduction into main.rs
cargo run

Expected Results

After 1s, I expect epoch_future to print "incrementing epoch", and then some time later I expect wasm_future to finish with an error because of a trap.

Actual Results

The program never finishes.

But other similar things do work

If I use a tokio::spawn instead of joining the futures, it works.

rust let engine_clone = engine.clone(); let epoch_future = async move { tokio::time::sleep(std::time::Duration::from_secs(1)).await; println!("incrementing epoch"); engine_clone.increment_epoch(); }; let wasm_future = func.call_async(&mut store, ()); tokio::spawn(epoch_future); let result = wasm_future.await;
If I put an engine.increment_epoch() in position 3 or 4 above, before I even start running the wasm function, it works. (But it doesn't work for position 1 or 2.)
If I use the sync version of this code instead of the async, it works. (But that uses a thread::spawn, which is pretty similar to the tokio::spawn test above.)

Versions and Environment

wasmtime version: 1.0.1

<details>
<summary>output of cargo tree</summary>

wasmtime-epoch v0.1.0 (/Users/bryan/personal/wasmtime-epoch)
├── futures v0.3.24
│   ├── futures-channel v0.3.24
│   │   ├── futures-core v0.3.24
│   │   └── futures-sink v0.3.24
│   ├── futures-core v0.3.24
│   ├── futures-executor v0.3.24
│   │   ├── futures-core v0.3.24
│   │   ├── futures-task v0.3.24
│   │   └── futures-util v0.3.24
│   │       ├── futures-channel v0.3.24 (*)
│   │       ├── futures-core v0.3.24
│   │       ├── futures-io v0.3.24
│   │       ├── futures-macro v0.3.24 (proc-macro)
│   │       │   ├── proc-macro2 v1.0.46
│   │       │   │   └── unicode-ident v1.0.5
│   │       │   ├── quote v1.0.21
│   │       │   │   └── proc-macro2 v1.0.46 (*)
│   │       │   └── syn v1.0.102
│   │       │       ├── proc-macro2 v1.0.46 (*)
│   │       │       ├── quote v1.0.21 (*)
│   │       │       └── unicode-ident v1.0.5
│   │       ├── futures-sink v0.3.24
│   │       ├── futures-task v0.3.24
│   │       ├── memchr v2.5.0
│   │       ├── pin-project-lite v0.2.9
│   │       ├── pin-utils v0.1.0
│   │       └── slab v0.4.7
│   │           [build-dependencies]
│   │           └── autocfg v1.1.0
│   ├── futures-io v0.3.24
│   ├── futures-sink v0.3.24
│   ├── futures-task v0.3.24
│   └── futures-util v0.3.24 (*)
├── tokio v1.21.2
│   ├── num_cpus v1.13.1
│   │   └── libc v0.2.135
│   ├── pin-project-lite v0.2.9
│   └── tokio-macros v1.8.0 (proc-macro)
│       ├── proc-macro2 v1.0.46 (*)
│       ├── quote v1.0.21 (*)
│       └── syn v1.0.102 (*)
│   [build-dependencies]
│   └── autocfg v1.1.0
└── wasmtime v1.0.1
    ├── anyhow v1.0.65
    ├── async-trait v0.1.57 (proc-macro)
    │   ├── proc-macro2 v1.0.46 (*)
    │   ├── quote v1.0.21 (*)
    │   └── syn v1.0.102 (*)
    ├── bincode v1.3.3
    │   └── serde v1.0.145
    │       └── serde_derive v1.0.145 (proc-macro)
    │           ├── proc-macro2 v1.0.46 (*)
    │           ├── quote v1.0.21 (*)
    │           └── syn v1.0.102 (*)
    ├── cfg-if v1.0.0
    ├── indexmap v1.9.1
    │   ├── hashbrown v0.12.3
    │   │   └── ahash v0.7.6
    │   │       ├── getrandom v0.2.7
    │   │       │   ├── cfg-if v1.0.0
    │   │       │   └── libc v0.2.135
    │   │       └── once_cell v1.15.0
    │   │       [build-dependencies]
    │   │       └── version_check v0.9.4
    │   └── serde v1.0.145 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── libc v0.2.135
    ├── log v0.4.17
    │   └── cfg-if v1.0.0
    ├── object v0.29.0
    │   ├── crc32fast v1.3.2
    │   │   └── cfg-if v1.0.0
    │   ├── hashbrown v0.12.3 (*)
    │   ├── indexmap v1.9.1 (*)
    │   └── memchr v2.5.0
    ├── once_cell v1.15.0
    ├── paste v1.0.9 (proc-macro)
    ├── psm v0.1.21
    │   [build-dependencies]
    │   └── cc v1.0.73
    │       └── jobserver v0.1.25
    │           └── libc v0.2.135
    ├── rayon v1.5.3
    │   ├── crossbeam-deque v0.8.2
    │   │   ├── cfg-if v1.0.0
    │   │   ├── crossbeam-epoch v0.9.11
    │   │   │   ├── cfg-if v1.0.0
    │   │   │   ├── crossbeam-utils v0.8.12
    │   │   │   │   └── cfg-if v1.0.0
    │   │   │   ├── memoffset v0.6.5
    │   │   │   │   [build-dependencies]
    │   │   │   │   └── autocfg v1.1.0
    │   │   │   └── scopeguard v1.1.0
    │   │   │   [build-dependencies]
    │   │   │   └── autocfg v1.1.0
    │   │   └── crossbeam-utils v0.8.12 (*)
    │   ├── either v1.8.0
    │   └── rayon-core v1.9.3
    │       ├── crossbeam-channel v0.5.6
    │       │   ├── cfg-if v1.0.0
    │       │   └── crossbeam-utils v0.8.12 (*)
    │       ├── crossbeam-deque v0.8.2 (*)
    │       ├── crossbeam-utils v0.8.12 (*)
    │       └── num_cpus v1.13.1 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── serde v1.0.145 (*)
    ├── target-lexicon v0.12.4
    ├── wasmparser v0.89.1
    │   └── indexmap v1.9.1 (*)
    ├── wasmtime-cache v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── base64 v0.13.0
    │   ├── bincode v1.3.3 (*)
    │   ├── directories-next v2.0.0
    │   │   ├── cfg-if v1.0.0
    │   │   └── dirs-sys-next v0.1.2
    │   │       └── libc v0.2.135
    │   ├── file-per-thread-logger v0.1.5
    │   │   ├── env_logger v0.9.1
    │   │   │   ├── atty v0.2.14
    │   │   │   │   └── libc v0.2.135
    │   │   │   ├── humantime v2.1.0
    │   │   │   ├── log v0.4.17 (*)
    │   │   │   ├── regex v1.6.0
    │   │   │   │   ├── aho-corasick v0.7.19
    │   │   │   │   │   └── memchr v2.5.0
    │   │   │   │   ├── memchr v2.5.0
    │   │   │   │   └── regex-syntax v0.6.27
    │   │   │   └── termcolor v1.1.3
    │   │   └── log v0.4.17 (*)
    │   ├── log v0.4.17 (*)
    │   ├── rustix v0.35.11
    │   │   ├── bitflags v1.3.2
    │   │   ├── errno v0.2.8
    │   │   │   └── libc v0.2.135
    │   │   ├── io-lifetimes v0.7.3
    │   │   └── libc v0.2.135
    │   ├── serde v1.0.145 (*)
    │   ├── sha2 v0.9.9
    │   │   ├── block-buffer v0.9.0
    │   │   │   └── generic-array v0.14.6
    │   │   │       └── typenum v1.15.0
    │   │   │       [build-dependencies]
    │   │   │       └── version_check v0.9.4
    │   │   ├── cfg-if v1.0.0
    │   │   ├── cpufeatures v0.2.5
    │   │   ├── digest v0.9.0
    │   │   │   └── generic-array v0.14.6 (*)
    │   │   └── opaque-debug v0.3.0
    │   ├── toml v0.5.9
    │   │   └── serde v1.0.145 (*)
    │   └── zstd v0.11.2+zstd.1.5.2
    │       └── zstd-safe v5.0.2+zstd.1.5.2
    │           ├── libc v0.2.135
    │           └── zstd-sys v2.0.1+zstd.1.5.2
    │               └── libc v0.2.135
    │               [build-dependencies]
    │               └── cc v1.0.73 (*)
    ├── wasmtime-cranelift v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── cranelift-codegen v0.88.1
    │   │   ├── arrayvec v0.7.2
    │   │   ├── bumpalo v3.11.0
    │   │   ├── cranelift-bforest v0.88.1
    │   │   │   └── cranelift-entity v0.88.1
    │   │   │       └── serde v1.0.145 (*)
    │   │   ├── cranelift-codegen-shared v0.88.1
    │   │   ├── cranelift-entity v0.88.1 (*)
    │   │   ├── gimli v0.26.2
    │   │   │   ├── fallible-iterator v0.2.0
    │   │   │   ├── indexmap v1.9.1 (*)
    │   │   │   └── stable_deref_trait v1.2.0
    │   │   ├── log v0.4.17 (*)
    │   │   ├── regalloc2 v0.3.2
    │   │   │   ├── fxhash v0.2.1
    │   │   │   │   └── byteorder v1.4.3
    │   │   │   ├── log v0.4.17
[message truncated]

Wasmtime GitHub notifications bot (Oct 14 2022 at 19:15):

bjorn3 commented on issue #5056:

Your wasm module is blocking the thread on which you run it. In async code this not allowed as it causes starvation. You have to run blocking code within tokio::task::spawn_blocking.

Wasmtime GitHub notifications bot (Oct 14 2022 at 19:30):

cfallin commented on issue #5056:

@bryanburgers -- if I understand your example correctly, it is creating a deadlock via a circular dependence: the async wasm call will not yield until the epoch changes, but the yield is only going to occur when the next async task runs (the other future you've created).

In theory a multithreaded async runtime might choose to create a second thread and thus get to the epoch-increment before the wasm call finishes; but it doesn't have to. Semantically the guarantee that every async task must provide is that it finishes in finite (ideally short) time; and a runtime is allowed to be designed assuming that.

The way out of this is to ensure that the epoch increment happens outside the context of the event loop. What we do (in the environment we designed epoch interruption for originally) is spawn a separate, dedicated thread that loops forever, sleeps for the epoch length and increments the epoch counter. The sleep duration gives an upper bound on the amount of time any async wasm function can execute in one timeslice.

Your wasm module is blocking the thread on which you run it. In async code this not allowed as it causes starvation. You have to run blocking code within tokio::task::spawn_blocking.

That's the point of epoch interruption though -- calling a function on an event-loop thread is safe because the epoch will eventually change and cause it to yield if it hasn't finished yet. The only issue is that we need to ensure the epoch increment happens :-)

Wasmtime GitHub notifications bot (Oct 14 2022 at 19:31):

cfallin commented on issue #5056:

Since the epoch interruption mechanism is working as intended (the bug here is that the embedding code is not calling increment_epoch()) I'll go ahead and close this issue; but please feel free to ask if you have any other questions on how to use this properly!

Wasmtime GitHub notifications bot (Oct 14 2022 at 19:31):

cfallin closed issue #5056:

I am running into a situation where epoch_interruption in async calls is not working as I expect.

When trying to reproduce, I saw strange behavior.

In the test case below as written, the program never returns. However, several different minor tweaks does cause it to return.

Test Case

// cargo new --bin wasmtime-epoch
// cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time

/// Infinite loop
const WAT: &str = r#"
(module
    (func $run
        (loop $loop
            br $loop))
    (export "run" (func $run)))
"#;

#[tokio::main]
async fn main() {
    // Basic setup
    let mut config = wasmtime::Config::new();
    config.async_support(true);
    config.epoch_interruption(true);
    let engine = wasmtime::Engine::new(&config).unwrap();
    // engine.increment_epoch(); // Position 1
    let mut store = wasmtime::Store::new(&engine, ());
    // engine.increment_epoch(); // Position 2
    store.set_epoch_deadline(1);
    // engine.increment_epoch(); // Position 3
    let module = wasmtime::Module::new(&engine, WAT.as_bytes()).unwrap();
    let instance = wasmtime::Instance::new_async(&mut store, &module, &[])
        .await
        .unwrap();
    let func = instance
        .get_typed_func::<(), (), _>(&mut store, "run")
        .unwrap();
    // engine.increment_epoch(); // Position 4

    let epoch_future = async {
        tokio::time::sleep(std::time::Duration::from_secs(1)).await;
        println!("incrementing epoch");
        engine.increment_epoch();
    };
    let wasm_future = func.call_async(&mut store, ());
    let (result, _) = futures::join!(wasm_future, epoch_future);
    if let Err(err) = result {
        println!("{}", err.display_reason());
    }
}

Steps to Reproduce

cargo new --bin wasmtime-epoch
cargo add wasmtime futures tokio -F tokio/rt-multi-thread -F tokio/macros -F tokio/time
Copy the above reproduction into main.rs
cargo run

Expected Results

After 1s, I expect epoch_future to print "incrementing epoch", and then some time later I expect wasm_future to finish with an error because of a trap.

Actual Results

The program never finishes.

But other similar things do work

If I use a tokio::spawn instead of joining the futures, it works.

rust let engine_clone = engine.clone(); let epoch_future = async move { tokio::time::sleep(std::time::Duration::from_secs(1)).await; println!("incrementing epoch"); engine_clone.increment_epoch(); }; let wasm_future = func.call_async(&mut store, ()); tokio::spawn(epoch_future); let result = wasm_future.await;
If I put an engine.increment_epoch() in position 3 or 4 above, before I even start running the wasm function, it works. (But it doesn't work for position 1 or 2.)
If I use the sync version of this code instead of the async, it works. (But that uses a thread::spawn, which is pretty similar to the tokio::spawn test above.)

Versions and Environment

wasmtime version: 1.0.1

<details>
<summary>output of cargo tree</summary>

wasmtime-epoch v0.1.0 (/Users/bryan/personal/wasmtime-epoch)
├── futures v0.3.24
│   ├── futures-channel v0.3.24
│   │   ├── futures-core v0.3.24
│   │   └── futures-sink v0.3.24
│   ├── futures-core v0.3.24
│   ├── futures-executor v0.3.24
│   │   ├── futures-core v0.3.24
│   │   ├── futures-task v0.3.24
│   │   └── futures-util v0.3.24
│   │       ├── futures-channel v0.3.24 (*)
│   │       ├── futures-core v0.3.24
│   │       ├── futures-io v0.3.24
│   │       ├── futures-macro v0.3.24 (proc-macro)
│   │       │   ├── proc-macro2 v1.0.46
│   │       │   │   └── unicode-ident v1.0.5
│   │       │   ├── quote v1.0.21
│   │       │   │   └── proc-macro2 v1.0.46 (*)
│   │       │   └── syn v1.0.102
│   │       │       ├── proc-macro2 v1.0.46 (*)
│   │       │       ├── quote v1.0.21 (*)
│   │       │       └── unicode-ident v1.0.5
│   │       ├── futures-sink v0.3.24
│   │       ├── futures-task v0.3.24
│   │       ├── memchr v2.5.0
│   │       ├── pin-project-lite v0.2.9
│   │       ├── pin-utils v0.1.0
│   │       └── slab v0.4.7
│   │           [build-dependencies]
│   │           └── autocfg v1.1.0
│   ├── futures-io v0.3.24
│   ├── futures-sink v0.3.24
│   ├── futures-task v0.3.24
│   └── futures-util v0.3.24 (*)
├── tokio v1.21.2
│   ├── num_cpus v1.13.1
│   │   └── libc v0.2.135
│   ├── pin-project-lite v0.2.9
│   └── tokio-macros v1.8.0 (proc-macro)
│       ├── proc-macro2 v1.0.46 (*)
│       ├── quote v1.0.21 (*)
│       └── syn v1.0.102 (*)
│   [build-dependencies]
│   └── autocfg v1.1.0
└── wasmtime v1.0.1
    ├── anyhow v1.0.65
    ├── async-trait v0.1.57 (proc-macro)
    │   ├── proc-macro2 v1.0.46 (*)
    │   ├── quote v1.0.21 (*)
    │   └── syn v1.0.102 (*)
    ├── bincode v1.3.3
    │   └── serde v1.0.145
    │       └── serde_derive v1.0.145 (proc-macro)
    │           ├── proc-macro2 v1.0.46 (*)
    │           ├── quote v1.0.21 (*)
    │           └── syn v1.0.102 (*)
    ├── cfg-if v1.0.0
    ├── indexmap v1.9.1
    │   ├── hashbrown v0.12.3
    │   │   └── ahash v0.7.6
    │   │       ├── getrandom v0.2.7
    │   │       │   ├── cfg-if v1.0.0
    │   │       │   └── libc v0.2.135
    │   │       └── once_cell v1.15.0
    │   │       [build-dependencies]
    │   │       └── version_check v0.9.4
    │   └── serde v1.0.145 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── libc v0.2.135
    ├── log v0.4.17
    │   └── cfg-if v1.0.0
    ├── object v0.29.0
    │   ├── crc32fast v1.3.2
    │   │   └── cfg-if v1.0.0
    │   ├── hashbrown v0.12.3 (*)
    │   ├── indexmap v1.9.1 (*)
    │   └── memchr v2.5.0
    ├── once_cell v1.15.0
    ├── paste v1.0.9 (proc-macro)
    ├── psm v0.1.21
    │   [build-dependencies]
    │   └── cc v1.0.73
    │       └── jobserver v0.1.25
    │           └── libc v0.2.135
    ├── rayon v1.5.3
    │   ├── crossbeam-deque v0.8.2
    │   │   ├── cfg-if v1.0.0
    │   │   ├── crossbeam-epoch v0.9.11
    │   │   │   ├── cfg-if v1.0.0
    │   │   │   ├── crossbeam-utils v0.8.12
    │   │   │   │   └── cfg-if v1.0.0
    │   │   │   ├── memoffset v0.6.5
    │   │   │   │   [build-dependencies]
    │   │   │   │   └── autocfg v1.1.0
    │   │   │   └── scopeguard v1.1.0
    │   │   │   [build-dependencies]
    │   │   │   └── autocfg v1.1.0
    │   │   └── crossbeam-utils v0.8.12 (*)
    │   ├── either v1.8.0
    │   └── rayon-core v1.9.3
    │       ├── crossbeam-channel v0.5.6
    │       │   ├── cfg-if v1.0.0
    │       │   └── crossbeam-utils v0.8.12 (*)
    │       ├── crossbeam-deque v0.8.2 (*)
    │       ├── crossbeam-utils v0.8.12 (*)
    │       └── num_cpus v1.13.1 (*)
    │   [build-dependencies]
    │   └── autocfg v1.1.0
    ├── serde v1.0.145 (*)
    ├── target-lexicon v0.12.4
    ├── wasmparser v0.89.1
    │   └── indexmap v1.9.1 (*)
    ├── wasmtime-cache v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── base64 v0.13.0
    │   ├── bincode v1.3.3 (*)
    │   ├── directories-next v2.0.0
    │   │   ├── cfg-if v1.0.0
    │   │   └── dirs-sys-next v0.1.2
    │   │       └── libc v0.2.135
    │   ├── file-per-thread-logger v0.1.5
    │   │   ├── env_logger v0.9.1
    │   │   │   ├── atty v0.2.14
    │   │   │   │   └── libc v0.2.135
    │   │   │   ├── humantime v2.1.0
    │   │   │   ├── log v0.4.17 (*)
    │   │   │   ├── regex v1.6.0
    │   │   │   │   ├── aho-corasick v0.7.19
    │   │   │   │   │   └── memchr v2.5.0
    │   │   │   │   ├── memchr v2.5.0
    │   │   │   │   └── regex-syntax v0.6.27
    │   │   │   └── termcolor v1.1.3
    │   │   └── log v0.4.17 (*)
    │   ├── log v0.4.17 (*)
    │   ├── rustix v0.35.11
    │   │   ├── bitflags v1.3.2
    │   │   ├── errno v0.2.8
    │   │   │   └── libc v0.2.135
    │   │   ├── io-lifetimes v0.7.3
    │   │   └── libc v0.2.135
    │   ├── serde v1.0.145 (*)
    │   ├── sha2 v0.9.9
    │   │   ├── block-buffer v0.9.0
    │   │   │   └── generic-array v0.14.6
    │   │   │       └── typenum v1.15.0
    │   │   │       [build-dependencies]
    │   │   │       └── version_check v0.9.4
    │   │   ├── cfg-if v1.0.0
    │   │   ├── cpufeatures v0.2.5
    │   │   ├── digest v0.9.0
    │   │   │   └── generic-array v0.14.6 (*)
    │   │   └── opaque-debug v0.3.0
    │   ├── toml v0.5.9
    │   │   └── serde v1.0.145 (*)
    │   └── zstd v0.11.2+zstd.1.5.2
    │       └── zstd-safe v5.0.2+zstd.1.5.2
    │           ├── libc v0.2.135
    │           └── zstd-sys v2.0.1+zstd.1.5.2
    │               └── libc v0.2.135
    │               [build-dependencies]
    │               └── cc v1.0.73 (*)
    ├── wasmtime-cranelift v1.0.1
    │   ├── anyhow v1.0.65
    │   ├── cranelift-codegen v0.88.1
    │   │   ├── arrayvec v0.7.2
    │   │   ├── bumpalo v3.11.0
    │   │   ├── cranelift-bforest v0.88.1
    │   │   │   └── cranelift-entity v0.88.1
    │   │   │       └── serde v1.0.145 (*)
    │   │   ├── cranelift-codegen-shared v0.88.1
    │   │   ├── cranelift-entity v0.88.1 (*)
    │   │   ├── gimli v0.26.2
    │   │   │   ├── fallible-iterator v0.2.0
    │   │   │   ├── indexmap v1.9.1 (*)
    │   │   │   └── stable_deref_trait v1.2.0
    │   │   ├── log v0.4.17 (*)
    │   │   ├── regalloc2 v0.3.2
    │   │   │   ├── fxhash v0.2.1
    │   │   │   │   └── byteorder v1.4.3
    │   │   │   ├── log v0.4.17 (*)

[message truncated]

Wasmtime GitHub notifications bot (Oct 17 2022 at 12:49):

bryanburgers commented on issue #5056:

@cfallin Thanks for the explanation. It took a bit to put it all together, but it makes sense now.

Last updated: Nov 22 2024 at 17:03 UTC