Stream: wasmtime

Topic: redfine/shadow wasi functions


view this post on Zulip Ben Linsay (Jan 23 2024 at 16:59):

hey all,

i'm messing around and wanted to see if it'd be possible to redefine an existing wasi function. I don't really have a goal in mind, but thought it'd be neat to define my own behavior for a small subset of existing wasi functions. :)

the tutorial and docs made it really easy to get up and running (thanks!) but I'm getting an error about memory exports I don't really understand.

my little snippet so far is this code, shoved into the middle of the example from 2.3.4 in the guide:

    let original_env_get: TypedFunc<(i32, i32), i32> = {
        let Some(wasmtime::Extern::Func(env_get)) =
            linker.get(&mut store, "wasi_snapshot_preview1", "environ_get")
        else {
            anyhow::bail!("missing environ_get");
        };
        env_get.typed(&store).unwrap()
    };

    linker.allow_shadowing(true);
    linker.func_wrap(
        "wasi_snapshot_preview1",
        "environ_get",
        move |caller: Caller<'_, _>, a: i32, b: i32| original_env_get.call(caller, (a, b)),
 )?;

when I run it with a guest that actually tries to get an env var, I get a backtrace that looks like this. I'm not sure what to make of it - am I accidentally clobbering other parts of the wasi component by shadowing? any tips for reading this error better?

$ cargo run -p host -- ./target/wasm32-wasi/debug/guest.wasm
   Compiling host v0.1.0 (/Users/benl/src/hello-wasmtime/host)
    Finished dev [unoptimized + debuginfo] target(s) in 0.88s
     Running `target/debug/host ./target/wasm32-wasi/debug/guest.wasm`
Error: error while executing at wasm backtrace:
    0: 0x9b66 - <unknown>!__wasi_environ_get
    1: 0x9b24 - <unknown>!__wasilibc_initialize_environ
    2: 0x9a96 - <unknown>!__wasilibc_ensure_environ
    3: 0x9c4d - <unknown>!getenv
    4: 0x3b2a - <unknown>!std::env::_var_os::ha6d753b2be7fe17f
    5: 0x3c99 - <unknown>!std::env::_var::hd7c38fcd0ed4a72d
    6: 0x1829 - <unknown>!std::env::var::hd342252c7c9996f3
    7: 0x14f4 - <unknown>!guest::main::hb3d545227282da60
    8:  0xac4 - <unknown>!core::ops::function::FnOnce::call_once::hdf6e14f7ebde9f53
    9:  0xf43 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::ha23fe57107e63372
   10:  0xecb - <unknown>!std::rt::lang_start::{{closure}}::h919e8dda9de0d648
   11: 0x3807 - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   12:  0xe68 - <unknown>!std::rt::lang_start::ha5e1690a3bb184c3
   13: 0x160e - <unknown>!__main_void
   14:  0x2f6 - <unknown>!_start
   15: 0x9b66 - <unknown>!__wasi_environ_get
   16: 0x9b24 - <unknown>!__wasilibc_initialize_environ
   17: 0x9a96 - <unknown>!__wasilibc_ensure_environ
   18: 0x9c4d - <unknown>!getenv
   19: 0x3b2a - <unknown>!std::env::_var_os::ha6d753b2be7fe17f
   20: 0x3c99 - <unknown>!std::env::_var::hd7c38fcd0ed4a72d
   21: 0x1829 - <unknown>!std::env::var::hd342252c7c9996f3
   22: 0x14f4 - <unknown>!guest::main::hb3d545227282da60
   23:  0xac4 - <unknown>!core::ops::function::FnOnce::call_once::hdf6e14f7ebde9f53
   24:  0xf43 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::ha23fe57107e63372
   25:  0xecb - <unknown>!std::rt::lang_start::{{closure}}::h919e8dda9de0d648
   26: 0x3807 - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   27:  0xe68 - <unknown>!std::rt::lang_start::ha5e1690a3bb184c3
   28: 0x160e - <unknown>!__main_void
   29:  0x2f6 - <unknown>!_start
note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information

Caused by:
    missing required memory export

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:16):

the original env get function is accessing the wasm module's memory, which is required to be an export named memory

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:17):

so, whatever module you are running needs to provide one of those

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:20):

I think it does?

        println!("exports:");
        for export in module.exports() {
            println!("  {}", export.name());
        }
exports:
  memory
  _start
  __main_void

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:21):

do I need to redeclare that the wrapped function needs memory somehow?

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:22):

my guest program is pretty close to hello world:

use std::env;

fn main() {
    let term = env::var("TERM").unwrap_or_else(|_| "unknown".to_string());
    println!("hello from wasm guest! your terminal is: {term}");
}

built with cargo build --target wasm32-wasi

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:22):

oh, hmm

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:23):

instead of passing env_get.typed(...) a &mut Store, can you pass it the Caller?

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:24):

oh, generate the typed wrapper on every call?

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:24):

not sure how else to get a Caller

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:26):

looks like the same error/stack trace:

    let original_env_get = {
        let Some(wasmtime::Extern::Func(env_get)) =
            linker.get(&mut store, "wasi_snapshot_preview1", "environ_get")
        else {
            anyhow::bail!("missing environ_get");
        };
        env_get
    };

    linker.allow_shadowing(true);
    linker.func_wrap(
        "wasi_snapshot_preview1",
        "environ_get",
        move |mut caller: Caller<'_, _>, a: i32, b: i32| {
            let typed: TypedFunc<(i32, i32), i32> = original_env_get.typed(&mut caller).unwrap();
            typed.call(caller, (a, b))
        },
    )?;

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:26):

yeah what if you make original_env_get bind to jut env_get instead of env_get.typed(store).unwrap

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:26):

hmm

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:26):

$ cargo run -p host -- ./target/wasm32-wasi/debug/guest.wasm
   Compiling host v0.1.0 (/Users/benl/src/hello-wasmtime/host)
    Finished dev [unoptimized + debuginfo] target(s) in 1.53s
     Running `target/debug/host ./target/wasm32-wasi/debug/guest.wasm`
exports:
  memory
  _start
  __main_void
Error: error while executing at wasm backtrace:
    0: 0x9b66 - <unknown>!__wasi_environ_get
    1: 0x9b24 - <unknown>!__wasilibc_initialize_environ
    2: 0x9a96 - <unknown>!__wasilibc_ensure_environ
    3: 0x9c4d - <unknown>!getenv
    4: 0x3b2a - <unknown>!std::env::_var_os::ha6d753b2be7fe17f
    5: 0x3c99 - <unknown>!std::env::_var::hd7c38fcd0ed4a72d
    6: 0x1829 - <unknown>!std::env::var::hd342252c7c9996f3
    7: 0x14f4 - <unknown>!guest::main::hb3d545227282da60
    8:  0xac4 - <unknown>!core::ops::function::FnOnce::call_once::hdf6e14f7ebde9f53
    9:  0xf43 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::ha23fe57107e63372
   10:  0xecb - <unknown>!std::rt::lang_start::{{closure}}::h919e8dda9de0d648
   11: 0x3807 - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   12:  0xe68 - <unknown>!std::rt::lang_start::ha5e1690a3bb184c3
   13: 0x160e - <unknown>!__main_void
   14:  0x2f6 - <unknown>!_start
   15: 0x9b66 - <unknown>!__wasi_environ_get
   16: 0x9b24 - <unknown>!__wasilibc_initialize_environ
   17: 0x9a96 - <unknown>!__wasilibc_ensure_environ
   18: 0x9c4d - <unknown>!getenv
   19: 0x3b2a - <unknown>!std::env::_var_os::ha6d753b2be7fe17f
   20: 0x3c99 - <unknown>!std::env::_var::hd7c38fcd0ed4a72d
   21: 0x1829 - <unknown>!std::env::var::hd342252c7c9996f3
   22: 0x14f4 - <unknown>!guest::main::hb3d545227282da60
   23:  0xac4 - <unknown>!core::ops::function::FnOnce::call_once::hdf6e14f7ebde9f53
   24:  0xf43 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::ha23fe57107e63372
   25:  0xecb - <unknown>!std::rt::lang_start::{{closure}}::h919e8dda9de0d648
   26: 0x3807 - <unknown>!std::rt::lang_start_internal::h409072ad2c29d9a2
   27:  0xe68 - <unknown>!std::rt::lang_start::ha5e1690a3bb184c3
   28: 0x160e - <unknown>!__main_void
   29:  0x2f6 - <unknown>!_start

Caused by:
    missing required memory export

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:27):

just in case you can see something interesting in the stack trace, there it is :)

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:28):

would you mind adding the RUST_BACKTRACE=1? thats the wasm backtrace, which is helpful, but not the native frames

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:29):

Stack backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2: std::backtrace::Backtrace::create
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/backtrace.rs:331:13
   3: anyhow::error::<impl anyhow::Error>::msg
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   4: anyhow::__private::format_err
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/lib.rs:684:13
   5: wasmtime_wasi::sync::snapshots::preview_1::add_wasi_snapshot_preview1_to_linker::{{closure}}::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-wasi-16.0.0/src/lib.rs:66:9
   6: wiggle::run_in_dummy_executor
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wiggle-16.0.0/src/lib.rs:1171:11
   7: wasmtime_wasi::sync::snapshots::preview_1::add_wasi_snapshot_preview1_to_linker::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-wasi-16.0.0/src/lib.rs:66:9
   8: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2),R>>::into_func::native_call_shim::{{closure}}::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1974:41
   9: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
  10: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panic/unwind_safe.rs:271:9
  11: std::panicking::try::do_call
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504:40
  12: ___rust_try
  13: std::panicking::try
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468:19
  14: std::panic::catch_unwind
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142:14
  15: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2),R>>::into_func::native_call_shim::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1969:29
  16: wasmtime::func::Caller<T>::with::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1786:13
  17: wasmtime_runtime::instance::Instance::from_vmctx
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/instance.rs:240:9
  18: wasmtime::func::Caller<T>::with
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1784:9
  19: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2),R>>::into_func::native_call_shim
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1958:34
  20: <(A1,A2) as wasmtime::func::typed::WasmParams>::invoke::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func/typed.rs:587:21
  21: <(A1,) as wasmtime::func::HostAbi>::call
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1693:18
  22: <(A1,A2) as wasmtime::func::typed::WasmParams>::invoke
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func/typed.rs:586:17
  23: wasmtime::func::typed::TypedFunc<Params,Results>::call_raw::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func/typed.rs:181:17
  24: wasmtime_runtime::traphandlers::catch_traps::call_closure
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/traphandlers.rs:241:18
  25: wasmtime_setjmp_16_0_0
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/helpers.c:66:3
  26: wasmtime_runtime::traphandlers::catch_traps::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/traphandlers.rs:219:13
  27: wasmtime_runtime::traphandlers::<impl wasmtime_runtime::traphandlers::call_thread_state::CallThreadState>::with::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/traphandlers.rs:353:44
  28: wasmtime_runtime::traphandlers::tls::set
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/traphandlers.rs:724:13
  29: wasmtime_runtime::traphandlers::<impl wasmtime_runtime::traphandlers::call_thread_state::CallThreadState>::with
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/traphandlers.rs:353:19
  30: wasmtime_runtime::traphandlers::catch_traps
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/traphandlers.rs:217:18
  31: wasmtime::func::invoke_wasm_and_catch_traps
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1371:22
  32: wasmtime::func::typed::TypedFunc<Params,Results>::call_raw
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func/typed.rs:177:22
  33: wasmtime::func::typed::TypedFunc<Params,Results>::call
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func/typed.rs:88:18
  34: host::main::{{closure}}
             at ./host/src/main.rs:37:13
  35: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2),R>>::into_func::native_call_shim::{{closure}}::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1974:41
  36: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
  37: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panic/unwind_safe.rs:271:9
  38: std::panicking::try::do_call
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504:40
  39: ___rust_try
  40: std::panicking::try
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468:19
  41: std::panic::catch_unwind
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142:14
  42: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2),R>>::into_func::native_call_shim::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1969:29
  43: wasmtime::func::Caller<T>::with::{{closure}}
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1786:13
  44: wasmtime_runtime::instance::Instance::from_vmctx
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-runtime-16.0.0/src/instance.rs:240:9
  45: wasmtime::func::Caller<T>::with
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1784:9
  46: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1,A2),R>>::into_func::native_call_shim
             at /Users/benl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-16.0.0/src/func.rs:1958:34
  47: <unknown>

view this post on Zulip Pat Hickey (Jan 23 2024 at 17:30):

hmm, yeah, thats where id expect it to come from, so im stumped at the moment and i need to run. ill see if i can get back to this later

view this post on Zulip Ben Linsay (Jan 23 2024 at 17:30):

appreciate it! thanks!

view this post on Zulip Ben Linsay (Jan 25 2024 at 19:07):

@Pat Hickey hey, have you had a chance to take another look? no rush, and if you want to just drop it, nbd :)

view this post on Zulip Pat Hickey (Jan 26 2024 at 01:51):

sorry, been busy with getting preview 2 out the door. if you can upload a reasonably small reproduction, i'll take a look tomorrow

view this post on Zulip Ben Linsay (Jan 26 2024 at 02:23):

congrats and no worries!

here's a gist: https://gist.github.com/blinsay/e9571a15fc1f59a355858cdba307946f

GitHub Gist: instantly share code, notes, and snippets.

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:37):

hi, i finally got time to look at the reproduction, thanks for waiting

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:38):

i believe what is going on here is that you have called a method in your linker without first creating an instance of the module

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:39):

or creating any instance at all, basically

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:40):

you still need to call linker.instantiate somewhere

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:40):

otherwise, wasmtime tries to be helpful about calling Funcs from the "host context" where there is essentially just an empty instance

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:40):

and that empty instance doesnt have a memory

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:46):

actually, nevermind that explanation, I changed it to how it "should" work and its still broken

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:46):

which was, for reference,

    // shadowing stuff ends

    let module = Module::from_file(&engine, prog_path)?;
    let inst = linker.instantiate(&mut store, &module)?;
    inst.get_func(&mut store, "_start")
        .ok_or_else(|| anyhow::anyhow!("_start export required"))?
        .typed::<(), ()>(&store)?
        .call(&mut store, ())?;

    Ok(())
}

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:51):

@Alex Crichton can you help me understand why this line uses the closure captured vmctx and not one belonging to the caller https://github.com/bytecodealliance/wasmtime/blob/ab5a4484ebac8d1f08f773d244baedb09b90a29b/crates/wasmtime/src/func.rs#L1959

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:53):

im trying to reason about why there are two different host states for those vmctxs

view this post on Zulip Pat Hickey (Jan 29 2024 at 19:55):

basically, the behavior of the host-to-host call of func isnt retaining the exports id expect to be present in the store

view this post on Zulip Pat Hickey (Jan 29 2024 at 20:01):

@Ben Linsay from the above questions to alex i am actually pretty stumped on whats causing this behavior. maybe alex will be able to help me understand whats up here

view this post on Zulip Alex Crichton (Jan 29 2024 at 20:04):

hm the answer to that is basically "it'll be memory unsafe if we didn't" in the sense that each vmctx has slightly different host state, and the closure-captured one there has the closure specified to the linker as its state and that's what's being loaded and called. The caller's host_state I don't think corresponds to T in Store<T> and most of this level of detail is divorced from the overall embedding API.

view this post on Zulip Alex Crichton (Jan 29 2024 at 20:04):

so I might need to understand more what's going on here to answer better

view this post on Zulip Alex Crichton (Jan 29 2024 at 20:17):

oh I think I see what you're getting at, and this is indeed confusing to me as well. I'll need to puzzle this out a bit

view this post on Zulip Alex Crichton (Jan 29 2024 at 20:20):

So I think it makes sense that this doesn't work right now. I think it's fine to consider that a bug, however.

The problem is that this has to do with how WASI finds the exported memory. When you work with WASI as it's set up by default with preview1 you never actually configure what memory to use, so each function has to look up on the caller what the memory export is. In this case though the "caller" is Rust host code itself, which means the "caller" doesn't have a memory export

view this post on Zulip Alex Crichton (Jan 29 2024 at 20:21):

right now there's no way to fake "as if this was called by wasm", so I don't think that there's a way around this right now unfortunately.

view this post on Zulip Pat Hickey (Jan 29 2024 at 20:35):

Yeah I thought the Caller would transfer that information but it is more subtle than that

view this post on Zulip Pat Hickey (Jan 29 2024 at 20:35):

Anyway, ok, thanks for your help Alex

view this post on Zulip Pat Hickey (Jan 29 2024 at 20:36):

@Ben Linsay I bet if you were using components and wasi preview 2 you wouldn’t have this problem. Want to try that instead? The distinction of being called from wasm goes away with components, and in general a lot of the complexity of an implementation of get environment is abstracted away

view this post on Zulip Pat Hickey (Jan 29 2024 at 20:38):

If you do indeed need the wasi-common preview 1 implementation we could get you dispatching to the wiggle trait function in your func_wrap, which won’t have the same problems with caller as getting it from the linker

view this post on Zulip Pat Hickey (Jan 29 2024 at 20:39):

But, I’m biased but if I had to recommend anything to authors of new systems, just use components! They solve tons of problems you may not even know you have yet :)

view this post on Zulip Ben Linsay (Jan 30 2024 at 14:18):

thanks for taking a look!

view this post on Zulip Ben Linsay (Jan 30 2024 at 14:20):

sure, components sound great. where do I find docs/examples?

view this post on Zulip Alex Crichton (Jan 30 2024 at 14:39):

I'd recommend starting at https://component-model.bytecodealliance.org/ for components

view this post on Zulip Ben Linsay (Jan 30 2024 at 15:57):

ty!

just took a quick skim and it's not immediately obvious to me how i'd use this to change the behavior of a specific wasi function. would I have to re-declare the entire wasi component and forward everything else along?

view this post on Zulip Pat Hickey (Jan 30 2024 at 16:01):

There’s a lot of ways. You could do it entirely with component composition - make a component that imports get-environment and exports the same, and use wasm-tools compose to interpose it

view this post on Zulip Pat Hickey (Jan 30 2024 at 16:01):

You can also use the host, like you were trying before, but with a wasmtime::component::Linker

view this post on Zulip Ben Linsay (Jan 30 2024 at 16:03):

oh interesting

view this post on Zulip Pat Hickey (Jan 30 2024 at 16:03):

If you do that you should consider using wasmtime::component::bindgen! to generate the bindings for the cli environment interface, and then add it to the linker

view this post on Zulip Ben Linsay (Jan 30 2024 at 16:03):

which way would you recommend?

view this post on Zulip Pat Hickey (Jan 30 2024 at 16:04):

Start with just the host, it’s easier I thibk

view this post on Zulip Ben Linsay (Jan 30 2024 at 16:04):

that checks out

view this post on Zulip Ben Linsay (Jan 30 2024 at 16:05):

do I still grab a reference to environ_get the same way I was doing before?

view this post on Zulip Ben Linsay (Jan 30 2024 at 16:06):

or is that going to come through the bingen! thing you're suggesting?

view this post on Zulip Pat Hickey (Jan 30 2024 at 16:07):

No, I don’t think so, you’ll call it by rust symbol instead of getting it from the Linker in components. We don’t have allow_shadowing on the component::Linker to my knowledge so, until we get around to adding it, copy the implementation of wasmtime_wasi::preview2::command::add_to_linker and then substitute just your cli::environment add_to_linker in there

view this post on Zulip Ben Linsay (Jan 30 2024 at 16:07):

cool, will sit down this afternoon and give it a whirl. thanks dude!

view this post on Zulip Ben Linsay (Feb 03 2024 at 00:25):

came back to this a few days later than I meant to, and I'm still a bit lost. I read through the docs for the bindgen! macro (which are nice and detailed, ty!) and tried to copy what was going on there just to get started.

I got bindings by copying the wit directory from wasi-cli into a local directory and I can get the bindgen macro making bindings for it with:

bindgen!({
    world: "imports"
});

That seems really reasonable, and I can struct MyEnv; impl Host for MyEnv {...} now, which is great, but I have no idea how to find the symbol for the original implementation of get-environment. Any suggestions on where to find it?

I did go digging into the docs for wasmtime-wasi and found out that the Host trait is already generated there. Do I need to do the bindgen dance at all?

view this post on Zulip Ben Linsay (Feb 03 2024 at 00:27):

I did notice that component::Linker has an allow_shadowing method btw :)

https://docs.rs/wasmtime/latest/wasmtime/component/struct.Linker.html#method.allow_shadowing

view this post on Zulip Ben Linsay (Feb 03 2024 at 00:28):

kinda dead-ended on the host approach right now. I'll go try the component composition approach later this weekend too.

view this post on Zulip Alex Crichton (Feb 03 2024 at 18:02):

If your goal is to override a small handful of WASI methods then you can probably skip the bindgen yeah. All the types you need are in the wasmtime-wasi crate and with allow_shadowing you can basically redefine an existing wasi function with your own custom closure (using the same signature as before). Accessing the previous implementation can be done by calling the Host trait method directly (e.g. wasmtime_wasi::preview2::bindings::wasi::cli::environment::Host::get_environment(...))

view this post on Zulip Ben Linsay (Feb 03 2024 at 20:38):

@Alex Crichton dumb question, what do I pass as self if I call Host methods directly? I don't see anything that actually implements the trait

view this post on Zulip Alex Crichton (Feb 03 2024 at 21:11):

The self argument is anything that implements the WasiView trait which is typically the T in Store<T>

view this post on Zulip Ben Linsay (Feb 03 2024 at 22:56):

okay, I'm still completely lost. I'm trying to mirror what I had originally but I haven't gotten anything involving wasmtime::component to compile. :(

Definitely feels like I'd benefit from doing the basic stuff first, so I'm going to go play with components and composing them before coming back to this. Thanks for all your help so far.

view this post on Zulip Pat Hickey (Feb 05 2024 at 18:49):

our tutorials and docs for this whole category are not as polished as they need to be, but we're going to work on it :)

view this post on Zulip Pat Hickey (Feb 05 2024 at 18:50):

https://component-model.bytecodealliance.org/ is a great start. please do report here (or file at https://github.com/bytecodealliance/component-docs) if anything in there isnt correct / doesnt work

Documentation around creating and using WebAssembly Components - GitHub - bytecodealliance/component-docs: Documentation around creating and using WebAssembly Components

view this post on Zulip Ben Linsay (Feb 06 2024 at 17:03):

the docs definitely got me started! I managed to understand what a component was, run through the examples, and get my own host program running a command component fairly easily. thanks :D

I'm back to trying to shadow get-environment. I can't figure out how to inspect the linker/components to see if I've actually gotten the right instance and redefined the right thing. any tips on where to go from here?

use anyhow::Context;
use std::env;
use wasmtime::component::Component;
use wasmtime::{Config, Engine, Store};
use wasmtime_wasi::preview2;

fn main() -> anyhow::Result<()> {
    let path = env::args().skip(1).next().unwrap();

    eprintln!("starting");
    let mut config = Config::default();
    config.wasm_component_model(true);

    let engine = Engine::new(&config)?;
    let wasi_view = ServerWasiView::new();
    let mut store = Store::new(&engine, wasi_view);

    let mut linker: wasmtime::component::Linker<ServerWasiView> =
        wasmtime::component::Linker::new(&engine);
    preview2::command::sync::add_to_linker(&mut linker).context("failed to link command world")?;

    // --- shadowing stuff: nothing here seems to get called ----
    linker.allow_shadowing(true);
    let mut instance = linker
        .instance("wasi/cli:environment")
        .context("getting the wasi/cli:environment instance")?;
    instance
        .func_wrap("get-environment", |mut ctx, ()| {
            eprintln!("called from the host wrapper");
            // NOTE: this is terribly janky
            let env = wasmtime_wasi::preview2::bindings::cli::environment::Host::get_environment(
                ctx.data_mut(),
            )?;
            Ok(env.first().unwrap().clone())
        })
        .context("overriding get-environment")?;
    // --- end shadowing stuff

    eprintln!("loading component");
    let component = Component::from_file(&engine, path).context("loading component failed")?;

    let (instance, _) = wasmtime_wasi::preview2::command::sync::Command::instantiate(
        &mut store,
        &component,
        &mut linker,
    )
    .context("failed to instantiate Command")?;

    let res = instance.wasi_cli_run().call_run(&mut store);
    println!("run -> {res:?}");

    Ok(())
}

view this post on Zulip Pat Hickey (Feb 06 2024 at 17:58):

syntax is a little off and you'll need to add the version number to the package- so wasi:cli@0.2.0/environment

view this post on Zulip Pat Hickey (Feb 06 2024 at 17:59):

we could add a getter to linker that fails if the thing isnt there

view this post on Zulip Ben Linsay (Feb 06 2024 at 18:14):

ahhhh that'd be really nice :)

view this post on Zulip Ben Linsay (Feb 06 2024 at 18:15):

it was really nice to be able to list imports/exports with modules. i don't see a way to do that with the component APIs. am I missing something or do those not exist (yet)?

view this post on Zulip Pat Hickey (Feb 06 2024 at 18:22):

nope, doesnt exist

view this post on Zulip Pat Hickey (Feb 06 2024 at 18:25):

agree that it should, we can add it


Last updated: Dec 23 2024 at 14:03 UTC