Instrumenting wasi calls · general

Is it possible to instrument wasmtime source code such that I can redirect all wasi calls to some monitor first before the actual wasi call?
Is this feasible?

Alex Crichton (Jan 09 2025 at 19:19):

At a conceptual level this is possible because WASI is inserted by calling add_to_linker and you can instead add whatever you want to the linker instead (and avoid calling add_to_linker). From a practical standpoint that's difficult to do because you'd have to duplicate all the type signatures and such, however.

celine santosh (Jan 10 2025 at 15:22):

Thanks, could you please also show a file in wasmtime repo where add_to_linker is called on a wasi function

Alex Crichton (Jan 10 2025 at 15:24):

Ah alas I don't think we have an example of doing this, but the rough idea is that you'd call functions like this manually.

Alex Crichton (Jan 10 2025 at 15:25):

e.g. you'd create a Linker, add a custom function to it, and your custom function would forward to that.

Alternatively, now that I think about it, something that would be much more robust would be to write your own impl of WasiSnapshotPreview1 which forwards to the wasmtime-wasi one, allowing you to interpose any function (although you're required to interpose all of them or at least forward arguments)

Joel Dice (Jan 10 2025 at 15:32):

Another option would be to create a component which both imports and exports all the interfaces in the wasi-cli imports world, forwarding every call to the host and also sending monitoring events via another, custom imported interface.

celine santosh (Jan 10 2025 at 19:22):

I would require network calls to be redirected , I guess func_wrap is the one which replaces it
but func_wrap() is not there in component model

celine santosh (Jan 10 2025 at 19:24):

Since I am using network calls especially tcp sockets this won't work in my case right

celine santosh (Jan 10 2025 at 19:25):

Is there simpler option like for example in wasmtime repository , in tcp.rs file where start-connect is defined, I can call my custom function there? Changing the source code of wasmtime is what I meant by instrumenting.
will it break any other functionality?

Joel Dice (Jan 10 2025 at 19:27):

Yeah, you can certainly fork wasmtime-wasi and add your custom code there. That's probably the quickest option, although you'd need to maintain that fork yourself if you plan to use it long-term. Perhaps if the changes are generic and broadly useful, they could be upstreamed.

celine santosh (Jan 12 2025 at 20:37):

celine santosh (Jan 20 2025 at 18:44):

So I have created a runner.rs where I create an engine , store, state linker etc.
and I have inserted all wasi calls except sockets/tcp.

pub fn add_to_linker_without_tcp<T: WasiView>(
    linker: &mut wasmtime::component::Linker<T>,
) -> anyhow::Result<()> {
    let l = linker;
    let closure = type_annotate::<T, _>(|t| WasiImpl(t));

    wasmtime_wasi::bindings::clocks::wall_clock::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::clocks::monotonic_clock::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::filesystem::types::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::filesystem::preopens::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::io::error::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::io::poll::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::io::streams::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::random::random::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::random::insecure::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::random::insecure_seed::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::exit::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::environment::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::stdin::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::stdout::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::stderr::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_input::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_output::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_stdin::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_stdout::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_stderr::add_to_linker_get_host(l, closure)?;
    //wasmtime_wasi::bindings::sync::sockets::tcp::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::tcp_create_socket::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::sockets::udp::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::udp_create_socket::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::instance_network::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::network::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::ip_name_lookup::add_to_linker_get_host(l, closure)?;
    Ok(())
}

So Instead of the wasi call start-connect I want to add my own custom start-connect from another package say mwasi:sockets/tcp which is in wrapper module.
this is the wit file of wrapper module

package mwasi:sockets@0.2.0;

interface tcp{

  use wasi:sockets/network@0.2.0.{network, error-code, ip-socket-address, ip-address-family};

  start-connect: func(networkinstance: network, remoteaddress: ip-socket-address) -> result<_, error-code>; //(result cm.Result[ error-code, struct{},  error-code])//result<_, error-code>;

}

world wrapper{
  import wasi:io/error@0.2.0;
  import wasi:sockets/tcp@0.2.0;
  import wasi:sockets/network@0.2.0;
  export wasi:cli/run@0.2.0;
  include wasi:cli/imports@0.2.0;
  export tcp;
}

How can I insert this interface tcp (which I created) into linker so that when main app calls start-connect ,my version is triggered. I have seen the wit-bindgen macro. Is that the way?
I tried this method https://docs.wasmtime.dev/api/wasmtime/component/bindgen_examples/_5_all_world_export_kinds/index.html
But I don't know where to write the implementation of my exported interface. Also I want to implement it in Go which will be easier for my further steps.
Or should I convert my wrapper module into wasm and load it as component in my runner and insert it in linker or something ?

Pat Hickey (Jan 20 2025 at 18:48):

Yes, you want to run the wasmtime::component::bindgen! macro on your wit file, and use with to point all of the wasi implementations at the wasmtime_wasi crate's implementations

Pat Hickey (Jan 20 2025 at 18:49):

Pat Hickey (Jan 20 2025 at 18:50):

also, you dont want your world wrapper to export both cli/run and tcp unless you intend to require that all components in your world provide both of those exports, my interpretation of the above is you really just want your components to be invoked by the start-connect function

Pat Hickey (Jan 20 2025 at 18:51):

(I cant offer any help with your components being written in go, i'm not up to speed on anything go related)

celine santosh (Jan 20 2025 at 19:15):

Thanks a lot.
Actually I want the main application to call my start-connect instead of wasi start-connect, so that I can redirect all the network calls to a monitor which checks for compliance , and then proceed with the normal wasi start-connect.
So as the first step I have disabled the wasi start-connect from linker.
Next I will proceed with wit bindgen macro but I am confused where do I give the implementation of my start-connect?
Even in this example there are no implementations of bytes-to-string and duration-to-string in units interface.

Pat Hickey (Jan 20 2025 at 20:02):

by exposing your package's export tcp and not `wasi:cli/run", your world is saying "the only way to invoke this component is to call tcp start-connect"

Pat Hickey (Jan 20 2025 at 20:05):

if instead what you want is for components, when they want to create a tcp connection, to go through your interface and not the one exposed by wasi:sockets, you want to change your wit to say import tcp, because its available as an import function to the component, just like import wasi:sockets/network is making the wasi network functions available as imports.

Pat Hickey (Jan 20 2025 at 20:07):

Pat Hickey (Jan 20 2025 at 20:08):

and the return value of wasi_cli_run has a function call_run for the function named run inside that interface run (yes, repeated word makes it confusing)

Pat Hickey (Jan 20 2025 at 20:10):

changing that to import tcp will mean bindgen produces a totally different kind of wrapper. instead it will make a mod your_namespace { mod your_packagename { mod tcp { trait Host { fn start_connect(...) }, fn add_to_linker(...) } } }

Pat Hickey (Jan 20 2025 at 20:11):

and then you can follow the example of wasmtime-wasi for how you author an impl of that Host trait, and then you call that generated add_to_linker function into the custom one you made above.

Pat Hickey (Jan 20 2025 at 20:12):

when you said "the implementation of start-connect is in go" thats what made me think you wanted to call a component written in go.

Pat Hickey (Jan 20 2025 at 20:14):

when your tcp.start-connect is an import, you need to provide an impl Host of that trait in Rust, so you'll either need to reimplement that start-connect go code in Rust, or call out to the go code by some other means, maybe with std::process::Command to call an external go executable? idk, i'm not a go person but theres no straightforward way to call between go and rust that i know of.

celine santosh (Jan 21 2025 at 19:01):

celine santosh (Jan 21 2025 at 19:02):

I have a monitor which is implemented in Go , so I need to call one function from my custom start-connect function which check compliance and then proceeds with wasi start-connect.
If I load my monitor as a wasm component in runner.rs and retrieve the function I want to call, then I can call it from my custom start-connect function even if its in rust right?

Joel Dice (Jan 21 2025 at 19:37):

My understanding is that both Rust (via extern "C") and Go (via Cgo) can make use of the C ABI, allowing Rust and Go code to be linked statically or dynamically. I have experience with the Rust side of that, but none with the Go side. You'll probably want to research using Cgo and ask in a Go-specific forum if you have questions about that.

celine santosh (Jan 21 2025 at 19:45):

oh okay , But I don't understand , we can actually make use of wasm's language agnostic feature right?
By loading monitor as a wasm component and call its function from rust implementation. Isn't that possible?

Joel Dice (Jan 21 2025 at 19:52):

It depends on which approach you plan to use. In this thread, we've laid out three possible approaches (at least):

Of those approaches, only #1 involves Wasm composition, whereas my understanding was that you are planning to use either #2 or #3.

Perhaps you're proposing #2 or #3 but also recursively loading another component inside the host function? If so, yes, that could work. You'd just need to make sure you're using a separate Store for the instrumentation component.

celine santosh (Jan 23 2025 at 15:51):

Thanks
Right now I am trying out #2.
So I think it would be easier to load monitor as a component .

pub fn add_to_linker_without_tcp<T: WasiView>(
    linker: &mut wasmtime::component::Linker<T>,
) -> anyhow::Result<()> {
    let l = linker;
    let closure = type_annotate::<T, _>(|t| WasiImpl(t));

    wasmtime_wasi::bindings::clocks::wall_clock::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::clocks::monotonic_clock::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::filesystem::types::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::filesystem::preopens::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::io::error::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::io::poll::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::io::streams::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::random::random::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::random::insecure::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::random::insecure_seed::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::exit::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::environment::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::stdin::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::stdout::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::stderr::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_input::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_output::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_stdin::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_stdout::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::cli::terminal_stderr::add_to_linker_get_host(l, closure)?;
    //wasmtime_wasi::bindings::sync::sockets::tcp::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::tcp_create_socket::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sync::sockets::udp::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::udp_create_socket::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::instance_network::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::network::add_to_linker_get_host(l, closure)?;
    wasmtime_wasi::bindings::sockets::ip_name_lookup::add_to_linker_get_host(l, closure)?;
    Wrapper::add_to_linker(l, closure)?;
    Ok(())
}

I am not able to add the wrapper module which is generated by wit bindgen into the linker
I get error

error[E0271]: expected {closure@main.rs:74:41} to be a closure that returns &mut _, but it returns WasiImpl<&mut T>

| ---------------------- ^^^^^^^ expected &mut _, found WasiImpl<&mut T>

found struct **WasiImpl<&mut T>**
note: required by a bound in _::<impl Wrapper>::add_to_linker

Stream: general

Topic: Instrumenting wasi calls

celine santosh (Jan 09 2025 at 19:09):