Stream: general

Topic: wasi_snapshot_preview1 imports in wasi export

Ari Seyhun (Nov 09 2022 at 13:51):

I've written a basic rust lib with an exported function, and compiled it to wasm32-wasi with crate-type set to cdylib.
And from another Rust crate, I'm trying to load this wasm module with wasmtime, and instantiate the exported function, though it gives an error "expected 4 imports, found 0".

When I inspect the wasm file to wat, I see that there are 4 imports defined:

  (import "wasi_snapshot_preview1" "fd_write" (func $wasi_snapshot_preview1.fd_write (type $t6)))
  (import "wasi_snapshot_preview1" "environ_get" (func $wasi_snapshot_preview1.environ_get (type $t5)))
  (import "wasi_snapshot_preview1" "environ_sizes_get" (func $wasi_snapshot_preview1.environ_sizes_get (type $t5)))
  (import "wasi_snapshot_preview1" "proc_exit" (func $wasi_snapshot_preview1.proc_exit (type $t0)))

I understand that I could define these functions and provide them as imports when instantiating the module, but I'm wondering if wasmtime provides any default imports for these common functions so I don't have to figure out the code for fd_write, environ_get, etc.?

Ari Seyhun (Nov 09 2022 at 14:17):

Oh I just came across the wasi crate.
Though, is there any examples on integrating this with wasmtime?

Ari Seyhun (Nov 09 2022 at 14:33):

I'm almost there.
I've found the wasmtime_wasi::add_to_linker, but I'm just trying to figure out how to make the WasiCtx::new. All I've managed to create is the Rng with Box::new(OsRng).

Joel Dice (Nov 09 2022 at 14:34):

e.g. WasiCtxBuilder::new().arg("<wasm module>")?.build()

Joel Dice (Nov 09 2022 at 14:35):

See https://docs.rs/wasmtime-wasi/2.0.1/wasmtime_wasi/sync/struct.WasiCtxBuilder.html for full docs.

Ari Seyhun (Nov 09 2022 at 14:35):

Ohh, I didnt see that!
What is .arg("<wasm module>") for examle? Is it needed?

Joel Dice (Nov 09 2022 at 14:36):

I just pasted that from some code I wrote a while ago. Probably not needed unless your module expects at least one argv value.

Joel Dice (Nov 09 2022 at 14:36):

Doesn't look like yours uses WASI arguments at all, so not relevant.

Ari Seyhun (Nov 09 2022 at 14:49):

This should probably be another topic post, but I'll just ask here.

I'm trying to define a function in my Rust wasm lib which will eventually be Vec<u8>, but since its wasm, I'm taking in offset and length:

#[no_mangle]
pub fn new(offset: usize, len: usize) {
    println!("{id_offset} {id_len}");
}

And from my wasmtime, I have written to the memory some random bytes:

let memory = instance.get_memory(store, "memory").unwrap();
memory.write(&mut store, 0, &[0, 1, 2])?; // write 0, 1, 2

let new = instance.get_typed_func::<(i32, i32), (), _>(&mut store, "new")?;
new.call(&mut store, (0, 3))?;

But now my question is, how do I actually read this memory from my new fn? I can print the offset and length correctly, but I don't know how to read my own wasm memory from the Rust file which is compiled.

Joel Dice (Nov 09 2022 at 14:55):

I think you'll need to cast offset to a const u8* and then use e.g. https://doc.rust-lang.org/std/slice/fn.from_raw_parts.html to produce a &[u8].

Joel Dice (Nov 09 2022 at 14:56):

I'd be pretty nervous about writing to guest memory like that without involving the guest allocator, though.

Ari Seyhun (Nov 09 2022 at 14:58):

Ohh, so this approach isn't really correct? I should be using some kind of allocator?
I saw in the docs for wasmtime memory, the only option really is .write()

Lann Martin (Nov 09 2022 at 15:00):

The usual approach is to export some equivalent of malloc from the guest which the host can use to allocate a destination for writes

Ari Seyhun (Nov 09 2022 at 15:01):

Ah okay, I've just come across this:
<https://radu-matei.com/blog/practical-guide-to-wasm-memory/>

I thought Rust would already export something like this for you, but I guess not

Lann Martin (Nov 09 2022 at 15:02):

Depending on your specific situation you could possibly export some other kind of pointer from the guest like a statically allocated buffer, but then you take responsibility for managing that

Ari Seyhun (Nov 09 2022 at 15:19):

Hmm, when I define my function to take a Rust type, such as String:

#[no_mangle]
pub fn new(value: String) { }

It produces this wat code:

(func $new (type $t0) (param $p0 i32)

Is there anywhere I can read up on what this param is? Is it just a pointer to the string?

bjorn3 (Nov 09 2022 at 15:33):

The rust abi is unstable. If you want to export a function from the wasm module you will have to use #[no_mangle] extern "C" fn new. You also can't use String as it has an unstable layout. Using wit-bindgen is probably the easiest way for exporting an interface from wasm that uses non-primitive types like strings.

Ari Seyhun (Nov 09 2022 at 15:36):

Thanks! I'm looking into &str now since it's just a pointer and a length.
When invoking the function, I'm sending:

let offset = 128; // arbitrary
memory.write(&mut store, 128, b"hey")?;
let base = memory.data_ptr(&mut store) as i32 + offset;
let new = instance.get_typed_func::<(i32, i32), (), _>(&mut store, "new")?;
new.call(&mut store, (base, 3))?;

But I get an error of out of bounds memory access.

Is memory.data_ptr() the right method to get the start address of the memory?

Lann Martin (Nov 09 2022 at 15:44):

The guest code sees its own linear memory starting at zero

Joel Dice (Nov 09 2022 at 15:44):

From the guest's point of view, memory starts at zero, so base should be 128. Note that since you didn't ask the guest to allocate memory you're using, you're clobbering whatever was already at that address, with unpredictable consequences.

Ari Seyhun (Nov 09 2022 at 15:46):

I see, though I'm only calling an exported function which should have no side effects. So I'd assume the memory would be compleltely empty the first time I invoke any function :thinking:

Ari Seyhun (Nov 09 2022 at 15:47):

Oh I actually got it working!

let mut data = 8_i32.to_le_bytes().to_vec();
data.extend(3_i32.to_le_bytes());
data.push(b"Hey");
memory.write(&mut store, 0, &data)?;

let new = instance.get_typed_func::<i32, (), _>(&mut store, "new")?;
new.call(&mut store, 0)?;

Lann Martin (Nov 09 2022 at 15:48):

I'd assume the memory would be compleltely empty the first time I invoke any function

Depends on the source language/runtime. Almost all languages will stick something somewhere on startup, at least in some contexts

Ari Seyhun (Nov 09 2022 at 15:50):

Since:

• I know exactly what functions are exported, and their types.
• The wasm will always come from Rust (for now).
  I think it's worth doing this approach for the sake of avoiding having to export an allocator, and writing glue code

Ari Seyhun (Nov 09 2022 at 15:51):

Though, I realise Rust didn't compile for that &str to be there, so when I write new memory, it could overwrite the memory of the string I guess...

Joel Dice (Nov 09 2022 at 15:52):

Also note that even a Rust function with no side effects and no explicit allocations may use the "shadow" stack, which lives on the heap, so you may find that whatever you write to memory gets clobbered once the function is run.

Lann Martin (Nov 09 2022 at 15:53):

And statically allocated data goes in the linear memory too. You really don't want to do it that way.

Joel Dice (Nov 09 2022 at 15:54):

It is possible to write a Rust guest that never touches memory, but it's surprisingly difficult and limiting.

Ari Seyhun (Nov 09 2022 at 15:57):

My guest will always export only these 3 basic functions:

#[no_mangle]
pub fn new(name: String) -> Vec<u8> { ... }

#[no_mangle]
pub fn apply(mut state: Vec<u8>, event: Vec<u8>) -> Vec<u8> { ... }

#[no_mangle]
pub fn handle(state: Vec<u8>, command: Vec<u8>) -> Vec<u8> { ... }

What if I export some static buffers for this instead of the parameters?

#[no_mangle]
pub static PARAM_1: [u8; 128] = [0; 128];

Would something like this work fine?

Lann Martin (Nov 09 2022 at 16:03):

It should. You'll have to beware buffer overflows

Ari Seyhun (Nov 09 2022 at 16:04):

Sounds like a nice solution then!
I can make some macros #[new], #[apply], and #[handle] which declare these statics and transform the bytes into the correct types.

Ramon Klass (Nov 09 2022 at 16:04):

still it would be strongly suggested to write those 3 functions with wit files, it will continue to work if the standard evolves

Ari Seyhun (Nov 09 2022 at 16:21):

I realised this gets complicated if I want to take a parameter of state: Vec<u8>, since state could become arbitrarily large.

Ari Seyhun (Nov 10 2022 at 04:14):

I'm trying to use wit-bindgen to compile a gust module from Rust, and run it in a host with wasmtime.

My guest code uses:

wit_bindgen_guest_rust::generate!({
    name: "module",
    export: "module.wit",
});

export_module!(MyModule);

struct MyModule { .. }
impl module::Module for MyModule { ... }

And I compile it to wasm32-wasi.

Then on the host side, I'm trying to use:

wit_bindgen_host_wasmtime_rust::generate!({
    name: "module",
    import: "module.wit",
});

struct ModuleHandler {}
impl module::Module for ModuleHandler { ... }

let module = Module::from_file(&self.engine, file)?;
module::add_to_linker::<Data, _>(&mut linker, |cx| &mut cx.module_handler)?;
/* ... */

Though my first problem came when it was complaining that my linker is from wasmtime::Linker, and it wanted wasmtime::component::Linker.

So I changed my items to use exports from wasmtime::component::{...}, but when running my code I get the error:

Error: failed to parse WebAssembly module

Caused by:
attempted to parse a wasm module with a component parser

Ari Seyhun (Nov 10 2022 at 04:25):

I'm quite confused why I'd need to implement a trait when I'm importing too. I would expect it would be something just like module::new(&instance, "param".to_string());

Ari Seyhun (Nov 10 2022 at 04:41):

Do I need to compile my guest rust to wasm in a different way to make it a "component"?
Any help would be appreciated, I cant figure this out at all

Ari Seyhun (Nov 10 2022 at 05:00):

Okay I've discovered the wit-component cli. But when trying to use it to convert my wasm file, I get this error:

module requires an import interface named wasi_snapshot_preview1

Ari Seyhun (Nov 10 2022 at 05:09):

❯ wit-component ./target/wasm32-wasi/release/counter.wasm --adapt ../../../../bytecodealliance/wit-bindgen/target/wasm32-wasi/release/wasi_snapshot_preview1.wasm
[2022-11-10T05:09:10Z ERROR] failed to encode a component from module `./target/wasm32-wasi/release/counter.wasm`

    Caused by:
        0: failed to reduce input adapter module to its minimal size
        1: locally-defined memories are not allowed define a local memory

Ari Seyhun (Nov 10 2022 at 05:38):

I got it working by compiling things with rustflags. Though now I'm trying to figure out why it says:

Error: failed to parse WebAssembly module

Caused by:
    WebAssembly component model feature not enabled

Ari Seyhun (Nov 10 2022 at 06:20):

I got it all working in the end :)

Last updated: Nov 22 2024 at 17:03 UTC