Stream: general

Topic: Exporting void do_stuff(const char*) functions to wasmtime

Alexandru Ene (Jul 23 2020 at 13:03):

For various _reasons_ I recently needed to link some host functions to a few wasm VMs. WASMTime to me seems to be the outlier here on how bindings work. (my experience is between wamr, wasm3, wasmtime). The other two engines allow to some degree exposing a function that takes a const char* or any pointer to some data as long as there's a size also involved (wamr enforces this).

But, in wasmtime, I was looking at the arguments and found AnyRef (behind a flag - wasmtime_config_wasm_reference_types_set).
The rest of accepted values are i32/i64,f32,f64 and func_ref.

I couldn't find an example on how to correctly use anyref, and exporting a function as taking anyref as a const char * seems to trigger a link_error in linker.compute_imports. I almost wanted to give up and cast a pointer to i64 and pass that to the host vm, grab the memory from wasmtime and then re-create the memory location to that offset as a workaround, but that is semi-hard in the bindings as they don't depend on an engine context (like it's readily available wasm3/wamr) and just the params/results arrays. This makes for some really ugly solutions.

What is the correct way to expose a method that takes a const char* to wasmtime?
My question would be, is there an example that I could reuse for anyref?

Alex Crichton (Jul 23 2020 at 14:04):

@Alexandru Ene hm I'm not quite sure I understand what the functionality is that you're asking for, can you elaborate on what this const char* is in wamr/wasm3? Perhaps there's some example code I could poke around?

Alexandru Ene (Jul 23 2020 at 14:10):

So for example you can expose a host VM function that's basically: void PrintMeThisStuff(const char* stuff) { printf("%s\n", stuff); }
In WASM3 you need to export it by wrapping it as (it's basically a static function with these params:

const void* PrintMeThisStuffWrapper(IM3Runtime runtime, uint64_t* _sp, void* _mem) {
   m3ApiGetArgMem(const char*, msg);
   PrintMeThisStuff(msg); //real method called here
}

Then you bind it like so:

m3_LinkRawFunction(io_module, i_moduleName, i_functionName, "v(*)", &PrintMeThisStuffWrapper);

WAMR has docs on how to do this: https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/export_native_api.md

But for wasmtime, i can't figure out what the equivalent of that would be

Alex Crichton (Jul 23 2020 at 14:11):

FWIW those examples look extremely dangerous because they're not doing any validation of the arguments given from wasm

Alexandru Ene (Jul 23 2020 at 14:11):

WAMR does some, as it forces the size parameter on the pointers

Alex Crichton (Jul 23 2020 at 14:12):

I believe the wasmtime equivalent is the last example here -- https://bytecodealliance.github.io/wasmtime/api/wasmtime/struct.Func.html#method.wrap

Alex Crichton (Jul 23 2020 at 14:12):

the wamr/wasm3 examples dont' look like they're actually validating the input provided by wasm

Alex Crichton (Jul 23 2020 at 14:12):

or the length provided

Alex Crichton (Jul 23 2020 at 14:12):

to make sure it's actually in-bounds in the wasm linear memory

Alexandru Ene (Jul 23 2020 at 14:13):

I think WAMR does some checks as they mandate the size to follow that pointer, but didn't dive super deep into their actual implementation.

Alex Crichton (Jul 23 2020 at 14:13):

hm ok, well in any case this isn't functionality built into wasmtime

Alex Crichton (Jul 23 2020 at 14:13):

this is something you'll need to do in your Func closure

Yury Delendik (Jul 23 2020 at 14:24):

it is not what https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/export_native_api.md#buffer-address-conversion-and-boundary-check describes

Yury Delendik (Jul 23 2020 at 14:25):

"When passing a pointer address from WASM to native, the address value must be converted to native address before the native function can access it."

Yury Delendik (Jul 23 2020 at 14:29):

hard to imagine that going from native to WASM will be different

Alexandru Ene (Jul 23 2020 at 14:53):

Yes, so i want to pass exactly that, pass an address from WASM to native

Yury Delendik (Jul 23 2020 at 14:58):

do you have to call wasm_runtime_addr_native_to_app for that?

Alexandru Ene (Jul 23 2020 at 14:58):

What makes this difficult in WASMtime, is that the function wrappers don't take the engine as a parameter, so there is no easy way to query the WASM memory base pointer.
This makes it hard to send a char * (or any pointer to WASM memory) from WASM land into C++ land and access it's contents (as you'd need the base offset, to get to a memory address that works out of the box in C++)
As my second idea was to just send an i32/i64 (the WASM address - basically the wasm memory offset), and together with the base pointer I could re-create a C++ accessible pointer.

Yury Delendik (Jul 23 2020 at 15:00):

it is hard to judge what WAMR is doing without complete example. Do you have working one?

Alex Crichton (Jul 23 2020 at 15:01):

@Alexandru Ene the Caller type is how wasmtime supports getting the memory's base pointer right now, does that work for you?

Alex Crichton (Jul 23 2020 at 15:01):

(or wasmtime_caller_t in the C API)

Alexandru Ene (Jul 23 2020 at 16:01):

@Alex Crichton Yes, if that allows me to get the base memory, I can reconstruct a host-vm accessible pointer from the WASMTIME pointer (offset). So this seems to solve my use-case.

Alexandru Ene (Jul 23 2020 at 16:02):

I just need to figure out how exactly to set this up, but it sounds promissing. thanks

Alexandru Ene (Jul 23 2020 at 16:22):

This is pretty cool. Would be interesting to see how this will evolve, as an user the WAMR approach seems sensible, as now everyone has to do the magical translation between address spaces themselves (with the associated risks)
But it would be interesting to mandate a pointer + size like WAMR when passing VM pointers to HOST program

Alexandru Ene (Jul 23 2020 at 16:22):

Maybe the interface proposal is something to deal with this

Yury Delendik (Jul 23 2020 at 16:24):

notice that from "native" side wasm pointers are not really "pointer" but just offsets

Yury Delendik (Jul 23 2020 at 16:24):

so basically I32 for wasm32

Alexandru Ene (Jul 23 2020 at 16:25):

Yes, what would be interesting is to have these things more enforced (passing raw data over the boundary of VMs)
I am sure things like these are worked on (I'm quite new to WASM in general)
Basically started when I started writing stuff in here :grinning:

Alexandru Ene (Jul 23 2020 at 16:27):

But stuff like forcing users to pass a pointer + size always, like WAMR tries to enforce is interesting. Of course you can just pass the offset directly and re-create the actual memory address on the host VM, but it's just easy to do it the "safer" way

Yury Delendik (Jul 23 2020 at 16:27):

you can choose to do what WASI is doing atm

Alexandru Ene (Jul 23 2020 at 16:27):

What is that?

Yury Delendik (Jul 23 2020 at 16:28):

generate wrappers from witx?

Yury Delendik (Jul 23 2020 at 16:29):

but your do_stuff(const char*) signature may not be valid there

Alexandru Ene (Jul 23 2020 at 16:33):

that's interesting, I will look into it

Alexandru Ene (Jul 23 2020 at 16:34):

I just thought that WASI functions have basically a special module in WASM engines that exposes some methods just like the other host vm methods

Alexandru Ene (Jul 23 2020 at 16:34):

I didn't realize that's generated

Till Schneidereit (Jul 23 2020 at 16:59):

@Alexandru Ene this blog post might be helpful: https://radu-matei.com/blog/wasm-api-witx/

Alexandru Ene (Jul 23 2020 at 17:03):

Thanks :slight_smile:

Alexandru Ene (Jul 23 2020 at 20:34):

Oh gosh, so I went back into this after taking a break, so there's a wasmtime_func_callback_t and a wasm_func_callback_t and just the first one supports the caller parameter. I was so confused between these two for a while

Alexandru Ene (Jul 24 2020 at 01:12):

Just an update on this, it works as advised, with a small correction: using wasm_name_new_from_string() with WASMTIME is not a good idea.
The generated names from that function are adding a '\0' to the name, making the rust str& be ['e', 'n', 'v', '\0'].
However, the imports from a WASM module are a str& of just "env", without a trailing '\0'.

Not sure where the change should be made so this interops nicer (maybe wasmtime can ignore trailing \0s for names?

Till Schneidereit (Jul 26 2020 at 11:28):

@Alexandru Ene would you mind filing an issue about this? It's possible that this is all by design, but in that case there might still be some documentation that could be improved. Or it's not working as intended and we should indeed fix it :slight_smile:

Alexandru Ene (Jul 26 2020 at 11:48):

It is correct I think, for C++ it makes sense to capture and copy the terminating '\0'. But rust's str& doesn't care about terminating zeros. It's most likely something that can be handled in the wasm_name_t rust wasmtime handling

Peter Huene (Jul 27 2020 at 00:34):

wasm_name_new_from_string is an inline function in the wasm.h header that specifically copies the null when calling wasm_byte_vec_new, so there's not much we can do in Wasmtime. I'd simply just use wasm_byte_vec_new directly with only the length of the string if the null was undesirable.

Peter Huene (Jul 27 2020 at 00:37):

i don't quite get the rationale of wasm_name_new_from_string copying the null; the only place in wasm.h that anything is null terminated is wasm_message_t used in traps.

Peter Huene (Jul 27 2020 at 00:43):

there's also no way for Wasmtime to distinguish a wasm_name_t from a wasm_byte_vec_t as the header defines the wasm_name_t functions in terms of the byte vec functions; that is, of course, assuming wasm_name_t is supposed to be null terminated as there's zero documentation of that.

Peter Huene (Jul 27 2020 at 00:44):

i'd argue this line: typedef wasm_name_t wasm_message_t; // null terminated indicating wasm_message_t being null terminated as an indication of wasm_name_t not supposed to be null terminated.

Peter Huene (Jul 27 2020 at 00:45):

at any rate, I'd first log an issue against the webassembly/c-api repo for some clarity around this since there aren't any actual wasm_name_t-related functions we can implement to distinguish names from byte vecs. I mean, we could check for null on every function taking a wasm_name_t and trim the underlying slice before handing it to the Rust API, but that seems unnecessary if wasm_name_t isn't supposed to be null terminated (undoc'd APIs = :shrug:)

Peter Huene (Jul 27 2020 at 01:01):

if i had to guess, I'd say the null being copied in wasm_name_new_from_string to be an upstream bug in wasm.h.

Peter Huene (Jul 27 2020 at 01:03):

It looks like it is intentional (https://github.com/WebAssembly/wasm-c-api/pull/34) but the way this function is used in this PR it should be called wasm_message_new_from_string as it's only used to create a trap message :shrug:

Alexandru Ene (Jul 27 2020 at 09:42):

I opened this: https://github.com/WebAssembly/wasm-c-api/issues/149

Last updated: Dec 23 2024 at 12:05 UTC