Stream: general

Topic: Exporting void do_stuff(const char*) functions to wasmtime


view this post on Zulip Alexandru Ene (Jul 23 2020 at 13:03):

For various _reasons_ I recently needed to link some host functions to a few wasm VMs. WASMTime to me seems to be the outlier here on how bindings work. (my experience is between wamr, wasm3, wasmtime). The other two engines allow to some degree exposing a function that takes a const char* or any pointer to some data as long as there's a size also involved (wamr enforces this).

But, in wasmtime, I was looking at the arguments and found AnyRef (behind a flag - wasmtime_config_wasm_reference_types_set).
The rest of accepted values are i32/i64,f32,f64 and func_ref.

I couldn't find an example on how to correctly use anyref, and exporting a function as taking anyref as a const char * seems to trigger a link_error in linker.compute_imports. I almost wanted to give up and cast a pointer to i64 and pass that to the host vm, grab the memory from wasmtime and then re-create the memory location to that offset as a workaround, but that is semi-hard in the bindings as they don't depend on an engine context (like it's readily available wasm3/wamr) and just the params/results arrays. This makes for some really ugly solutions.

What is the correct way to expose a method that takes a const char* to wasmtime?
My question would be, is there an example that I could reuse for anyref?

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:04):

@Alexandru Ene hm I'm not quite sure I understand what the functionality is that you're asking for, can you elaborate on what this const char* is in wamr/wasm3? Perhaps there's some example code I could poke around?

view this post on Zulip Alexandru Ene (Jul 23 2020 at 14:10):

So for example you can expose a host VM function that's basically: void PrintMeThisStuff(const char* stuff) { printf("%s\n", stuff); }
In WASM3 you need to export it by wrapping it as (it's basically a static function with these params:

const void* PrintMeThisStuffWrapper(IM3Runtime runtime, uint64_t* _sp, void* _mem) {
   m3ApiGetArgMem(const char*, msg);
   PrintMeThisStuff(msg); //real method called here
}

Then you bind it like so:

m3_LinkRawFunction(io_module, i_moduleName, i_functionName, "v(*)", &PrintMeThisStuffWrapper);

WAMR has docs on how to do this: https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/export_native_api.md

But for wasmtime, i can't figure out what the equivalent of that would be

WebAssembly Micro Runtime (WAMR). Contribute to bytecodealliance/wasm-micro-runtime development by creating an account on GitHub.

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:11):

FWIW those examples look extremely dangerous because they're not doing any validation of the arguments given from wasm

view this post on Zulip Alexandru Ene (Jul 23 2020 at 14:11):

WAMR does some, as it forces the size parameter on the pointers

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:12):

I believe the wasmtime equivalent is the last example here -- https://bytecodealliance.github.io/wasmtime/api/wasmtime/struct.Func.html#method.wrap

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:12):

the wamr/wasm3 examples dont' look like they're actually validating the input provided by wasm

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:12):

or the length provided

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:12):

to make sure it's actually in-bounds in the wasm linear memory

view this post on Zulip Alexandru Ene (Jul 23 2020 at 14:13):

I think WAMR does some checks as they mandate the size to follow that pointer, but didn't dive super deep into their actual implementation.

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:13):

hm ok, well in any case this isn't functionality built into wasmtime

view this post on Zulip Alex Crichton (Jul 23 2020 at 14:13):

this is something you'll need to do in your Func closure

view this post on Zulip Yury Delendik (Jul 23 2020 at 14:24):

it is not what https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/export_native_api.md#buffer-address-conversion-and-boundary-check describes

WebAssembly Micro Runtime (WAMR). Contribute to bytecodealliance/wasm-micro-runtime development by creating an account on GitHub.

view this post on Zulip Yury Delendik (Jul 23 2020 at 14:25):

"When passing a pointer address from WASM to native, the address value must be converted to native address before the native function can access it."

view this post on Zulip Yury Delendik (Jul 23 2020 at 14:29):

hard to imagine that going from native to WASM will be different

view this post on Zulip Alexandru Ene (Jul 23 2020 at 14:53):

Yes, so i want to pass exactly that, pass an address from WASM to native

view this post on Zulip Yury Delendik (Jul 23 2020 at 14:58):

do you have to call wasm_runtime_addr_native_to_app for that?

view this post on Zulip Alexandru Ene (Jul 23 2020 at 14:58):

What makes this difficult in WASMtime, is that the function wrappers don't take the engine as a parameter, so there is no easy way to query the WASM memory base pointer.
This makes it hard to send a char * (or any pointer to WASM memory) from WASM land into C++ land and access it's contents (as you'd need the base offset, to get to a memory address that works out of the box in C++)
As my second idea was to just send an i32/i64 (the WASM address - basically the wasm memory offset), and together with the base pointer I could re-create a C++ accessible pointer.

view this post on Zulip Yury Delendik (Jul 23 2020 at 15:00):

it is hard to judge what WAMR is doing without complete example. Do you have working one?

view this post on Zulip Alex Crichton (Jul 23 2020 at 15:01):

@Alexandru Ene the Caller type is how wasmtime supports getting the memory's base pointer right now, does that work for you?

view this post on Zulip Alex Crichton (Jul 23 2020 at 15:01):

(or wasmtime_caller_t in the C API)

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:01):

@Alex Crichton Yes, if that allows me to get the base memory, I can reconstruct a host-vm accessible pointer from the WASMTIME pointer (offset). So this seems to solve my use-case.

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:02):

I just need to figure out how exactly to set this up, but it sounds promissing. thanks

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:22):

This is pretty cool. Would be interesting to see how this will evolve, as an user the WAMR approach seems sensible, as now everyone has to do the magical translation between address spaces themselves (with the associated risks)
But it would be interesting to mandate a pointer + size like WAMR when passing VM pointers to HOST program

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:22):

Maybe the interface proposal is something to deal with this

view this post on Zulip Yury Delendik (Jul 23 2020 at 16:24):

notice that from "native" side wasm pointers are not really "pointer" but just offsets

view this post on Zulip Yury Delendik (Jul 23 2020 at 16:24):

so basically I32 for wasm32

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:25):

Yes, what would be interesting is to have these things more enforced (passing raw data over the boundary of VMs)
I am sure things like these are worked on (I'm quite new to WASM in general)
Basically started when I started writing stuff in here :grinning:

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:27):

But stuff like forcing users to pass a pointer + size always, like WAMR tries to enforce is interesting. Of course you can just pass the offset directly and re-create the actual memory address on the host VM, but it's just easy to do it the "safer" way

view this post on Zulip Yury Delendik (Jul 23 2020 at 16:27):

you can choose to do what WASI is doing atm

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:27):

What is that?

view this post on Zulip Yury Delendik (Jul 23 2020 at 16:28):

generate wrappers from witx?

view this post on Zulip Yury Delendik (Jul 23 2020 at 16:29):

but your do_stuff(const char*) signature may not be valid there

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:33):

that's interesting, I will look into it

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:34):

I just thought that WASI functions have basically a special module in WASM engines that exposes some methods just like the other host vm methods

view this post on Zulip Alexandru Ene (Jul 23 2020 at 16:34):

I didn't realize that's generated

view this post on Zulip Till Schneidereit (Jul 23 2020 at 16:59):

@Alexandru Ene this blog post might be helpful: https://radu-matei.com/blog/wasm-api-witx/

In this short article we explore how to get started with WebAssembly interface types by defining a simple API layer, then implementing that using Wiggle and Wasmtime

view this post on Zulip Alexandru Ene (Jul 23 2020 at 17:03):

Thanks :slight_smile:

view this post on Zulip Alexandru Ene (Jul 23 2020 at 20:34):

Oh gosh, so I went back into this after taking a break, so there's a wasmtime_func_callback_t and a wasm_func_callback_t and just the first one supports the caller parameter. I was so confused between these two for a while

view this post on Zulip Alexandru Ene (Jul 24 2020 at 01:12):

Just an update on this, it works as advised, with a small correction: using wasm_name_new_from_string() with WASMTIME is not a good idea.
The generated names from that function are adding a '\0' to the name, making the rust str& be ['e', 'n', 'v', '\0'].
However, the imports from a WASM module are a str& of just "env", without a trailing '\0'.

Not sure where the change should be made so this interops nicer (maybe wasmtime can ignore trailing \0s for names?

view this post on Zulip Till Schneidereit (Jul 26 2020 at 11:28):

@Alexandru Ene would you mind filing an issue about this? It's possible that this is all by design, but in that case there might still be some documentation that could be improved. Or it's not working as intended and we should indeed fix it :slight_smile:

view this post on Zulip Alexandru Ene (Jul 26 2020 at 11:48):

It is correct I think, for C++ it makes sense to capture and copy the terminating '\0'. But rust's str& doesn't care about terminating zeros. It's most likely something that can be handled in the wasm_name_t rust wasmtime handling

view this post on Zulip Peter Huene (Jul 27 2020 at 00:34):

wasm_name_new_from_string is an inline function in the wasm.h header that specifically copies the null when calling wasm_byte_vec_new, so there's not much we can do in Wasmtime. I'd simply just use wasm_byte_vec_new directly with only the length of the string if the null was undesirable.

view this post on Zulip Peter Huene (Jul 27 2020 at 00:37):

i don't quite get the rationale of wasm_name_new_from_string copying the null; the only place in wasm.h that anything is null terminated is wasm_message_t used in traps.

view this post on Zulip Peter Huene (Jul 27 2020 at 00:43):

there's also no way for Wasmtime to distinguish a wasm_name_t from a wasm_byte_vec_t as the header defines the wasm_name_t functions in terms of the byte vec functions; that is, of course, assuming wasm_name_t is supposed to be null terminated as there's zero documentation of that.

view this post on Zulip Peter Huene (Jul 27 2020 at 00:44):

i'd argue this line: typedef wasm_name_t wasm_message_t; // null terminated indicating wasm_message_t being null terminated as an indication of wasm_name_t not supposed to be null terminated.

view this post on Zulip Peter Huene (Jul 27 2020 at 00:45):

at any rate, I'd first log an issue against the webassembly/c-api repo for some clarity around this since there aren't any actual wasm_name_t-related functions we can implement to distinguish names from byte vecs. I mean, we could check for null on every function taking a wasm_name_t and trim the underlying slice before handing it to the Rust API, but that seems unnecessary if wasm_name_t isn't supposed to be null terminated (undoc'd APIs = :shrug:)

view this post on Zulip Peter Huene (Jul 27 2020 at 01:01):

if i had to guess, I'd say the null being copied in wasm_name_new_from_string to be an upstream bug in wasm.h.

view this post on Zulip Peter Huene (Jul 27 2020 at 01:03):

It looks like it is intentional (https://github.com/WebAssembly/wasm-c-api/pull/34) but the way this function is used in this PR it should be called wasm_message_new_from_string as it's only used to create a trap message :shrug:

Introduce a reference type wasm::Trap / wasm_trap_t to represent traps (backed by a JS Error instance -- enables them to maintain stack traces etc). Change Result / wasm_result_t to carry a trap in...

view this post on Zulip Alexandru Ene (Jul 27 2020 at 09:42):

I opened this: https://github.com/WebAssembly/wasm-c-api/issues/149

Hello, I stumbled into this problem when using the C-API with WASMTIME. wasmname_t can be created with wasm_name_new_from_string. This function is implemented like so: static inline void wasm_name_...

Last updated: Nov 22 2024 at 17:03 UTC