Extracting exports for a structure generated with Wiggle · general

I'm exploring how to write a custom API layer in WITX, then link the module at runtime when instantiating with Wasmtime.
The first part is trivial using Wiggle:

wiggle::from_witx!({
    witx: ["custom_witx.witx"],
    ctx: MyCtx,
});
...
}

However, I haven't been able to find how to extract the exports and link them when instantiating a module:

    let imports = module
        .imports()
        .map(|i| {
            let export = match i.module() {
                "wasi_snapshot_preview1" => wasi_snapshot_preview1.get_export(i.name()),
                "wasi_unstable" => wasi_unstable.get_export(i.name()),

                "custom_witx" => panic!("get exports"),

            };

I started down the path of exploring how this is done in WASI (the define_struct function in wig), but before I go too deep, I want to make sure there isn't an easier way of doing this (i.e reusing it / parts of it, rather than re-implementing it for every custom module).

Jakub Konka (May 18 2020 at 10:31):

Hey @Radu M, if I understood your needs right, you essentially want the VM to export your own module at runtime, right? In that case, you'll need to call at least some bits of wig::wasi_define_struct_for_wiggle. Just to quickly bring you up to speed, wig::wasi_define_struct_for_wiggle is called from wasmtime_wasi crate to load all syscalls from wasi_common, generate appropriate C shims, and export them to any linking WASI module at runtime. So to be able to link your own module at runtime, you'll need at the very least run wig::wasi_define_struct_for_wiggle in wasmtime_wasi crate to register your syscalls with the VM. It might actually be a good exercise in extracting some common bits from wig that could be re-used for different modules that we'd like to link with that are not necessarily wasi-common specific. Anyhow, I'm happy to help you out here if you get blocked by anything.

Radu Matei (May 21 2020 at 20:17):

I think this _could_ be also achieved with some changes to the Linker structure, if it accepted a set of existing imports.
(it would not be as seamless as the WASI imports, because you would still have to manually map the function names, but it could be a slightly easier start, if I am not missing something).

Radu Matei (May 22 2020 at 20:29):

I ended up taking this approach, and now I link all all WASI exports using a Linker. This seems to work fine in the limited tests I've done so far, and it's a bit more straightforward, in that you can now use the same method to link both WASI and your custom modules / Rust functions.

Radu Matei (May 22 2020 at 20:37):

I did have to add a new public method to the linker though, similar to Linker::func, but accepting an Extern directly:

diff --git a/crates/wasmtime/src/linker.rs b/crates/wasmtime/src/linker.rs
index 888ad1cff..dcfdf79ea 100644
--- a/crates/wasmtime/src/linker.rs
+++ b/crates/wasmtime/src/linker.rs
@@ -215,6 +215,11 @@ impl Linker {
         self._define(module, name, Func::wrap(&self.store, func).into())
     }

+    /// Links an external item
+    pub fn add_extern(&mut self, module: &str, name: &str, item: Extern) -> Result<&mut Self> {
+        self.define(module, name, item)
+    }

I'd be happy to chat to better understand if the Linker is indeed supposed to be able to do this, or I am abusing its role in this way.

Dan Gohman (May 22 2020 at 20:41):

Dan Gohman (May 22 2020 at 20:42):

Radu Matei (May 22 2020 at 20:49):

Ah, there is _define, which is private, and define which is public - you're right, I can use it directly.

That being said, do you see any issues with instantiating the modules using the linker?

Dan Gohman (May 22 2020 at 21:01):

Radu Matei (May 22 2020 at 21:49):

The idea is to instantiate a module and provide the imports for both WASI and custom modules / in the same way.
Ultimately, I'd like to express an API layer as WITX and link that, but I'm not there yet.

Stream: general

Topic: Extracting exports for a structure generated with Wiggle

Radu Matei (May 18 2020 at 07:26):