Stream: wit-bindgen

Topic: c/c++ host generator?

Gordon Smith (Apr 18 2023 at 07:16):

I am going through the steps here: https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/export_native_api.md to create a c++ native host which exposes functions to the wasm.
Is there a c/c++ generator similar to JCO?
If not does anyone know if someone is already planning / discussing it?
If not and someone was inclined to create one, is there an official antlr grammar?
Or would it be better to extend something like the JCO or the rust bindgen macro?

Christof Petig (May 02 2023 at 21:47):

I think that the engine independent C API https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/wasm_c_api.md would be a better target for a C/++ host code generator.
Also as we plan to work with WIT on adaptive AUTOSAR, a higher-level C++ binding (leveraging expected, optional and variant) would be something I could cooperate on.

Till Schneidereit (May 03 2023 at 07:44):

We've unfortunately found the engine-independent C/C++ API not to be a good fit for many use cases of Wasmtime, and have decided to limit our support for it to somewhat basic functionality. That means that we'll not bring Components support to it (which would entail substantially expanding the API in any case.)

In addition to the severe limitations listed in the FYI section of the document you linked to, you can read more about the reasons here and here

Gordon Smith (May 03 2023 at 08:30):

Till - Thanks for this insight! Should I be focusing on https://github.com/bytecodealliance/wasmtime-cpp? My other question still stands, does anyone know if there is any work being done on generating a c/c++ host bindings from WIT files?

Alex Crichton (May 03 2023 at 14:24):

I'm at least not personally aware of any C/C++ host generator for WIT at this time. Not because it can't be done, though, just because it's work and that takes some time. Whether or not the C API itself or the C++ is the right way to go is somewhat up to you. I would ideally recommend Rust as embeddings of WebAssembly probably aren't dealing with trusted code and having as much safety as you can probably helps, but otherwise I'd recommend going with C++ to handle most of the resource management, such as strings into components, for you.

Ralph (May 03 2023 at 18:30):

we're going to need one. That doesn't mean the team here needs to provide it.

Gordon Smith (May 04 2023 at 09:07):

This is something which I wouldn't mind taking a stab at and have some experience with ANTLR, code generators and associated IDE support. Has anyone done an ANTLR grammar yet?

Christof Petig (May 04 2023 at 11:10):

Oh. @Till Schneidereit thank you for this interesting material. You can clearly see at https://github.com/bytecodealliance/wasm-micro-runtime/pull/2173 that there are problems with wamr as well...

Alex Crichton (May 04 2023 at 14:28):

Ralph said:

we're going to need one. That doesn't mean the team here needs to provide it.

Oh of course! And contributions are of course always welcome, and I think it's natural and inevitable to have wit-bindgen-generated-bindings for many languages, including C/C++!

Gordon Smith said:

Has anyone done an ANTLR grammar yet?

Not currently, no. I'll say though that parsing WIT is the easy part, the code generation part is much more involved (in my personal opinion at least). Many code generators are currently written in Rust because all the WIT tooling is written in Rust right now. For example the wasmtime-py host generator is written in Rust and then compiled to WebAssembly to get run in Python.

Gordon Smith (May 05 2023 at 05:17):

Alex Crichton said:

Not currently, no. I'll say though that parsing WIT is the easy part, the code generation part is much more involved (in my personal opinion at least). Many code generators are currently written in Rust because all the WIT tooling is written in Rust right now. For example the wasmtime-py host generator is written in Rust and then compiled to WebAssembly to get run in Python.

Shame ANTLR doesn't have target support for Rust, but it would be a nice single point of truth WRT the grammar. Also it makes life a lot easier when creating support for things like VSCode extensions...

Unfortunately, I haven't had time learn Rust, but wonder if a practical approach would be to "simply" generate the Rust host bindings and then expose them to c via the techniques outlined here:
https://docs.rust-embedded.org/book/interoperability/rust-with-c.html?

Alex Crichton (May 05 2023 at 17:28):

My hunch is that the "simply" would not be so simple there. The devil is in the details of memory management and understanding structures on the C/C++ side of things, so that would all still need to be taken care of

Christof Petig (May 06 2023 at 13:31):

Gordon Smith said:

Unfortunately, I haven't had time learn Rust, but wonder if a practical approach would be to "simply" generate the Rust host bindings and then expose them to c via the techniques outlined here:
https://docs.rust-embedded.org/book/interoperability/rust-with-c.html?

Given that you started this thread with a link to the wamr specific host interface document, I feel that wrapping the Rust host binding for wasmtime in C++ won't help you that much. :thinking:
I started a fork to create unified C API host targeted bindings at https://github.com/cpetig/wit-bindgen and hope to get something simple ready during this weekend. This should work fine for host directed calls even with threads, but for calls into wasm some infrastructure argument will be necessary to use thread specific instances of the runtime. [We had our share of thread issues calling into wasmer on veloren, see #plugins for more details.]

Christof Petig (May 07 2023 at 22:24):

I ended up at the, probably expected, dead-end of trying to access the correct store from a callback in a portable way.
While I was able to get wasm calls and memory access working with multiple threads on wamr, and I was able to access the store from wasmer (FunctionEnvMut), you can't portably create a mere trap inside the callback. I see that wasmtime has provided its own solution to work around this (wasmtime_trap_new).

Christof Petig (May 08 2023 at 19:18):

I now have a fully working prototype of a multi-threaded guest program which calls host functions, some of which allocate memory via cabi_realloc - using the wamr native API. The code has become much more readable compared to the C API and can work without global variables. I will now continue work on my wit-bindgen backend for this environment.

Gordon Smith (Mar 03 2025 at 19:48):

@Christof Petig - after many iterations I have a C++ host ABI implementation for lowering and lifting of the data types.

The main highlight is that all the typing (including the calculation of the flat_types array) are calculated at compile time.

Example from https://github.com/GordonSmith/component-model-cpp/blob/trunk/test/main.cpp

    struct PersonStruct
    {
        string_t name;
        uint16_t age;
        uint32_t weight;
    };
    using Person = record_t<PersonStruct>;

    struct PersonEx3Struct
    {
        string_t name;
        uint16_t age;
        uint32_t weight;
        string_t address;
        list_t<string_t> phones;
        tuple_t<uint16_t, uint32_t> vital_stats;
        Person p;
    };
    using PersonEx3 = record_t<PersonEx3Struct>;
    PersonEx3 pex3_in = {"John", 42, 200, "123 Main St.", {"555-1212", "555-1234"}, {42, 43}, {"Jane", 43, 150}};
    v = lower_flat(*cx, pex3_in);
    ...call function get response etc. ...
    auto pex3_out = lift_flat<PersonEx3>(*cx, v);

Mats Taraldsvik (Mar 04 2025 at 22:23):

@Gordon Smith I'm eagerly awaiting support for running/embedding wasm+component model in a c++ application/host. Is this it? :)

Gordon Smith (Mar 05 2025 at 07:00):

Its only part of the picture, yes it could be used to simplifying a wit -> c++ generator (IMO it should make that exercise easier). This is very much a side project, so I wouldn't hold my breath waiting for a handsfree solution.

Mats Taraldsvik (Mar 05 2025 at 13:17):

@Gordon Smith Ah, ok :) I guess it's still early days wrt host embedding (with host functions)+component model in C++.

Could you please elaborate on "the whole picture", i.e. which parts are needed to accomplish C++ host-embedding in wasmtime or any wasm-runtime? (I assume this: https://github.com/bytecodealliance/wasmtime/issues/8036#issuecomment-2593846815 is a crucial part, perhaps overlapping with your work?)

Gordon Smith (Mar 05 2025 at 14:04):

Needed bits:

• More ABI support (calling functions + various semantics, streaming, async etc.). FWIW I plan on going as far as function calling before investigating the other parts (this ABI support is agnostic to the wasm runtime being used).
• WIT -> Host C++. Parsing the WIT and generating the C++ data structures and function declarations (my hope is that targeting the cmcpp ABI implementation should make this step simpler).
• WebAssembly runtime adapters - this is probably just sample code, but would show how to embed a webassembly runtime and create an ABI compatible context.
• Tooling helpers - ability to extract a components wit and merge components etc. This exists already in many forms, but folks seem to expect a cli that provides all the above in one place, specific for their host language.

Mats Taraldsvik (Mar 06 2025 at 21:50):

Thanks for the explanation. It does look like a large or even massive undertaking. I'm just a regular C++-dev who want to enable wasm component-model plugins in a c++-application. @Gordon Smith

I don't understand the relationship between (c-api) component model host support in wasmtime and your project.

I guess I'm trying to work out which project I should be waiting for, and perhaps even try to contribute to if I'm able.

Last updated: Apr 09 2025 at 05:03 UTC