Stream: general

Topic: code generation from wit

Jens Gersdorf (Oct 26 2023 at 09:54):

Hi,

I have been looking into WebAssembly, the new component model and WIT this week for the first time. It's all a little bit overwhelming, and I am always trying to figure out how all the things work together.

Today I had a look into the brand new hello-wasi-http. Works fine, concept is clear. But what I found a little bit annoying when trying to figure things out, understanding the used types, is the code generation. As I always value feedbacks from newbies in my projects, I thought I just share my view.

The magic cargo_component_bindings::generate!() macro generates the code from the wit files (here the wasi http types). I can expand the macro and see the code, but rust analyzer support is limited. If I want to go to the definition of let's say the IncomingRequest, I just jump to the line with the macro. There is as far as I know no possiblitiy to jump to e.g. the generated type. So I keep searching manually in the expanded macro code for the things I am interested in.

I can still rely on code suggestions to see what the types offer for methods and fields, which is all nice and fine. But all generated code is missing documentation. The wit files are currently only using the // comments, but even if I add documentation to the wit file using ///, the macro generation seems not to use them to generate documentation style comments on the generated code.

The alternative for code generation using a proc-macro would be to generate the code up front with a build.rs. For the usecase "generation rust code from some wit files", I would have opted for this solution, as it makes the developer experience IMHO a little bit nicer. Idealy with a lot of documentation being taken over from the wit file.

For the standardized wasi interfaces I wonder if it also makes sense to offer e.g. for wasi-http a crate which contains the rust-code generated from the wit files. Currently, the hello-wasi-http project has a dependency to cargo-component-bindings only to be able to generate the rust code from wit. It a crate with this code would exist, the need for code generation would cease to exist.

Are there any plans of going into this direction?

Anyway: Thanks for the great work, I am looking forward to see WebAssembly and wasmtime evolving in the future!

Jens Gersdorf (Oct 26 2023 at 12:15):

Ok, stupid me, I can also just use wit-bindgen just as I wanted to do.

Jens Gersdorf (Oct 26 2023 at 12:18):

Interesting enough, the code generated by wit-bindgen also includes the documentation I added to the wit file. Wonder why the macro generated code did leave that out.

Jens Gersdorf (Oct 26 2023 at 12:28):

That leaves as only question: Will there be a crate that contains the generated code from wasm-bindgen for a specific wasi package? I found that there is a plaeholder crate wasi-http.

Alex Crichton (Oct 26 2023 at 14:16):

cc @Peter Huene as you're likely interested in this feedback

One possible idea I had reading this, could cargo-component generate a file into the OUT_DIR of a Cargo project and then the macro expands to #[path = "..."] mod bindings or include!("....")? That might assist with jump-to-definition with an IDE since there's an actual file on the filesystem with the expanded contents.

Are there any plans of going into this direction?

I think this makes sense to do eventually yeah! For example the wasi crate contains bindings for wasi_snapshot_preview1. Once things are more stable I think it would make sense to update the crate.

Wonder why the macro generated code did leave that out.

This might be a "simple bug" in cargo component, as it should use wit-bindgen under the hood so they should both generate docs

Peter Huene (Oct 26 2023 at 15:39):

One possible idea I had reading this, could cargo-component generate a file into the OUT_DIR of a Cargo project and then the macro expands to #[path = "..."] mod bindings or include!("....")? That might assist with jump-to-definition with an IDE since there's an actual file on the filesystem with the expanded contents

This sounds doable and might be a proper fix for the rust-analyzer issue.

We'll need to move the macro arguments into Cargo.toml, but that's easy enough to do.

This might be a "simple bug" in cargo component, as it should use wit-bindgen under the hood so they should both generate docs

They definitely should be generated (assuming they're being round-tripped though the binary encoding), but with the rust-analyzer issue where changes to the bindings wasm file don't cause a re-expansion of the macro, changes to the WIT aren't reflected until the rust-analyzer server restarts.

I can look into this.

Alex Crichton (Oct 26 2023 at 17:51):

We'll need to move the macro arguments into Cargo.toml, but that's easy enough to do.

Oh my thinking was that the macro would generate a file when invoked and expand to include! or #[path] instead of expanding to the module directly

Alex Crichton (Oct 26 2023 at 17:51):

so no changes needed to the options/invocation site, only to the tail end of the macro invocation itself

Alex Crichton (Oct 26 2023 at 17:51):

er, tail end of the implementation that is

Peter Huene (Oct 26 2023 at 18:06):

but the arguments are required to generate the file, no?

Peter Huene (Oct 26 2023 at 18:07):

such as what types implement the target world and resources

Peter Huene (Oct 26 2023 at 18:07):

which cargo-component picks some defaults for, thus generally folks don't have arguments to the macro

Peter Huene (Oct 26 2023 at 18:08):

but if we're going to generate the code up-front to include!, then we'll need to know those things when cargo-component does the generating of the file

Peter Huene (Oct 26 2023 at 18:09):

oh i misread, the macro generates the file, hmm

Peter Huene (Oct 26 2023 at 18:10):

might as well have cargo-component do the generation, though, as then we don't have to round-trip the target Resolve to a wasm

Peter Huene (Oct 26 2023 at 18:10):

which is the crux of the rust-analyzer problem as they do nothing with include_bytes! for dependency checks

Peter Huene (Oct 26 2023 at 18:13):

either way, i'll see what I can come up with today

Alex Crichton (Oct 26 2023 at 18:13):

ah ok yeah if it's good to optimize anyway I think that'd work well to

Alex Crichton (Oct 26 2023 at 18:14):

sounds like you understand what I'm saying though so I'll leave it to your judgement how best to incorporate into cargo-component

Jeff Parsons (Oct 26 2023 at 23:05):

This approach (codegen + include!) sounds like it would make cargo-component a lot more friendly in practice. (I've recently been considering switching away from cargo-component to wit-bindgen + my own shell scripts for many of the same reasons as the OP. It would be a weird outcome if that was the "less hassle" path.)

Peter Huene (Oct 26 2023 at 23:43):

yeah, should hopefully drastically cut down the compilation time as well (back to what it used to be when it generated a bindings crate)

Peter Huene (Oct 26 2023 at 23:43):

i should hopefully have a PR up tomorrow

Peter Huene (Oct 26 2023 at 23:44):

have to prove that the approach will solve the rust-analyzer problem first, though

Peter Huene (Oct 27 2023 at 00:24):

i've verified that include! of the pre-generated bindings source does fix rust-analyzer

Peter Huene (Oct 27 2023 at 00:40):

to summarize: rust-analyzer now works (along with go-to-source), comments work, and dropped from 50 deps to 13 (cut most of wit-bindgen deps) for the build of a new component

Peter Huene (Oct 27 2023 at 00:40):

so pre-generating is the way to go

Victor Adossi (Oct 27 2023 at 00:42):

Seems like along the way a fix for https://github.com/bytecodealliance/cargo-component/issues/139 as well?

Peter Huene (Oct 27 2023 at 00:45):

yep, that's the rust-analyzer issue

Peter Huene (Oct 27 2023 at 16:03):

PR is here: https://github.com/bytecodealliance/cargo-component/pull/167

Peter Huene (Oct 27 2023 at 21:44):

I'm about to release 0.4.0 of cargo-component with the bindings changes, i'll provide upgrade instructions in the release notes

Peter Huene (Oct 27 2023 at 22:12):

crates are in the process of publishing: https://github.com/bytecodealliance/cargo-component/releases/tag/v0.4.0

Last updated: Oct 23 2024 at 20:03 UTC