Stream: git-wasmtime

Topic: wasmtime / PR #8622 wasmtime: Introduce the `test-macros`...


view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 00:12):

saulecabrera opened PR #8622 from saulecabrera:add-test-macros to bytecodealliance:main:

This commit introduces the test-macros crate. The whole idea behind this crate is to export a single or multiple macros to make it easier to configure Wasmtime for integration tests. The main use-case at this time, is running a subset of the integration tests with Cranelift and Winch. This crate could be extended to serve other use-cases, like testing pooling allocator and/or GC configurations.

This commit introduces a single example of how this macro could be used. If there's agreement on merging this change in some shape or form, I'll follow up with migrating the current tests to use #[wasmtime_test] where applicable.

Part of what's implemented in this PR was discussed in Cranelift's meeting on April 24th, 2024, however there are several discussion points that are still "in the air", like for example, what's the best way to avoid the combinatorial explosion problem for the potential test matrix.

--

cc/ @alexcrichton @fitzgen I'm opening this a draft, but feel free to leave any feedback according the discussion in the Cranelift meeting. I'm a bit concerned because I don't really have a good answer to the combinatorial problem of the test matrix, at least initially I suspect this won't be an issue necessarily, but if this macro grows in functionality I think it might be a problem.

<!--
Please make sure you include the following information:

Our development process is documented in the Wasmtime book:
https://docs.wasmtime.dev/contributing-development-process.html

Please ensure all communication follows the code of conduct:
https://github.com/bytecodealliance/wasmtime/blob/main/CODE_OF_CONDUCT.md
-->

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 00:14):

saulecabrera updated PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 00:16):

saulecabrera edited PR #8622:

This commit introduces the test-macros crate. The whole idea behind this crate is to export a single or multiple macros to make it easier to configure Wasmtime for integration tests. The main use-case at this time, is running a subset of the integration tests with Cranelift and Winch. This crate could be extended to serve other use-cases, like testing pooling allocator and/or GC configurations.

This commit introduces a single example of how this macro could be used. If there's agreement on merging this change in some shape or form, I'll follow up with migrating the current tests to use #[wasmtime_test] where applicable.

Part of what's implemented in this PR was discussed in Cranelift's meeting on April 24th, 2024, however there are several discussion points that are still "in the air", like for example, what's the best way to avoid the combinatorial explosion problem for the potential test matrix.

--

cc/ @alexcrichton @fitzgen I'm opening this a draft, but feel free to leave any feedback according the discussion in the Cranelift meeting. I'm a bit concerned because I don't really have a good answer to the combinatorial problem of the test matrix, at least initially I suspect this won't be an issue necessarily, but if this macro grows in functionality I think it might be a problem.

<!--
Please make sure you include the following information:

Our development process is documented in the Wasmtime book:
https://docs.wasmtime.dev/contributing-development-process.html

Please ensure all communication follows the code of conduct:
https://github.com/bytecodealliance/wasmtime/blob/main/CODE_OF_CONDUCT.md
-->

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 10:36):

saulecabrera updated PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 10:38):

saulecabrera updated PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 10:39):

saulecabrera updated PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 11:52):

saulecabrera updated PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 12:22):

saulecabrera updated PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 15:18):

fitzgen commented on PR #8622:

Excited for this! Will take a look today

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 17:19):

alexcrichton submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 17:19):

alexcrichton submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 17:19):

alexcrichton created PR review comment:

I might recommend replacing strategies with something like config: TestConfig which internally has Vec<(String,Ident)> or similar to make this more easily growable in the future to multiple configuration options.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 17:19):

alexcrichton created PR review comment:

Mind also adding publish = false here? (may also require changes to publish.rs, I sort of forget)

Also, to handle the clippy issues on CI you can add:

[lints]
workspace = true

to inherit the defaults of the workspace which allows clippy lints by default.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 17:19):

alexcrichton created PR review comment:

One possible concern about ItemFn here is that in the limit the entire test crate will be tagged with #[wasmtime_test] and that means the entire crate is parsed through syn and emitted back out. Technically the only needed part here is the function signature.

I'm not sure if it's worth it, but this could try to be a bit clever where it parses syn::ItemFn manually up to the block and then keeps the block as an opaque TokenStream (since it's known that it's brace-delimited anyway).

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 17:19):

alexcrichton created PR review comment:

It might be good to replace this with parse_nested_meta perhaps?

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 18:08):

fitzgen submitted PR review:

Definitely going in a good direction!

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 18:08):

fitzgen submitted PR review:

Definitely going in a good direction!

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 18:08):

fitzgen created PR review comment:

This shouldn't be necessary because a crate in the workspace depends on this crate.

view this post on Zulip Wasmtime GitHub notifications bot (May 15 2024 at 18:08):

fitzgen created PR review comment:

Yep, this is exactly what I was imagining usage would look like :+1:

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera updated PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera has marked PR #8622 as ready for review.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera requested wasmtime-core-reviewers for a review on PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera requested pchickey for a review on PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera requested wasmtime-default-reviewers for a review on PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera created PR review comment:

Done, thanks!

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:54):

saulecabrera created PR review comment:

Done!

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:55):

saulecabrera submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:55):

saulecabrera created PR review comment:

Good point. Took me a bit to figure out, but I was able to make this idea work by introduce a partial function parser as per your suggestion. Let me know what you think!

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:55):

saulecabrera submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:55):

saulecabrera created PR review comment:

Done!

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:55):

saulecabrera submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:55):

saulecabrera created PR review comment:

Done, thanks!

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:56):

saulecabrera commented on PR #8622:

Thanks for the review Alex and Nick. I've addressed all the comments, I think this is ready for another round of reviews.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:56):

saulecabrera edited a comment on PR #8622:

Thanks for the reviews Alex and Nick. I've addressed all the comments, I think this is ready for another round of reviews.

view this post on Zulip Wasmtime GitHub notifications bot (May 24 2024 at 18:59):

saulecabrera edited PR review comment.

view this post on Zulip Wasmtime GitHub notifications bot (May 28 2024 at 14:42):

alexcrichton submitted PR review:

Looks great to me, thanks!

I'm going to go ahead and merge, but some other assorted thoughts (which are fine as follow-ups or to ignore entirely):

In any case it might make more sense to expand the current strategy implemented and see how it looks after?

view this post on Zulip Wasmtime GitHub notifications bot (May 28 2024 at 14:57):

alexcrichton merged PR #8622.

view this post on Zulip Wasmtime GitHub notifications bot (May 29 2024 at 13:44):

saulecabrera commented on PR #8622:

Those are good suggestions Alex, I like the idea of opting out instead of opting in, it has the potential to make the macro less verbose and easier to integrate. I'll start migrating the tests and see how things shape up and update the macro definition if needed.


Last updated: Dec 23 2024 at 12:05 UTC