Wasmtime GitHub notifications bot (Dec 20 2024 at 16:30): Wasmtime GitHub notifications bot (Dec 20 2024 at 16:30):alexcrichton opened issue #9881:
In yesterday's Wasmtime meeting we had a discussion about testing, CI, and the artifacts that we produce. Currently the actual released artifacts are not actually tested in CI at all, they're simply produced. This isn't a great situation and ideally we would test the exact artifacts that CI produces rather than a different build configuration.
There's a few challenges to tackle when handling this though:
- Most of the test suite is not architected to run the
wasmtimeCLI or thelibwasmtime.soC API. Some oftests/all/cli_tests.rsruns thewasmtimeexecutable, but the executable is produced as part ofcargo testand isn't a released artifact. The C API is only tested through the C examples (so very lightly).- Testing artifacts might require a different command to execute or new infrastructure other than
cargo test. It's unclear what this infrastructure would look like.- Testing "all possible configurations" isn't feasible, even with the generous concurrency that we already have. For example we can't test the matrix of {platform, opt/debug, debug-asserts/no, crate} in CI as that would be too much.
We didn't reach many firm conclusions in the meeting but I wanted to open a follow-up issue to track further progress. A few points we did identify were:
- We should probably have one CI job test with
--releaseto test with optimizations (just one platform though, probably not all of them)- We should ideally run at least a "smoke test" for released artifacts.
cc @tschneidereit
Last updated: Dec 23 2024 at 12:05 UTC