Till Schneidereit (Feb 03 2020 at 13:31): Till Schneidereit (Feb 03 2020 at 13:31):I just saw that the test logs contain lots of mentions of assert_malformed and assert_invalid with unexpected results, but tests marked as passing. See e.g. this log: https://github.com/bytecodealliance/wasmtime/runs/421975382?check_suite_focus=true#step:12:515
Is this something where our test harness just prints things that aren't important, or something where we actually have wrong results, but the harness somehow swallows them?
cc @Nick Fitzgerald, because I see a lot of multi-value in there.
fitzgen (he/him) (Feb 03 2020 at 18:44):My understanding was that these are tests that contain Wasm that's expected to fail (eg assert_malformed is asserting that the given Wasm is malformed, etc) and that we log the errors. We can probably stop logging them, since they're expected. The one complication, IIRC, is that the spec testsuite has specific error messages it is testing for, and we don't always match the message exactly. I'll take a peek again to confirm this.
cc @Yury Delendik -- I think you originally wrote the spec testsuite integration?
Till Schneidereit (Feb 03 2020 at 18:46):I might misunderstand the logs, but it seems to me like the tests expect a specific error, but we're getting a different one
fitzgen (he/him) (Feb 03 2020 at 18:48):
assert_invalid: expected type mismatch, got stack item type does not match block item type (at offset 31)
this one, for example, looks like a disagreement for what the error message should be
fitzgen (he/him) (Feb 03 2020 at 18:48):but there certainly could be others that are not so seemingly benign
fitzgen (he/him) (Feb 03 2020 at 19:05):Looks like we have https://github.com/bytecodealliance/wasmtime/issues/492 on file
Till Schneidereit (Feb 03 2020 at 19:07):ah, good, I hadn't seen that
Last updated: Oct 23 2024 at 20:03 UTC