Stream: rust-toolchain

Topic: Correct way to run tests in wasm

Alexandru Ene (Aug 11 2020 at 18:25):

I was thinking a nice thing I could do with WASMTime is to have rust (rust compiled to WASM) [test] tests for testing various bindings and interaction with the host program (by calling exposed functions, etc.).

It all works well if tests pass, however on panic! or a failed assert! the wasm program just stops execution. This is unlike panic! in a normal program where if uncaught you get the panic message still.

What's the correct way to run rust tests that are compiled to WASM? Does this even make sense?

Alexandru Ene (Aug 11 2020 at 18:26):

There are various advantages to doing this (having rust tests in WASM), such as getting a test executor out of the box, but the drawbacks seem a bit big and I don't know if anyone has tried this or was able to deal with these things.

Alexandru Ene (Aug 11 2020 at 18:33):

It exits with "Failed to invoke function" error, but I don't know what function that is yet. This was the other VM, wasm3 that reports that, wasmtime just silently exits with no error returned.

Yury Delendik (Aug 11 2020 at 18:35):

is it a bin (not lib) for "wasm32-wasi" target?

Yury Delendik (Aug 11 2020 at 18:37):

/me checks if cargo build --target wasm32-wasi --test ... works

Alexandru Ene (Aug 11 2020 at 18:37):

So the wasm code is build like this: cargo build --release --target wasm32-wasi --tests

Alexandru Ene (Aug 11 2020 at 18:37):

But whenever a test fails (try panic! or assert!(false) inside of one, the wasmtime VM just exits and doesn't continue executing remaining tests.

Alexandru Ene (Aug 11 2020 at 18:38):

On a successful run:

running 2 tests
test always_fail_test ... ok
test  good_test ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Alexandru Ene (Aug 11 2020 at 18:39):

the always_fail_test in this case does nothing, however if i have panic!("asdf"); in it, the output is:

running 2 tests
test always_fail_test ...

Alexandru Ene (Aug 11 2020 at 18:40):

Then it quits the VM, returning from the invoked function, not printing it failed and not executing the other test.

Yury Delendik (Aug 11 2020 at 18:56):

there are many restrictions on what "wasm32" target can do, e.g. as listed in https://github.com/rust-lang/rust/blob/f3a9de9b08659e20ce7c282ed77bc43ddd149107/library/test/src/lib.rs -- I bet it is threads support?

Alexandru Ene (Aug 11 2020 at 18:57):

Aren't tests from a module effectively running on a single threead?

Alexandru Ene (Aug 11 2020 at 18:57):

They pass ok if no test asserts/fails

Yury Delendik (Aug 11 2020 at 18:58):

in wasm or native?

Alexandru Ene (Aug 11 2020 at 18:58):

In native (and wasm)

Alexandru Ene (Aug 11 2020 at 18:58):

I guess that should make them fail even if no test fails (if it's threads)

Alexandru Ene (Aug 11 2020 at 18:58):

But you do bring a good point, I guess nobody tried it

Yury Delendik (Aug 11 2020 at 18:58):

"By default, all tests are run in parallel."

Alexandru Ene (Aug 11 2020 at 18:59):

Hmm, but they do work just fine if tests pass (i see the output, etc,) As far as I know the current std thread impl panics at initialization. Maybe that mechanism kicks in after one of them fails/takes too long? But that's still suspicious.

Yury Delendik (Aug 11 2020 at 19:00):

in wasm, if thing goes unreachable the thread is effectively dead

Yury Delendik (Aug 11 2020 at 19:01):

if process single threaded -- it is dead too

Alexandru Ene (Aug 11 2020 at 19:02):

I see, the thing that confuses me is how i am able to get the successful output from WASM, as I'd expect it to reach unreachable regardless of test results.

Yury Delendik (Aug 11 2020 at 19:02):

/me thinks term "poisoned" fits here better

Alexandru Ene (Aug 11 2020 at 19:08):

I could run it with test_threads=1? Not sure if this is different from what concurrency returns in the context of a WASM VM

Yury Delendik (Aug 11 2020 at 19:09):

does it match when --test_threads=1 used on native test runs?

Alexandru Ene (Aug 11 2020 at 19:10):

It seems so, I didn't run this rust program on a native test run as it depends on methods that are only available to it through VM bindings, so I didn't test it in any other way than a WASM binary

Alexandru Ene (Aug 11 2020 at 19:21):

Maybe what I need is: https://github.com/japaric/utest

Alexandru Ene (Aug 11 2020 at 19:31):

But to ask another way, does anyone have a way of running unit [tests] in WASM?

Pat Hickey (Aug 11 2020 at 19:56):

we (fastly folks) do not

Pat Hickey (Aug 11 2020 at 19:57):

its a significant gap that we have not figured out a detailed plan to address.

Alexandru Ene (Aug 11 2020 at 20:03):

Just to follow-up on this, I generated 1000 #[test] methods that pass, and i can run them successfully with wasmtime, so i am starting to believe it may not be thread related, or if it is thread-related this is triggered once an assert/panic occurs

Alexandru Ene (Aug 11 2020 at 20:04):

One not so ideal way of doing it in a CI environment is to just search for test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out in the output after the test module is executed, if that's missing, mark it as failed and debugging starts (not ideal at all)

Alexandru Ene (Aug 11 2020 at 20:05):

Final lines of my experiment:

test good_test995 ... ok
test good_test996 ... ok
test good_test997 ... ok
test good_test998 ... ok
test good_test999 ... ok

test result: ok. 1000 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Result: 0

Alexandru Ene (Aug 11 2020 at 20:07):

It's kind of all-or-nothing, but on the bright-side we can execute them.

@Pat Hickey if you want to team-up for a more proper solution to this we would likely want to colaborate on a solution, as it's quite annoying. Building our own test runner seems an easy way out, just as utest did, but with less knobs would be great.

Pat Hickey (Aug 18 2020 at 19:58):

hey, sorry, i never got back to this. @fitzgen (he/him) says that tests "just work" if you use cargo wasi

Pat Hickey (Aug 18 2020 at 19:58):

i was not aware of this!

Alexandru Ene (Aug 18 2020 at 21:28):

I didn't know about cargo wasi. But how would they work if in the tests i'm caling stuff that I exposed throught the WasmtimeVM from my host program. Something like AlexsHostProgramDoStuff(.....) and this exists only there. From what I can tell cargo wasi just automagically launches a wasmtime standalone VM, but that doesn't have my extra exposed functions to the vm :smile:

Pat Hickey (Aug 18 2020 at 21:29):

yeah, there's no story for that yet afaik

Alexandru Ene (Aug 18 2020 at 21:29):

I guess they work if they are standalone, tho that's bringing up the question, what build settings does cargo wasi use for that wasmtime to not insta-bail on failure

Pat Hickey (Aug 18 2020 at 21:29):

I don't know what that would look like.

Alexandru Ene (Aug 18 2020 at 21:30):

Because then I can compile wasmtime the same way in my test setup, and have my exposed functions be there, and just run the tests generated by cargo build --test --target wasm32-wasi

Alexandru Ene (Aug 18 2020 at 21:31):

It seems to me that right now wasmtime could run tests, if compiled / booted-up properly (something I likely don't do).

Alexandru Ene (Aug 18 2020 at 21:32):

Unless cargo wasi does some extra stuff to patch the test runner before compilation (I didn't check this). Will do some more investigation. Thanks.

Last updated: Dec 23 2024 at 13:07 UTC