announcing `wasm-smith`, a WebAssembly test case generator · wasm

Stream: wasm

Topic: announcing `wasm-smith`, a WebAssembly test case generator

fitzgen (he/him) (Aug 21 2020 at 22:44):

The README has the sales pitch. I've already used this to find a bunch of bugs in wasmparser. Isn't published on crates.io yet, because I need to upstream some changes to the arbitrary crate first.

cc @Yury Delendik @Alex Crichton @Jonathan Foote

fitzgen/wasm-smith

A WebAssembly test case generator. Contribute to fitzgen/wasm-smith development by creating an account on GitHub.

Alex Crichton (Aug 22 2020 at 00:06):

Nice! Definitely looking forward to seeing this hooked up to wasmtime

Alex Crichton (Aug 22 2020 at 00:20):

@fitzgen (he/him) this might be good to include in the wasm tools repo?

fitzgen (he/him) (Aug 22 2020 at 01:25):

Perhaps!

fitzgen (he/him) (Aug 22 2020 at 20:39):

Now up on crates.io: https://crates.io/crates/wasm-smith

fitzgen (he/him) (Aug 24 2020 at 15:38):

blog post describing its implementation, if anyone is interested: https://fitzgeraldnick.com/2020/08/24/writing-a-test-case-generator.html

Alex Crichton (Aug 26 2020 at 05:34):

and let the fuzz bugs begin...

Alex Crichton (Aug 26 2020 at 18:06):

@fitzgen (he/him) btw the auto-reduction oss-fuzz does is really good with wasm-smith

Alex Crichton (Aug 26 2020 at 18:07):

it reduced a 30kb input to 4kb which generated a module with just one type in it

Alex Crichton (Aug 26 2020 at 18:07):

so the pattern you're using for generating a wasm module I think is really amenable to auto-reduction

Alex Crichton (Aug 26 2020 at 18:07):

although you probably knew this already heh

fitzgen (he/him) (Aug 26 2020 at 18:12):

yeah, I think it is a combo of two things:

wasm-opt -ttf was cycling back over the raw input and doing funky stuff like xoring it with a counter, rather than simply padding the end of the input with zeroes. this meant that in order to reproduce a bug that involves something generated after it has cycled back around the input, you have to have the whole prefix. there is a lot of unintentional stuff entrained in the test case.
I switched a few things in the arbitrary crate to be more fuzzing friendly. we are more lax about errors (eg a string that isn't perfectly utf-8 or when we need more input than the fuzzer provided us) and I also made it so that collections are built up via a "continuation byte" rather than by reading a length and then reading that many elements. This latter one was inspired by a paper about test case reduction from the python hypothesis testing framework. See https://github.com/rust-fuzz/arbitrary/issues/52 for details

Generate collections via a "continuation byte" rather than reading a length · Issue #52 · rust-fuzz/arbitrary

Rather than doing let mut collection = Collection::new(); let n = u.arbitrary_len()?; for _ in 0..n { collection.insert(u.arbitrary()?); } to create a collection, we should consider doing let mut c...

Alex Crichton (Aug 26 2020 at 18:14):

yeah I saw that about the keep_going bool which I thought was pretty clever

Alex Crichton (Aug 26 2020 at 18:14):

I do think though that one part about wasm-smith is that it's difficult to have a stable corpus, because as we tweak wasm-smith a previous input could be reinterpreted totally differently (e.g. with the patch I just sent)

Alex Crichton (Aug 26 2020 at 18:14):

not really a huge issue, but also not sure how we'd fix it

fitzgen (he/him) (Aug 26 2020 at 18:16):

agreed. I don't think it is a huge deal and I'm also not sure how to fix it where the fix wouldn't be worse than just rediscovering some coverage

Last updated: Nov 22 2024 at 16:03 UTC