Stream: wasmtime

Topic: fuzzing for other repos


view this post on Zulip Alex Crichton (May 19 2020 at 21:44):

@fitzgen (he/him) hey so I've been looking again at the fuzzers for other repos other than wasmtime which aren't hooked up to oss-fuzz

view this post on Zulip Alex Crichton (May 19 2020 at 21:44):

and I was struck with an idea, what if we used github actions to fuzz everything?

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:45):

like on a cron job?

view this post on Zulip Alex Crichton (May 19 2020 at 21:45):

I've got a loose idea that looks like this -- https://github.com/alexcrichton/wasm-tools/blob/fdd674a95ac8aa9963e6f7da316393775e07814e/.github/workflows/fuzz.yml

Low level tooling for WebAssembly . Contribute to alexcrichton/wasm-tools development by creating an account on GitHub.

view this post on Zulip Alex Crichton (May 19 2020 at 21:45):

where you basically drop that file in your repo

view this post on Zulip Alex Crichton (May 19 2020 at 21:45):

and all of a sudden you're continuously running fuzzers

view this post on Zulip Alex Crichton (May 19 2020 at 21:45):

yeah I'm thinking we'd do like a daily cron job

view this post on Zulip Alex Crichton (May 19 2020 at 21:45):

where it builds all the fuzzers then forks off a bunch of parallel targets to run each fuzzer

view this post on Zulip Alex Crichton (May 19 2020 at 21:45):

and if anything fails it automatically uploads crash logs as well as files an issue

view this post on Zulip Alex Crichton (May 19 2020 at 21:46):

e.g. https://github.com/alexcrichton/wasm-tools/issues/1

https://github.com/alexcrichton/wasm-tools/actions/runs/109745591

view this post on Zulip Alex Crichton (May 19 2020 at 21:46):

wdyt about doing this for things like walrus/wasmparser/etc?

view this post on Zulip Alex Crichton (May 19 2020 at 21:46):

one thing I don't know what to do about is the corpus

view this post on Zulip Alex Crichton (May 19 2020 at 21:46):

I'm also not sure whether it's worth it to run all the fuzzers in separate builders

view this post on Zulip Alex Crichton (May 19 2020 at 21:47):

b/c it means I can't easily use cargo fuzz to run the fuzzers, which seems like a bummer

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:47):

semi related: I just set up gimli's CI for fuzzing so that it builds all fuzz targets in one job, and then runs each one in N parallel jobs: https://github.com/gimli-rs/gimli/pull/512/commits/b7d1a194c6c4742f69ee77f3c0e3afb532fa84ab

also, if fuzzing finds a crash/panic/etc, it uploads the failing input as an artifact that you can download for debugging

This adds a few fuzz targets for various DWARF sections, and also integrates it with our CI, so that the fuzz targets are built in one job, and then each fuzz target is run in N parallel jobs for f...

view this post on Zulip Alex Crichton (May 19 2020 at 21:47):

I think you and I just converged on the same thing

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:47):

one thing I don't know what to do about is the corpus

I think a separate repo, like we do with wasmtime (and I also did with gimli) is the way to go

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:48):

I think we can be better about triggers tho, than what you have sketched out

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:48):

one sec

view this post on Zulip Alex Crichton (May 19 2020 at 21:48):

it'd be awesome if we could package this all up and put it in the cargo-fuzz org

view this post on Zulip Alex Crichton (May 19 2020 at 21:48):

so we just slap a "here's a tiny snippet to include" and it just runs

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:48):

https://github.com/bytecodealliance/wasmtime/blob/master/.github/workflows/subscribe-to-label.yml#L5-L10

we can actually use a scheduled thing, instead of a weird auto-fuzz branch

Standalone JIT-style runtime for WebAssembly, using Cranelift - bytecodealliance/wasmtime

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:49):

it'd be awesome if we could package this all up and put it in the cargo-fuzz org

totally

view this post on Zulip Alex Crichton (May 19 2020 at 21:49):

oh auto-fuzz is just my own local testing

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:49):

:+1:

view this post on Zulip Alex Crichton (May 19 2020 at 21:50):

I was thinking it might be best to not do this on PRs since it'd slow them down

view this post on Zulip Alex Crichton (May 19 2020 at 21:50):

but should be easy enough to queue it up for pushes to master

view this post on Zulip Alex Crichton (May 19 2020 at 21:50):

in that ideally we'd let the fuzzers run for like an hour

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:50):

oh nice, you auto open an issue!

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:51):

that's sweet

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:52):

the time limit for an action is like 50 minutes or something, right?

we could probably just do a cron job every hour to fuzz for ~40 minutes

also, maybe makes sense to do a corpus minification run once per day, and push that back to the corpus repo

view this post on Zulip Alex Crichton (May 19 2020 at 21:52):

apparently it's 6 hours (!)

view this post on Zulip Alex Crichton (May 19 2020 at 21:52):

this would eat into the max parallism of the repo so I don't think we'd want it running semi-permanently

view this post on Zulip Alex Crichton (May 19 2020 at 21:53):

but man auto-compaction would be nice

view this post on Zulip Alex Crichton (May 19 2020 at 21:53):

I'd ideally like the corpus to auto-expand over time too

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:54):

for relatively low-volume repos, like wasm-tools, I think it would be fine modulo maybe not running every fuzz target in parallel

view this post on Zulip Alex Crichton (May 19 2020 at 21:55):

so libfuzzer automatically adds to the corpus, right?

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:55):

expanding the corpus is just

cd corpus
git add .
git commit -m "update corpus"
git push

after fuzzing

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:56):

so libfuzzer automatically adds to the corpus, right?

when it finds inputs that trigger new code paths, yes

view this post on Zulip Alex Crichton (May 19 2020 at 21:56):

ah ok

view this post on Zulip Alex Crichton (May 19 2020 at 21:56):

I do think a downside of this though is it'll just keep reporting the same bugs once it hits a crash

view this post on Zulip Alex Crichton (May 19 2020 at 21:56):

b/c it won't know it's already reported

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:56):

yeah, oss-fuzz has a lot of nice bug de-dupe infra, even if it doesn't understand rust's backtraces very well

view this post on Zulip Alex Crichton (May 19 2020 at 21:57):

ideally we'd just hook up more projects to oss-fuzz

view this post on Zulip Alex Crichton (May 19 2020 at 21:57):

but it seems like that's somewhat higher overhead...

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:57):

I don't think that should be too hard; I saw that serde is in their now too

view this post on Zulip Alex Crichton (May 19 2020 at 21:58):

oh interesting...

view this post on Zulip Alex Crichton (May 19 2020 at 21:58):

I found https://google.github.io/oss-fuzz/getting-started/continuous-integration/ as well btw

view this post on Zulip Alex Crichton (May 19 2020 at 21:58):

which we may want to consider for wasmtime if it doesn't take too too long

view this post on Zulip fitzgen (he/him) (May 19 2020 at 21:59):

Alex Crichton said:

I found https://google.github.io/oss-fuzz/getting-started/continuous-integration/ as well btw

oh nice! I think this is pretty new, because I don't remember seeing it in their docs when I last browsed them

view this post on Zulip Alex Crichton (May 19 2020 at 21:59):

yeah same

view this post on Zulip Alex Crichton (May 19 2020 at 22:01):

hm ok I'll probably hold off on this sort of continuous fuzzing for now

view this post on Zulip Alex Crichton (May 19 2020 at 22:02):

mainly b/c of the issue spam possibility

view this post on Zulip fitzgen (he/him) (May 19 2020 at 22:02):

yeah, I think in conclusion, testing the waters with more oss-fuzz integration is probably the way to go, and if that doesn't work out, then investigate a github actions-based thing

view this post on Zulip Alex Crichton (May 19 2020 at 22:04):

looking at https://github.com/google/oss-fuzz/pull/3785 tbh it looks like you don't even need a formal request

serde-json is a core library for working with json in Rust. Cross-referencing This has been coordinated with the authors of serde-json, in particular @dtolnay.

view this post on Zulip Alex Crichton (May 19 2020 at 22:04):

that was just a PR

view this post on Zulip Till Schneidereit (May 19 2020 at 22:09):

Do we know GitHub's position on this kind of usage pattern? I.e., could there be a concern of them seeing it as improper use of resources?

view this post on Zulip fitzgen (he/him) (May 19 2020 at 22:11):

We don't know their position, but I find it unlikely that they would think of it as improper use, and even if they did, I don't think they would be angry so much as chagrinned and apologetic.


Last updated: Dec 23 2024 at 14:03 UTC