Alex Crichton (May 19 2020 at 21:44): Alex Crichton (May 19 2020 at 21:44):@fitzgen (he/him) hey so I've been looking again at the fuzzers for other repos other than wasmtime which aren't hooked up to oss-fuzz
Alex Crichton (May 19 2020 at 21:44):and I was struck with an idea, what if we used github actions to fuzz everything?
fitzgen (he/him) (May 19 2020 at 21:45):like on a cron job?
Alex Crichton (May 19 2020 at 21:45):I've got a loose idea that looks like this -- https://github.com/alexcrichton/wasm-tools/blob/fdd674a95ac8aa9963e6f7da316393775e07814e/.github/workflows/fuzz.yml
Alex Crichton (May 19 2020 at 21:45):where you basically drop that file in your repo
Alex Crichton (May 19 2020 at 21:45):and all of a sudden you're continuously running fuzzers
Alex Crichton (May 19 2020 at 21:45):yeah I'm thinking we'd do like a daily cron job
Alex Crichton (May 19 2020 at 21:45):where it builds all the fuzzers then forks off a bunch of parallel targets to run each fuzzer
Alex Crichton (May 19 2020 at 21:45):and if anything fails it automatically uploads crash logs as well as files an issue
Alex Crichton (May 19 2020 at 21:46):e.g. https://github.com/alexcrichton/wasm-tools/issues/1
Alex Crichton (May 19 2020 at 21:46):wdyt about doing this for things like walrus/wasmparser/etc?
Alex Crichton (May 19 2020 at 21:46):one thing I don't know what to do about is the corpus
Alex Crichton (May 19 2020 at 21:46):I'm also not sure whether it's worth it to run all the fuzzers in separate builders
Alex Crichton (May 19 2020 at 21:47):b/c it means I can't easily use cargo fuzz to run the fuzzers, which seems like a bummer
fitzgen (he/him) (May 19 2020 at 21:47):semi related: I just set up gimli's CI for fuzzing so that it builds all fuzz targets in one job, and then runs each one in N parallel jobs: https://github.com/gimli-rs/gimli/pull/512/commits/b7d1a194c6c4742f69ee77f3c0e3afb532fa84ab
also, if fuzzing finds a crash/panic/etc, it uploads the failing input as an artifact that you can download for debugging
Alex Crichton (May 19 2020 at 21:47):I think you and I just converged on the same thing
fitzgen (he/him) (May 19 2020 at 21:47):one thing I don't know what to do about is the corpus
I think a separate repo, like we do with wasmtime (and I also did with gimli) is the way to go
fitzgen (he/him) (May 19 2020 at 21:48):I think we can be better about triggers tho, than what you have sketched out
fitzgen (he/him) (May 19 2020 at 21:48):one sec
Alex Crichton (May 19 2020 at 21:48):it'd be awesome if we could package this all up and put it in the cargo-fuzz org
Alex Crichton (May 19 2020 at 21:48):so we just slap a "here's a tiny snippet to include" and it just runs
fitzgen (he/him) (May 19 2020 at 21:48):we can actually use a scheduled thing, instead of a weird auto-fuzz branch
fitzgen (he/him) (May 19 2020 at 21:49):it'd be awesome if we could package this all up and put it in the cargo-fuzz org
totally
Alex Crichton (May 19 2020 at 21:49):oh auto-fuzz is just my own local testing
fitzgen (he/him) (May 19 2020 at 21:49)::+1:
Alex Crichton (May 19 2020 at 21:50):I was thinking it might be best to not do this on PRs since it'd slow them down
Alex Crichton (May 19 2020 at 21:50):but should be easy enough to queue it up for pushes to master
Alex Crichton (May 19 2020 at 21:50):in that ideally we'd let the fuzzers run for like an hour
fitzgen (he/him) (May 19 2020 at 21:50):oh nice, you auto open an issue!
fitzgen (he/him) (May 19 2020 at 21:51):that's sweet
fitzgen (he/him) (May 19 2020 at 21:52):the time limit for an action is like 50 minutes or something, right?
we could probably just do a cron job every hour to fuzz for ~40 minutes
also, maybe makes sense to do a corpus minification run once per day, and push that back to the corpus repo
Alex Crichton (May 19 2020 at 21:52):apparently it's 6 hours (!)
Alex Crichton (May 19 2020 at 21:52):this would eat into the max parallism of the repo so I don't think we'd want it running semi-permanently
Alex Crichton (May 19 2020 at 21:53):but man auto-compaction would be nice
Alex Crichton (May 19 2020 at 21:53):I'd ideally like the corpus to auto-expand over time too
fitzgen (he/him) (May 19 2020 at 21:54):for relatively low-volume repos, like wasm-tools, I think it would be fine modulo maybe not running every fuzz target in parallel
Alex Crichton (May 19 2020 at 21:55):so libfuzzer automatically adds to the corpus, right?
fitzgen (he/him) (May 19 2020 at 21:55):expanding the corpus is just
cd corpus
git add .
git commit -m "update corpus"
git push
after fuzzing
fitzgen (he/him) (May 19 2020 at 21:56):so libfuzzer automatically adds to the corpus, right?
when it finds inputs that trigger new code paths, yes
Alex Crichton (May 19 2020 at 21:56):ah ok
Alex Crichton (May 19 2020 at 21:56):I do think a downside of this though is it'll just keep reporting the same bugs once it hits a crash
Alex Crichton (May 19 2020 at 21:56):b/c it won't know it's already reported
fitzgen (he/him) (May 19 2020 at 21:56):yeah, oss-fuzz has a lot of nice bug de-dupe infra, even if it doesn't understand rust's backtraces very well
Alex Crichton (May 19 2020 at 21:57):ideally we'd just hook up more projects to oss-fuzz
Alex Crichton (May 19 2020 at 21:57):but it seems like that's somewhat higher overhead...
fitzgen (he/him) (May 19 2020 at 21:57):I don't think that should be too hard; I saw that serde is in their now too
Alex Crichton (May 19 2020 at 21:58):oh interesting...
Alex Crichton (May 19 2020 at 21:58):I found https://google.github.io/oss-fuzz/getting-started/continuous-integration/ as well btw
Alex Crichton (May 19 2020 at 21:58):which we may want to consider for wasmtime if it doesn't take too too long
fitzgen (he/him) (May 19 2020 at 21:59):Alex Crichton said:
I found https://google.github.io/oss-fuzz/getting-started/continuous-integration/ as well btw
oh nice! I think this is pretty new, because I don't remember seeing it in their docs when I last browsed them
Alex Crichton (May 19 2020 at 21:59):yeah same
Alex Crichton (May 19 2020 at 22:01):hm ok I'll probably hold off on this sort of continuous fuzzing for now
Alex Crichton (May 19 2020 at 22:02):mainly b/c of the issue spam possibility
fitzgen (he/him) (May 19 2020 at 22:02):yeah, I think in conclusion, testing the waters with more oss-fuzz integration is probably the way to go, and if that doesn't work out, then investigate a github actions-based thing
Alex Crichton (May 19 2020 at 22:04):looking at https://github.com/google/oss-fuzz/pull/3785 tbh it looks like you don't even need a formal request
Alex Crichton (May 19 2020 at 22:04):that was just a PR
Till Schneidereit (May 19 2020 at 22:09):Do we know GitHub's position on this kind of usage pattern? I.e., could there be a concern of them seeing it as improper use of resources?
fitzgen (he/him) (May 19 2020 at 22:11):We don't know their position, but I find it unlikely that they would think of it as improper use, and even if they did, I don't think they would be angry so much as chagrinned and apologetic.
Last updated: Nov 22 2024 at 17:03 UTC