Stream: git-wasmtime

Topic: wasmtime / issue #9449 Wasmtime fuzzing: manually inject ...


view this post on Zulip Wasmtime GitHub notifications bot (Oct 10 2024 at 21:50):

cfallin opened issue #9449:

In the Wasmtime meeting today, there was a lot of good discussion about our procedures and standards around fuzzing, and how to make process improvements to make sure fuzzing catches issues in on-by-default features. I thought I'd expand a bit on an idea I proposed there.

The basic problem is: the configuration space of Wasm features that get fuzzed is the result of a fairly nontrivial interaction between feature selection logic in multiple places and crates. There are plausible reasons why this isn't all centralized; even if we do eventually get to a point where there is One Main Function that determines what is fuzzed, there is still the question of whether the plumbing that flows outward from that definition faithfully preserves the feature.

I'm suggesting as a principle that the only way to really know whether fuzzing will catch bugs is to... make fuzzing catch bugs. An "integration test" wrapped around our whole setup, as it were. The idea (which I'll maybe call a "fire drill" approach) is:

The main idea is that we want to test the whole pipeline all the way to oss-fuzz-email-in-inboxes. Nothing else tests this whole pipeline today. We simply don't know if it works, except when we get emails. The idea here is to turn the email rate up from "0 + actual errors" to "fault-injected stream + actual errors", so it's always nonzero, so we can keep verifying it works.

There were a few objections or alternative ideas in discussions today:

Thoughts?

view this post on Zulip Wasmtime GitHub notifications bot (Oct 10 2024 at 22:44):

alexcrichton commented on issue #9449:

One idea I had sort of along these lines (but also relatively orthogonal) which was similar to what Nick was saying about tests in wasmtime-fuzzing is https://github.com/bytecodealliance/wasmtime/pull/9452. Mostly cc'ing that as related-to, but definitely not replacing this.

I personally really like the idea of testing "oss-fuzz-email-in-inboxes" as oss-fuzz infrastructure itself has hiccups relatively frequently and this helps us diagnose that as well. The success of this testing though I think will be based in how little effort we need to apply to it. Automating as much as we can with "just hit the button" or something like that I think would be best. I wouldn't want to have to maintain a document explaining all the various steps of updating branches, how/when to revert, what cadence, etc.

That being said I think this is all quite doable. GitHub Actions gives us cron jobs and we can leave an unprotected branch able to get modified by it. Having a button to hit for "ok revert that top commit" which ideally auto-merges or just doesn't even go through PRs would be pretty nice too.

view this post on Zulip Wasmtime GitHub notifications bot (Oct 10 2024 at 22:54):

cfallin commented on issue #9449:

Ah, perhaps this is workable: note certain points in the code with a magic comment string. // delete-to-test-fuzzing before a line of code or somesuch. Then a GH Actions cronjob could pick one of these at random and update the branch once a week, as you say. (The selection of such lines needs a little human thought, but not much: should be deletions that won't break compilation trivially, and will do interesting things: misalign the stack, generate the wrong value, skip a GC write barrier, ...)

view this post on Zulip Wasmtime GitHub notifications bot (Oct 10 2024 at 22:59):

alexcrichton commented on issue #9449:

Oh man now I really like that. One could imagine dialing that up to 11 and fully automate this entire system. For example a branch could be made and if a maintainer doesn't flag somewhere "hey the fuzzers found the bug" then an issue would automatically get opened and filed. Otherwise the management of the "fuzz this broken wasmtime" branch would be 100% automated in theory.


Last updated: Dec 23 2024 at 13:07 UTC