Stream: git-wasmtime

Topic: wasmtime / issue #5967 cranelift: Fuzz mid-end optimizations

Wasmtime GitHub notifications bot (Mar 08 2023 at 21:05):

jameysharp opened issue #5967:

Feature

Currently, the cranelift-fuzzgen fuzz target performs differential fuzzing between our Cranelift interpreter and the native backend for the host. The native backend may be configured to run various mid-end optimizations before lowering, but as far as I can tell the interpreter is never configured to run any optimizations.

I think it might be useful to also differentially fuzz the interpreter without optimizations against the interpreter with various optimizations applied.

Benefit

This should give us an additional source of confidence that our mid-end optimization rules are not changing the behavior of generated code in observable ways.

Implementation

I think cranelift-fuzzgen can choose at random whether to use the native backend or the interpreter as the second implementation. I'm not sure how difficult it'll be to feed the interpreter the results of mid-end optimization, but I suspect the infrastructure we have for optimization filetests indicates that we have all the pieces we should need.

Alternatives

We could add a new fuzz target that's specifically for differentially fuzzing the interpreter against itself, but I think it would mostly duplicate infrastructure that's already in the cranelift-fuzzgen target.

Wasmtime GitHub notifications bot (Mar 08 2023 at 21:09):

jameysharp labeled issue #5967:

Feature

Currently, the cranelift-fuzzgen fuzz target performs differential fuzzing between our Cranelift interpreter and the native backend for the host. The native backend may be configured to run various mid-end optimizations before lowering, but as far as I can tell the interpreter is never configured to run any optimizations.

I think it might be useful to also differentially fuzz the interpreter without optimizations against the interpreter with various optimizations applied.

Benefit

This should give us an additional source of confidence that our mid-end optimization rules are not changing the behavior of generated code in observable ways.

Implementation

I think cranelift-fuzzgen can choose at random whether to use the native backend or the interpreter as the second implementation. I'm not sure how difficult it'll be to feed the interpreter the results of mid-end optimization, but I suspect the infrastructure we have for optimization filetests indicates that we have all the pieces we should need.

Alternatives

We could add a new fuzz target that's specifically for differentially fuzzing the interpreter against itself, but I think it would mostly duplicate infrastructure that's already in the cranelift-fuzzgen target.

Wasmtime GitHub notifications bot (Mar 08 2023 at 21:09):

jameysharp labeled issue #5967:

Feature

Currently, the cranelift-fuzzgen fuzz target performs differential fuzzing between our Cranelift interpreter and the native backend for the host. The native backend may be configured to run various mid-end optimizations before lowering, but as far as I can tell the interpreter is never configured to run any optimizations.

I think it might be useful to also differentially fuzz the interpreter without optimizations against the interpreter with various optimizations applied.

Benefit

This should give us an additional source of confidence that our mid-end optimization rules are not changing the behavior of generated code in observable ways.

Implementation

I think cranelift-fuzzgen can choose at random whether to use the native backend or the interpreter as the second implementation. I'm not sure how difficult it'll be to feed the interpreter the results of mid-end optimization, but I suspect the infrastructure we have for optimization filetests indicates that we have all the pieces we should need.

Alternatives

We could add a new fuzz target that's specifically for differentially fuzzing the interpreter against itself, but I think it would mostly duplicate infrastructure that's already in the cranelift-fuzzgen target.

Wasmtime GitHub notifications bot (Mar 08 2023 at 21:09):

jameysharp labeled issue #5967:

Feature

Currently, the cranelift-fuzzgen fuzz target performs differential fuzzing between our Cranelift interpreter and the native backend for the host. The native backend may be configured to run various mid-end optimizations before lowering, but as far as I can tell the interpreter is never configured to run any optimizations.

I think it might be useful to also differentially fuzz the interpreter without optimizations against the interpreter with various optimizations applied.

Benefit

This should give us an additional source of confidence that our mid-end optimization rules are not changing the behavior of generated code in observable ways.

Implementation

I think cranelift-fuzzgen can choose at random whether to use the native backend or the interpreter as the second implementation. I'm not sure how difficult it'll be to feed the interpreter the results of mid-end optimization, but I suspect the infrastructure we have for optimization filetests indicates that we have all the pieces we should need.

Alternatives

We could add a new fuzz target that's specifically for differentially fuzzing the interpreter against itself, but I think it would mostly duplicate infrastructure that's already in the cranelift-fuzzgen target.

Wasmtime GitHub notifications bot (Mar 10 2023 at 20:05):

jameysharp commented on issue #5967:

A little more detail:

The file which I think needs to be modified here is fuzz/fuzz_targets/cranelift-fuzzgen.rs, which you can run by installing cargo fuzz and running cargo +nightly fuzz run cranelift-fuzzgen.

Currently, that file calls run_in_interpreter first, and if that succeeds then for the same program and input it calls run_in_host. The idea here is that instead of calling run_in_host, we'd call run_in_interpreter a second time, but the CLIF we'd pass it would be the result of running mid-end optimizations.

For cranelift-fuzzgen to decide whether to use run_in_host or to optimize and run in the interpreter instead, we need to add a bool field to TestCase, and set that in TestCase::generate from the result of calling gen.u.arbitrary().

To get the optimized functions, we need something like this, which I've more or less borrowed from cranelift/filetests/src/test_optimize.rs:

    let optimized: Vec<Function> = testcase
        .functions
        .iter()
        .map(|func| {
            let comp_ctx = cranelift_codegen::Context::for_function(function.clone);
            comp_ctx.optimize(&testcase.isa).unwrap();
            comp_ctx.func
        })
        .collect();

Wasmtime GitHub notifications bot (Mar 11 2023 at 00:21):

fitzgen commented on issue #5967:

I was talking with @jacarte the other day as well and he had a similar idea: do the wasm-mutate e-graphs thing (select a random e-node from the e-class rather than the best e-node). I guess this would also fit in with the chaos mode work.

Wasmtime GitHub notifications bot (Mar 11 2023 at 18:34):

afonso360 commented on issue #5967:

We have bugpoint which already implements some shrinking mutations, we can make that a library. Some of them are not semantics preserving, but can maybe also help with the chaos mode work! (CC: #4799)

Wasmtime GitHub notifications bot (Mar 11 2023 at 18:43):

bjorn3 commented on issue #5967:

Most mutations in bugpoint are written to cut away big chunks of the ir by eg replacing instructions with a constant or a trap. None of them are meant to be semantics preserving and I don't think they are useful for fuzzing either. Just for reducing a test case.

Wasmtime GitHub notifications bot (Mar 13 2023 at 20:25):

jameysharp commented on issue #5967:

I love the idea of using random semantics-preserving mutations as part of cranelift-fuzzgen. @Jacarte or @fitzgen, could one of you write up an issue for that?

Thanks for reminding me of #4799, @afonso360. Re-reading the history there, I still think my comment was right and your idea was good: for the cranelift-icache target, bugpoint could potentially reflect a lot of edits that are useful for validating that the incremental cache doesn't reuse old compile results for functions which have changed.

For cranelift-fuzzgen though, I think @bjorn3 is right because we need to preserve semantics since we're going to compare the results of actually running the functions.

At any rate, that is all a topic for another issue. I think #5998 will address this current issue nicely.

Wasmtime GitHub notifications bot (Mar 13 2023 at 20:49):

fitzgen commented on issue #5967:

Filed #6009

Wasmtime GitHub notifications bot (Mar 17 2023 at 03:46):

alexcrichton closed issue #5967:

Feature

Currently, the cranelift-fuzzgen fuzz target performs differential fuzzing between our Cranelift interpreter and the native backend for the host. The native backend may be configured to run various mid-end optimizations before lowering, but as far as I can tell the interpreter is never configured to run any optimizations.

I think it might be useful to also differentially fuzz the interpreter without optimizations against the interpreter with various optimizations applied.

Benefit

This should give us an additional source of confidence that our mid-end optimization rules are not changing the behavior of generated code in observable ways.

Implementation

I think cranelift-fuzzgen can choose at random whether to use the native backend or the interpreter as the second implementation. I'm not sure how difficult it'll be to feed the interpreter the results of mid-end optimization, but I suspect the infrastructure we have for optimization filetests indicates that we have all the pieces we should need.

Alternatives

We could add a new fuzz target that's specifically for differentially fuzzing the interpreter against itself, but I think it would mostly duplicate infrastructure that's already in the cranelift-fuzzgen target.

Wasmtime GitHub notifications bot (Mar 17 2023 at 16:05):

Jacarte commented on issue #5967:

I love the idea of using random semantics-preserving mutations as part of cranelift-fuzzgen. @Jacarte or @fitzgen, could one of you write up an issue for that?

Thanks for reminding me of #4799, @afonso360. Re-reading the history there, I still think my comment was right and your idea was good: for the cranelift-icache target, bugpoint could potentially reflect a lot of edits that are useful for validating that the incremental cache doesn't reuse old compile results for functions which have changed.

For cranelift-fuzzgen though, I think @bjorn3 is right because we need to preserve semantics since we're going to compare the results of actually running the functions.

At any rate, that is all a topic for another issue. I think #5998 will address this current issue nicely.

Actually, It does not need to be semantizally equivalent. The rewriting rule can be, in theory, wathever transformation. For example x => x + 1 is not semantically equivalent but still the e-graph will be constructed on that.

Last updated: Nov 22 2024 at 16:03 UTC