Stream: git-wasmtime

Topic: wasmtime / PR #4932 fuzzgen: Avoid `int_divz` traps

view this post on Zulip Wasmtime GitHub notifications bot (Sep 20 2022 at 16:01):

afonso360 opened PR #4932 from fuzz-traps to main:

:wave: Hey,

This avoids int_divz traps generated by fuzzgen by doing a pass over generated code and inserting additional instructions if a particular opcode can cause a int_divz trap.

Currently we decide per function if we are going to avoid all int_divz's. However I ran into something a bit unexpected. Even with one in a million inputs being allowed to trap, we still get a somewhat big amount of runs trapping (~2-3%). I'm not quite sure why this happens.

I suspect this is somehow related to a single input generating a bunch of runs and them all failing and inflating those numbers, but I'm not too sure.

I also did a sanity check run where I always inserted the int_divz check, and predictably got 0%.

Here are the benchmarks:

<details>
<summary><h3>Baseline:</h3></summary>

#49977  NEW    cov: 31759 ft: 183553 corp: 4114/8186Kb lim: 393599 exec/s: 29 rss: 979Mb L: 2926/316258 MS: 1 EraseBytes-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 26348 (52.7%)
Total Runs: 656246
Successful Runs: 539632 (82.2% of Total Runs)
Timed out Runs: 155 (0.0% of Total Runs)
Traps:
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: int_ovf: 8 (0.0% of Total Runs)
        user code: int_divz: 116451 (17.7% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        resumable: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)

</details>

<details>
<summary><h3>This PR:</h3></summary>

#49982  NEW    cov: 32009 ft: 184999 corp: 3984/8155Kb lim: 393599 exec/s: 25 rss: 975Mb L: 5077/393581 MS: 1 CopyPart-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 25940 (51.9%)
Total Runs: 660751
Successful Runs: 639007 (96.7% of Total Runs)
Timed out Runs: 125 (0.0% of Total Runs)
Traps:
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: int_ovf: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: int_divz: 21619 (3.3% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        resumable: 0 (0.0% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)

</details>

<details>
<summary><h3>This PR with 100% insertion rate:</h3></summary>

#49907  NEW    cov: 31943 ft: 184612 corp: 4012/8336Kb lim: 393599 exec/s: 24 rss: 1060Mb L: 687/393581 MS: 2 ChangeByte-ChangeASCIIInt-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 25911 (51.8%)
Total Runs: 834983
Successful Runs: 834813 (100.0% of Total Runs)
Timed out Runs: 170 (0.0% of Total Runs)
Traps:
        resumable: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)
        user code: int_ovf: 0 (0.0% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        user code: int_divz: 0 (0.0% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)

</details>

CC: @jameysharp

view this post on Zulip Wasmtime GitHub notifications bot (Sep 20 2022 at 16:01):

afonso360 edited PR #4932 from fuzz-traps to main:

:wave: Hey,

This avoids int_divz traps generated by fuzzgen by doing a pass over generated code and inserting additional instructions if a particular opcode can cause a int_divz trap.

Currently we decide per function if we are going to avoid all int_divz's. However I ran into something a bit unexpected. Even with one in a million inputs being allowed to trap, we still get a somewhat big amount of runs trapping (~2-3%). I'm not quite sure why this happens (if anyone has any ideas, let me know!).

I suspect this is somehow related to a single input generating a bunch of runs and them all failing and inflating those numbers, but I'm not too sure.

I also did a sanity check run where I always inserted the int_divz check, and predictably got 0%.

Here are the benchmarks:

<details>
<summary><h3>Baseline:</h3></summary>

#49977  NEW    cov: 31759 ft: 183553 corp: 4114/8186Kb lim: 393599 exec/s: 29 rss: 979Mb L: 2926/316258 MS: 1 EraseBytes-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 26348 (52.7%)
Total Runs: 656246
Successful Runs: 539632 (82.2% of Total Runs)
Timed out Runs: 155 (0.0% of Total Runs)
Traps:
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: int_ovf: 8 (0.0% of Total Runs)
        user code: int_divz: 116451 (17.7% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        resumable: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)

</details>

<details>
<summary><h3>This PR:</h3></summary>

#49982  NEW    cov: 32009 ft: 184999 corp: 3984/8155Kb lim: 393599 exec/s: 25 rss: 975Mb L: 5077/393581 MS: 1 CopyPart-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 25940 (51.9%)
Total Runs: 660751
Successful Runs: 639007 (96.7% of Total Runs)
Timed out Runs: 125 (0.0% of Total Runs)
Traps:
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: int_ovf: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: int_divz: 21619 (3.3% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        resumable: 0 (0.0% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)

</details>

<details>
<summary><h3>This PR with 100% insertion rate:</h3></summary>

#49907  NEW    cov: 31943 ft: 184612 corp: 4012/8336Kb lim: 393599 exec/s: 24 rss: 1060Mb L: 687/393581 MS: 2 ChangeByte-ChangeASCIIInt-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 25911 (51.8%)
Total Runs: 834983
Successful Runs: 834813 (100.0% of Total Runs)
Timed out Runs: 170 (0.0% of Total Runs)
Traps:
        resumable: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)
        user code: int_ovf: 0 (0.0% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        user code: int_divz: 0 (0.0% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)

</details>

CC: @jameysharp

view this post on Zulip Wasmtime GitHub notifications bot (Sep 20 2022 at 17:05):

afonso360 edited PR #4932 from fuzz-traps to main:

:wave: Hey,

This avoids int_divz traps generated by fuzzgen by doing a pass over generated code and inserting additional instructions if a particular opcode can cause a int_divz trap.

Currently we decide per function if we are going to avoid all int_divz's. However I ran into something a bit unexpected. Even with one in a million inputs being allowed to trap, we still get a somewhat big amount of runs trapping (~2-3%). I'm not quite sure why this happens (if anyone has any ideas, let me know!).

I suspect this is somehow related to a single input generating a bunch of runs and them all failing and inflating those numbers, or maybe the fuzzer really likes to explore the inputs where we allow int_divz's? I'm not sure.

I also did a sanity check run where I always inserted the int_divz check, and predictably got 0%.

Here are the benchmarks:

<details>
<summary><h3>Baseline:</h3></summary>

#49977  NEW    cov: 31759 ft: 183553 corp: 4114/8186Kb lim: 393599 exec/s: 29 rss: 979Mb L: 2926/316258 MS: 1 EraseBytes-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 26348 (52.7%)
Total Runs: 656246
Successful Runs: 539632 (82.2% of Total Runs)
Timed out Runs: 155 (0.0% of Total Runs)
Traps:
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: int_ovf: 8 (0.0% of Total Runs)
        user code: int_divz: 116451 (17.7% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        resumable: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)

</details>

<details>
<summary><h3>This PR:</h3></summary>

#49982  NEW    cov: 32009 ft: 184999 corp: 3984/8155Kb lim: 393599 exec/s: 25 rss: 975Mb L: 5077/393581 MS: 1 CopyPart-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 25940 (51.9%)
Total Runs: 660751
Successful Runs: 639007 (96.7% of Total Runs)
Timed out Runs: 125 (0.0% of Total Runs)
Traps:
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: int_ovf: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: int_divz: 21619 (3.3% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        resumable: 0 (0.0% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)

</details>

<details>
<summary><h3>This PR with 100% insertion rate:</h3></summary>

#49907  NEW    cov: 31943 ft: 184612 corp: 4012/8336Kb lim: 393599 exec/s: 24 rss: 1060Mb L: 687/393581 MS: 2 ChangeByte-ChangeASCIIInt-
== FuzzGen Statistics  ====================
Total Inputs: 50000
Valid Inputs: 25911 (51.8%)
Total Runs: 834983
Successful Runs: 834813 (100.0% of Total Runs)
Timed out Runs: 170 (0.0% of Total Runs)
Traps:
        resumable: 0 (0.0% of Total Runs)
        user code: table_oob: 0 (0.0% of Total Runs)
        user code: interrupt: 0 (0.0% of Total Runs)
        user code: bad_sig: 0 (0.0% of Total Runs)
        user code: icall_null: 0 (0.0% of Total Runs)
        user debug: 0 (0.0% of Total Runs)
        user code: heap_misaligned: 0 (0.0% of Total Runs)
        user code: int_ovf: 0 (0.0% of Total Runs)
        user code: bad_toint: 0 (0.0% of Total Runs)
        user code: stk_ovf: 0 (0.0% of Total Runs)
        user code: heap_oob: 0 (0.0% of Total Runs)
        user code: int_divz: 0 (0.0% of Total Runs)
        user code: unreachable: 0 (0.0% of Total Runs)

</details>

CC: @jameysharp

view this post on Zulip Wasmtime GitHub notifications bot (Sep 20 2022 at 20:03):

jameysharp submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 20 2022 at 20:03):

jameysharp created PR review comment:

There's nothing wrong with the way you've written this, but I like this alternative:

    matches!(opcode, Opcode::Sdiv | Opcode::Udiv | Opcode::Srem | Opcode::Urem)

view this post on Zulip Wasmtime GitHub notifications bot (Sep 20 2022 at 20:03):

jameysharp submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 21 2022 at 06:15):

afonso360 submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 21 2022 at 06:15):

afonso360 created PR review comment:

I do like that version better as well!

view this post on Zulip Wasmtime GitHub notifications bot (Sep 21 2022 at 06:21):

afonso360 updated PR #4932 from fuzz-traps to main.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 23 2022 at 17:19):

jameysharp merged PR #4932.

Last updated: Nov 22 2024 at 17:03 UTC