Wasmtime GitHub notifications bot (Sep 14 2022 at 19:24): Wasmtime GitHub notifications bot (Sep 14 2022 at 19:24):afonso360 commented on issue #4894:
Hey, I think I addressed most of the feedback above (Thanks for the great review!).
I've also done some additional changes:
- Added the
backwards_block_ratioto the config- Moved all of the block generation code to inside
generate_blocks
Wasmtime GitHub notifications bot (Sep 14 2022 at 19:25):afonso360 edited a comment on issue #4894:
Hey, I think I addressed most of the feedback above (Thanks for the great review!). I've added this to the last two commits so that its easier to review.
I've also done some additional changes:
- Added the
backwards_block_ratioto the config- Moved all of the block generation code to inside
generate_blocksEdit: Oops, it now has conflicts, I'll rebase.
Wasmtime GitHub notifications bot (Sep 14 2022 at 19:25):afonso360 edited a comment on issue #4894:
Hey, I think I addressed most of the feedback above (Thanks for the great review!). I've added this to the last two commits so that its easier to review.
I've also done some additional changes:
- Added the
backwards_block_ratioto the config- Moved all of the block generation code inside
generate_blocksEdit: Oops, it now has conflicts, I'll rebase.
Wasmtime GitHub notifications bot (Sep 14 2022 at 19:37):afonso360 edited a comment on issue #4894:
Hey, I think I addressed most of the feedback above (Thanks for the great review!). I've added this to the last two commits so that its easier to review.
I've also done some additional changes:
- Added the
backwards_block_ratioto the config- Moved all of the block generation code inside
generate_blocksEdit: Oops, it now has conflicts, I'll rebase.
Edit2: Rebased, But I'm going to fuzz this for a bit
Wasmtime GitHub notifications bot (Sep 14 2022 at 20:02):afonso360 commented on issue #4894:
Something went wrong in the rebase and its now crashing, I'm going to take a look at this again tomorrow.
Wasmtime GitHub notifications bot (Sep 15 2022 at 14:25):afonso360 commented on issue #4894:
So, it turns out nothing went wrong in the rebase and it was a preexisting issue with this PR. We were sometimes selecting
block0as a target for branching, which makes the verifier very unhappy. I didn't notice this because these inputs were silently being rejected and #4896 finally exposed this.I think this may explain the difference between the results in this PR and the original stats in https://github.com/bytecodealliance/wasmtime/issues/3050#issuecomment-1243037213. Although here we have 10k stacks where we don't know what happened and that seems quite a bit.
I'm re running the benchmarks to see if this discrepancy has gone away.
Wasmtime GitHub notifications bot (Sep 15 2022 at 18:12):afonso360 commented on issue #4894:
So, here are the results.
This is using a custom patch on top of the arbitrary crate to report failures. See https://github.com/bytecodealliance/wasmtime/issues/3050#issuecomment-1244761070 for more details.
<details>
<summary><h3>Baseline</h3></summary>#99826 NEW cov: 32083 ft: 189603 corp: 5384/10156Kb lim: 1048576 exec/s: 10 rss: 1030Mb L: 2586/1048576 MS: 4 EraseBytes-ChangeASCIIInt-ShuffleBytes-PersAutoDict- DE: "Q\x03\x00\x00\x00\x00\x00\x0 0"- == FuzzGen Statistics ==================== Total Inputs: 100000 Valid Inputs: 61534 (61.5%) Total Runs: 426680 Successful Runs: 211464 (49.6% of Total Runs) Timed out Runs: 34331 (8.0% of Total Runs) Traps: user code: int_divz: 180882 (42.4% of Total Runs) user code: heap_misaligned: 0 (0.0% of Total Runs) user code: icall_null: 0 (0.0% of Total Runs) user code: bad_toint: 0 (0.0% of Total Runs) resumable: 0 (0.0% of Total Runs) user code: int_ovf: 3 (0.0% of Total Runs) user code: interrupt: 0 (0.0% of Total Runs) user debug: 0 (0.0% of Total Runs) user code: stk_ovf: 0 (0.0% of Total Runs) user code: bad_sig: 0 (0.0% of Total Runs) user code: unreachable: 0 (0.0% of Total Runs) user code: heap_oob: 0 (0.0% of Total Runs) user code: table_oob: 0 (0.0% of Total Runs)[BUILD] stack.rs Loaded 10000 Loaded 20000 Loaded 30000 Loaded 38472 stacks First Line duplicates 100.0 - Count: 38472 - Line: EmptyChoose Two Line duplicates 6.3 - Count: 2415 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::generate_br_table 8.3 - Count: 3199 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::generate_jumptables 1.4 - Count: 534 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::generate_switch 20.2 - Count: 7779 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::generate_target_block 35.2 - Count: 13526 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::get_variable_of_type 21.9 - Count: 8413 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::stack_slot_with_size 6.8 - Count: 2606 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::insert_call</details>
<details>
<summary><h3>This PR</h3></summary>#99990 REDUCE cov: 31890 ft: 186525 corp: 4925/10508Kb lim: 393599 exec/s: 19 rss: 1034Mb L: 144/316625 MS: 1 EraseBytes- == FuzzGen Statistics ==================== Total Inputs: 100000 Valid Inputs: 74409 (74.4%) Total Runs: 1540097 Successful Runs: 1240133 (80.5% of Total Runs) Timed out Runs: 356 (0.0% of Total Runs) Traps: user code: unreachable: 0 (0.0% of Total Runs) user code: stk_ovf: 0 (0.0% of Total Runs) user code: int_divz: 299534 (19.4% of Total Runs) user code: heap_oob: 0 (0.0% of Total Runs) user code: bad_toint: 0 (0.0% of Total Runs) user code: int_ovf: 74 (0.0% of Total Runs) user code: table_oob: 0 (0.0% of Total Runs) resumable: 0 (0.0% of Total Runs) user debug: 0 (0.0% of Total Runs) user code: heap_misaligned: 0 (0.0% of Total Runs) user code: icall_null: 0 (0.0% of Total Runs) user code: bad_sig: 0 (0.0% of Total Runs) user code: interrupt: 0 (0.0% of Total Runs)[BUILD] stack.rs Loaded 10000 Loaded 20000 Loaded 25591 stacks First Line duplicates 100.0 - Count: 25591 - Line: EmptyChoose Two Line duplicates 59.0 - Count: 15098 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::get_variable_of_type 29.9 - Count: 7645 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::FunctionGenerator::stack_slot_with_size 11.1 - Count: 2848 - Line: EmptyChoose 0: arbitrary::trace_error 1: arbitrary::unstructured::Unstructured::choose_index 2: cranelift_fuzzgen::function_generator::insert_call</details>
Wow! #4895 makes a big difference in
int_divztraps!For baseline we have
38472stacks and(100000-61534)=38466invalid inputs. That is close enough that I'm going to call it valid. I suspect the 6 extra stacks come from the fact that I run this with-runs=100010and 6 out of those 10 extra executions failed.I think this should be good for merging now. We no longer see the discrepancy in stacks.
Wasmtime GitHub notifications bot (Sep 15 2022 at 18:39):jameysharp commented on issue #4894:
This is fantastic. To summarize your statistics:
- 21% increase in valid inputs
- more than 3x as many total runs while doubling executions per second
- almost 6x as many successful runs
- eliminated almost 99% of timeouts
Last updated: Dec 23 2024 at 12:05 UTC