Stream: git-wasmtime

Topic: wasmtime / PR #4958 fuzzgen: Add `fcvt_*` ops

view this post on Zulip Wasmtime GitHub notifications bot (Sep 25 2022 at 15:05):

afonso360 opened PR #4958 from fuzzgen-fcvt to main:

:wave: Hey,

Following up from #4884 (Thanks!) this PR introduces fcvt_* ops to the fuzzer. Additionally it also adds a pass to minimize the number of traps generated by these ops.

Like the int_divz pass, the number of traps generated doesn't seem to correspond with the amount requested in the config. I think I've actually figured out something here, if I limit the amount of bytes the fuzzer uses the amount of traps is reduced by a lot. (I got 0.4% int_ovf with -max_len=2048)

That leads me to believe that we get such a high number of traps due to executing a lot of inputs. This is something that I want to look further into since I think there could actually be a performance benefit from limiting this.

As a sanity check I also tried to benchmark with 100% insertion rate, and as expected we don't get the other traps.

Benchmarks

<details>
<summary><h3>Baseline</h3></summary>

#92127  REDUCE cov: 30535 ft: 173903 corp: 4510/10465Kb lim: 393617 exec/s: 24 rss: 2448Mb L: 124/393581 MS: 1 EraseBytes-
== FuzzGen Statistics  ====================
Valid Inputs: 50000
Total Runs: 1469777
Successful Runs: 1415404 (96.3% of Total Runs)
Timed out Runs: 425 (0.0% of Total Runs)
Traps:
        user code: int_divz: 53948 (3.7% of Total Runs)
#92200  NEW    cov: 30535 ft: 173904 corp: 4511/10470Kb lim: 393617 exec/s: 23 rss: 2448Mb L: 4614/393581 MS: 3 ChangeASCIIInt-ChangeByte-CMP- DE: "\x1e\x00\x00\x00"-

</details>

<details>
<summary><h3>This PR</h3></summary>

I've seen the int_ovf trap count vary a lot while testing, from 0.4% up to almost 7%.

#86636  NEW    cov: 31377 ft: 178040 corp: 4651/8643Kb lim: 119720 exec/s: 25 rss: 2375Mb L: 4548/65539 MS: 2 ChangeASCIIInt-CopyPart-
== FuzzGen Statistics  ====================
Valid Inputs: 50000
Total Runs: 1184585
Successful Runs: 1099371 (92.8% of Total Runs)
Timed out Runs: 1066 (0.1% of Total Runs)
Traps:
        user code: int_ovf: 40522 (3.4% of Total Runs)
        user code: int_divz: 33112 (2.8% of Total Runs)
        user code: bad_toint: 10514 (0.9% of Total Runs)
#86713  REDUCE cov: 31377 ft: 178042 corp: 4652/8645Kb lim: 119720 exec/s: 25 rss: 2375Mb L: 1761/65539 MS: 2 InsertRepeatedBytes-InsertRepeatedBytes-

</details>

<details>
<summary><h3>This PR while always applying the trap pass</h3></summary>

#89715  REDUCE cov: 31208 ft: 177781 corp: 4618/8793Kb lim: 136654 exec/s: 25 rss: 2342Mb L: 2397/136620 MS: 1 PersAutoDict- DE: "\x00\x00\x00Y"-
== FuzzGen Statistics  ====================
Valid Inputs: 50000
Total Runs: 1315468
Successful Runs: 1273898 (96.8% of Total Runs)
Timed out Runs: 1016 (0.1% of Total Runs)
Traps:
        user code: int_divz: 40554 (3.1% of Total Runs)
#89889  NEW    cov: 31208 ft: 177782 corp: 4619/8808Kb lim: 136654 exec/s: 25 rss: 2342Mb L: 15765/136620 MS: 4 CopyPart-CopyPart-InsertByte-CrossOver-

</details>

cc: @jameysharp

view this post on Zulip Wasmtime GitHub notifications bot (Sep 25 2022 at 15:18):

afonso360 updated PR #4958 from fuzzgen-fcvt to main.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 25 2022 at 15:22):

afonso360 updated PR #4958 from fuzzgen-fcvt to main.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 25 2022 at 15:23):

afonso360 updated PR #4958 from fuzzgen-fcvt to main.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 26 2022 at 09:51):

afonso360 edited PR #4958 from fuzzgen-fcvt to main:

:wave: Hey,

Following up from #4884 (Thanks!) this PR introduces fcvt_* ops to the fuzzer. Additionally it also adds a pass to minimize the number of traps generated by these ops.

Like the int_divz pass, the number of traps generated doesn't seem to correspond with the amount requested in the config. I think I've actually figured out something here, if I limit the amount of bytes the fuzzer uses, the amount of traps is reduced by a lot. (I got 0.4% int_ovf with -max_len=2048)

That leads me to believe that we get such a high number of traps due to executing a lot of inputs. This is something that I want to look further into since I think there could actually be a performance benefit from limiting this.

As a sanity check I also tried to benchmark with 100% insertion rate, and as expected we don't get the other traps.

Benchmarks

<details>
<summary><h3>Baseline</h3></summary>

#92127  REDUCE cov: 30535 ft: 173903 corp: 4510/10465Kb lim: 393617 exec/s: 24 rss: 2448Mb L: 124/393581 MS: 1 EraseBytes-
== FuzzGen Statistics  ====================
Valid Inputs: 50000
Total Runs: 1469777
Successful Runs: 1415404 (96.3% of Total Runs)
Timed out Runs: 425 (0.0% of Total Runs)
Traps:
        user code: int_divz: 53948 (3.7% of Total Runs)
#92200  NEW    cov: 30535 ft: 173904 corp: 4511/10470Kb lim: 393617 exec/s: 23 rss: 2448Mb L: 4614/393581 MS: 3 ChangeASCIIInt-ChangeByte-CMP- DE: "\x1e\x00\x00\x00"-

</details>

<details>
<summary><h3>This PR</h3></summary>

I've seen the int_ovf trap count vary a lot while testing, from 0.4% up to almost 7%.

#86636  NEW    cov: 31377 ft: 178040 corp: 4651/8643Kb lim: 119720 exec/s: 25 rss: 2375Mb L: 4548/65539 MS: 2 ChangeASCIIInt-CopyPart-
== FuzzGen Statistics  ====================
Valid Inputs: 50000
Total Runs: 1184585
Successful Runs: 1099371 (92.8% of Total Runs)
Timed out Runs: 1066 (0.1% of Total Runs)
Traps:
        user code: int_ovf: 40522 (3.4% of Total Runs)
        user code: int_divz: 33112 (2.8% of Total Runs)
        user code: bad_toint: 10514 (0.9% of Total Runs)
#86713  REDUCE cov: 31377 ft: 178042 corp: 4652/8645Kb lim: 119720 exec/s: 25 rss: 2375Mb L: 1761/65539 MS: 2 InsertRepeatedBytes-InsertRepeatedBytes-

</details>

<details>
<summary><h3>This PR while always applying the trap pass</h3></summary>

#89715  REDUCE cov: 31208 ft: 177781 corp: 4618/8793Kb lim: 136654 exec/s: 25 rss: 2342Mb L: 2397/136620 MS: 1 PersAutoDict- DE: "\x00\x00\x00Y"-
== FuzzGen Statistics  ====================
Valid Inputs: 50000
Total Runs: 1315468
Successful Runs: 1273898 (96.8% of Total Runs)
Timed out Runs: 1016 (0.1% of Total Runs)
Traps:
        user code: int_divz: 40554 (3.1% of Total Runs)
#89889  NEW    cov: 31208 ft: 177782 corp: 4619/8808Kb lim: 136654 exec/s: 25 rss: 2342Mb L: 15765/136620 MS: 4 CopyPart-CopyPart-InsertByte-CrossOver-

</details>

cc: @jameysharp

view this post on Zulip Wasmtime GitHub notifications bot (Sep 27 2022 at 16:52):

jameysharp submitted PR review.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 27 2022 at 16:52):

jameysharp merged PR #4958.

Last updated: Nov 22 2024 at 16:03 UTC