Wasmtime GitHub notifications bot (Mar 02 2022 at 20:13): Wasmtime GitHub notifications bot (Mar 02 2022 at 20:13):alexcrichton labeled issue #3875:
I ended up going down a pretty long rabbit hole this morning for a number of reasons around our fuzzers. The main conclusion I reached so far is that Wasmtime's fuzzing on oss-fuzz has stopped as of 2022-02-24, which is about a week ago at this point. After talking with the oss-fuzz folks the conclusion is that the likely cause of this is that the size of all of Wasmtime's compiled fuzzers is too large and exceeds the disk space capacity of the machines running the fuzzers. A local build of the oss-fuzz image shows that Wasmtime produces ~6.8G of fuzzers as of right-this-red-hot-second.
The easiest fix to apply for now is to pass the
--strip-dead-codeoption tocargo fuzz, which disables passing-Clink-dead-codewhich greatly reduces the size of the output directory to 3.4G (~50% reduction). This should hopefully buy us some runway. According to this linking dead code is intentional and fixes some errors printed out by libfuzzer, but locally I was unable to reproduce those errors and given the recent introduction of the--strip-dead-code-flagit seems like it at least works for others, so I'm hopeful that passing this flag will not impact the actual fuzzing process.In any case though I consider the
--strip-dead-codeargument to be more of a quick fix than a permanent solution. This will continue to limit our ability to add more fuzzers since each new fuzzers is hundreds of megabytes as it statically links all of Wasmtime. My proposed solution to this is to instead have a scheme where thewasmtime_fuzzingcrate is built as adylib(Rust crate type) which all the fuzzers then link to. Each fuzzer would then individually be quite small and we'd only have one copy of Wasmtime on disk for the build artifacts, probably reducing the output size by another 2-3G or so. There's some documentation online about how we need to be careful about dynamic library dependencies for the fuzzer, but that shouldn't be too hard to codify.Overall I think this is something we need to fix before adding more fuzzers, but for the time being if https://github.com/google/oss-fuzz/pull/7354 works then we don't need to fix this immediately because we'll be back to fuzzing and have a good deal of headroom since we're 50% of the limit.
Wasmtime GitHub notifications bot (Mar 02 2022 at 20:13):alexcrichton opened issue #3875:
I ended up going down a pretty long rabbit hole this morning for a number of reasons around our fuzzers. The main conclusion I reached so far is that Wasmtime's fuzzing on oss-fuzz has stopped as of 2022-02-24, which is about a week ago at this point. After talking with the oss-fuzz folks the conclusion is that the likely cause of this is that the size of all of Wasmtime's compiled fuzzers is too large and exceeds the disk space capacity of the machines running the fuzzers. A local build of the oss-fuzz image shows that Wasmtime produces ~6.8G of fuzzers as of right-this-red-hot-second.
The easiest fix to apply for now is to pass the
--strip-dead-codeoption tocargo fuzz, which disables passing-Clink-dead-codewhich greatly reduces the size of the output directory to 3.4G (~50% reduction). This should hopefully buy us some runway. According to this linking dead code is intentional and fixes some errors printed out by libfuzzer, but locally I was unable to reproduce those errors and given the recent introduction of the--strip-dead-code-flagit seems like it at least works for others, so I'm hopeful that passing this flag will not impact the actual fuzzing process.In any case though I consider the
--strip-dead-codeargument to be more of a quick fix than a permanent solution. This will continue to limit our ability to add more fuzzers since each new fuzzers is hundreds of megabytes as it statically links all of Wasmtime. My proposed solution to this is to instead have a scheme where thewasmtime_fuzzingcrate is built as adylib(Rust crate type) which all the fuzzers then link to. Each fuzzer would then individually be quite small and we'd only have one copy of Wasmtime on disk for the build artifacts, probably reducing the output size by another 2-3G or so. There's some documentation online about how we need to be careful about dynamic library dependencies for the fuzzer, but that shouldn't be too hard to codify.Overall I think this is something we need to fix before adding more fuzzers, but for the time being if https://github.com/google/oss-fuzz/pull/7354 works then we don't need to fix this immediately because we'll be back to fuzzing and have a good deal of headroom since we're 50% of the limit.
Wasmtime GitHub notifications bot (Mar 02 2022 at 20:17):alexcrichton commented on issue #3875:
Oh some more information on why this hit us now: I believe this all boils down to https://github.com/bytecodealliance/wasmtime/pull/3836 which was merged the day before we stopped having fuzzers run on oss-fuzz. By linking dead code in to all binaries the way that PR is architected the
wasm-mutatecrate is now linked into all of wasmtime's fuzzers, and thewasm-mutatecrate is quite hefty at around 50MB or so. This means that when multiplied across all our fuzzers we must have _just_ gone over the size limit which means that all our fuzzing stopped after that.I actually confirmed this locally originally because my builds were all failing on arm64 due to the fuzzing builds being too large, and bisection pointed at #3836 as the point when fuzzers stopped building for me locally. Locally I was able to build on arm64 once I passed
--strip-dead-code, however.
Wasmtime GitHub notifications bot (Mar 02 2022 at 20:20):github-actions[bot] commented on issue #3875:
Subscribe to Label Action
cc @fitzgen
<details>
This issue or pull request has been labeled: "fuzzing"Thus the following users have been cc'd because of the following labels:
- fitzgen: fuzzing
To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.
Learn more.
</details>
Last updated: Dec 23 2024 at 12:05 UTC