github-actions[bot] commented on issue #3843:
Subscribe to Label Action
cc @fitzgen
<details>
This issue or pull request has been labeled: "fuzzing"Thus the following users have been cc'd because of the following labels:
- fitzgen: fuzzing
To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.
Learn more.
</details>
abrown commented on issue #3843:
Nice work! Looks like there's just a few annoying bits to fix. I suggest that however we solve the "
ocamlfind
is not present" error (and the gmp paths as well), we make it easy on ourselves when we set up the environment for this to run on oss-fuzz. I had to tweak this Dockerfile a few times to get the OCaml bits in place; maybe we should just have a script in Wasmtime to do this for us?
conrad-watt commented on issue #3843:
I'm not sure how to tackle the fuzzing CI failures. Should I be looking at one of the scripts in the
ci
folder?
alexcrichton commented on issue #3843:
The fuzz CI failure looks to be:
make: ocamlfind: Command not found
The CI configuration lives here and this may be as simple as installing
ocamlfind
as a package, but I don't know much about building ocaml myself. In any case though editing those lines around ocaml installation is where updates would go to get this building.
conrad-watt commented on issue #3843:
The fuzz targets CI seems to be working now (fingers crossed).
BTW, the tests I fixed in my latest commit don't appear to be run as part of the CI. Is this intended? They can be run manually with
cargo test --features build-libinterpret
in thecrates/fuzzing/wasm-spec-interpreter
subdir.
conrad-watt edited a comment on issue #3843:
The fuzz targets CI seems to be working now (fingers crossed).
BTW, the tests I fixed in my latest commit don't appear to be run as part of the CI. Is this intended? They can be run manually with
cargo test --features build-libinterpret
in thecrates/fuzzing/wasm-spec-interpreter
subdir.EDIT: if I'm understanding correctly, this means that the default value of
LIBGMP_PATHS
in my tweaked makefile is actually successfully pointing to the place where the worker installs that dependency.
conrad-watt edited a comment on issue #3843:
The fuzz targets CI seems to be working now (fingers crossed).
EDIT: if I'm understanding correctly, this means that the default value of
LIBGMP_PATHS
in my tweaked makefile is actually successfully pointing to the place where the worker installs that dependency.BTW, the tests I fixed in my latest commit don't appear to be run as part of the CI. Is this intended? They can be run manually with
cargo test --features build-libinterpret
in thecrates/fuzzing/wasm-spec-interpreter
subdir.
alexcrichton commented on issue #3843:
To get this fuzzed on oss-fuzz we also need to update oss-fuzz's
Dockerfile
for wasmtime. I attempted a patch that addslibgmp-dev
and then usesopam install
forocamlfind
as well aszarith
(and I had to tweak the CFLAGS forzarith
to get it to compile right since it shouldn't use the default fuzzing flags since we're using the system compiler at the docker-image-build-time rather than the llvm compiler). When building wasmtime though I get:error: failed to run custom build command for `wasm-spec-interpreter v0.1.0 (/src/wasmtime/crates/fuzzing/wasm-spec-interpreter)` Caused by: process didn't exit successfully: `/src/wasmtime/target/release/build/wasm-spec-interpreter-2c2fa92e1c9017fb/build-script-build` (exit status: 101) --- stdout cargo:rerun-if-changed=ocaml/interpret.ml cargo:rerun-if-changed=ocaml/Makefile make -C spec/interpreter libopt make[1]: Entering directory '/src/wasmtime/crates/fuzzing/wasm-spec-interpreter/ocaml/spec/interpreter' ls binary/decode.ml binary/decode.mli binary/encode.ml binary/encode.mli binary/utf8.ml binary/utf8.mli exec/eval.ml exec/eval.mli exec/eval_num.ml exec/eval_num.mli exec/eval_vec.ml exec/eval_vec.mli exec/f32.ml exec/f32_convert.ml exec/f32_convert.mli exec/f64.ml exec/f64_convert.ml exec/f64_convert.mli exec/fxx.ml exec/i16.ml exec/i32.ml exec/i32_convert.ml exec/i32_convert.mli exec/i64.ml exec/i64_convert.ml exec/i64_convert.mli exec/i8.ml exec/ixx.ml exec/v128.ml exec/v128.mli host/env.ml host/spectest.ml host/spectest_isa.ml main/flags.ml main/main.ml runtime/func.ml runtime/func.mli runtime/global.ml runtime/global.mli runtime/instance.ml runtime/memory.ml runtime/memory.mli runtime/table.ml runtime/table.mli script/import.ml script/import.mli script/js.ml script/js.mli script/run.ml script/run.mli script/script.ml syntax/ast.ml syntax/free.ml syntax/free.mli syntax/operators.ml syntax/types.ml syntax/values.ml tests/smallint.ml text/arrange.ml text/arrange.mli text/lexer.mli text/lexer.mll text/parse.ml text/parse.mli text/parser.mly text/print.ml text/print.mli util/error.ml util/error.mli util/lib.ml util/lib.mli util/sexpr.ml util/sexpr.mli util/source.ml util/source.mli valid/valid.ml valid/valid.mli verified/WasmRef_Isa_m.ml verified_wrapper/ast_convert.ml verified_wrapper/f32Wrapper.ml verified_wrapper/f32Wrapper_convert.ml verified_wrapper/f64Wrapper.ml verified_wrapper/f64Wrapper_convert.ml verified_wrapper/i32Wrapper_convert.ml verified_wrapper/i64Wrapper_convert.ml verified_wrapper/implWrapper.ml verified_wrapper/implWrapperTypes.ml verified_wrapper/libAux.ml \ | sed 's:\(.*/\)\{0,1\}\(.*\)\.[^\.]*:\2:' \ | grep -v main \ | sort | uniq \ >wasm.mlpack echo >_tags "true: bin_annot" echo >>_tags "true: debug" echo >>_tags "true: optimize(classic)" echo >>_tags "<{util,syntax,binary,text,valid,runtime,exec,script,host,main,tests,verified,verified_wrapper}/*.cmx>: for-pack(Wasm)" ocamlbuild -use-ocamlfind -lexflags -ml -cflags '-w +a-3-4-8-11-20-26-27-32-39-42-44-45 -warn-error +a-3 -unboxed-types' -I util -I syntax -I binary -I text -I valid -I runtime -I exec -I script -I host -I main -I tests -I verified -I verified_wrapper -pkgs zarith -libs bigarray -quiet wasm.cmx + ocamlfind ocamlc -c -w +a-3-4-8-11-20-26-27-32-39-42-44-45 -warn-error +a-3 -unboxed-types -g -bin-annot -package zarith -I exec -I text -I binary -I runtime -I host -I util -I script -I verified_wrapper -I syntax -I valid -I verified -I main -I tests -o exec/fxx.cmo exec/fxx.ml File "exec/fxx.ml", line 1: Error (warning 70 [missing-mli]): Cannot find interface file. Command exited with code 2. rm wasm.mlpack _tags make[1]: Leaving directory '/src/wasmtime/crates/fuzzing/wasm-spec-interpreter/ocaml/spec/interpreter' --- stderr Cloning into 'ocaml/spec'... make[1]: *** [Makefile:101: _build/wasm.cmx] Error 10 make: *** [Makefile:33: spec/interpreter/_build/wasm.cmxa] Error 2 thread 'main' panicked at 'Failed to build the OCaml library using 'make'.', crates/fuzzing/wasm-spec-interpreter/build.rs:58:5 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
I'm not familiar enough with ocaml builds to know what's going on here, but @conrad-watt do you know what the error is here? If you'd like to reproduce locally you can check out https://github.com/google/oss-fuzz, apply the patch above, and then run
python infra/helper.py build_fuzzers wasmtime
conrad-watt commented on issue #3843:
I think this is a new warning (promoted to error by
-warn-error
) specific to Ocaml 4.13, so this environment must be building with a very new version of OCaml. Since the error is occuring in the repo I have control over, I've just pushed a small change to the makefile that should get rid of it.
conrad-watt edited a comment on issue #3843:
I think this is a new warning (promoted to error by
-warn-error
) specific to OCaml 4.13, so this environment must be building with a very new version of OCaml. Since the error is occuring in the repo I have control over, I've just pushed a small change to the makefile that should get rid of it.
conrad-watt edited a comment on issue #3843:
I think this is a new warning (promoted to error by
-warn-error
) specific to OCaml 4.13, so this environment must be building with a very new version of OCaml. Since the error is occuring in the repo I have control over, I've just pushed a small change to the makefile that should get rid of it.https://github.com/conrad-watt/spec/commit/7485eb0084b74871f96f261b9f916864596d8f1d
alexcrichton commented on issue #3843:
Ah that worked perfectly, thanks!
alexcrichton commented on issue #3843:
https://github.com/google/oss-fuzz/pull/7346 managed to get in in time and the latest build of wasmtime on 2022-03-02 succeeded and
differential_spec
is in the logs there, so we should start seeing fuzzing on oss-fuzz soon, although hopefully no crashes!This reminds me actually, if we get a crash we'll reduce it and figure out what's causing it, but @conrad-watt if it's a bug in the spec interpreter where would you prefer issues to be filed? I think the two options are here with you cc'd or your spec repo, and either way should work for us
conrad-watt commented on issue #3843:
As a default, if it's a Wasm-semantics bug in the spec interpreter I'd prefer it as an issue in my repo - if it's a bug with the build or Rust FFI I guess it can go here (or as a new issue in this repo).
alexcrichton commented on issue #3843:
Some further updates! We discovered yesterday our fuzzers hadn't been running for almost a week due to completely unrelated issues to this PR. After making the fuzzer binaries smaller fuzzing is indeed running and we got our first "fuzz bug" on the spec interpreter:
Direct leak of 8192 byte(s) in 1 object(s) allocated from: -- | #0 0x55ceeef332be in malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 | #1 0x55ceef1cc8e4 in caml_setup_stack_overflow_detection /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/signals_nat.c:300:15 | #2 0x55ceef1cc8e4 in caml_init_signals /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/signals_nat.c:279:7 | #3 0x55ceef1cae50 in caml_startup_common /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/startup_nat.c:140:3 | #4 0x55ceef1caefa in caml_startup_exn /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/startup_nat.c:163:10 | #5 0x55ceef1caefa in caml_startup /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/startup_nat.c:168:15 | #6 0x55ceef1eaf2b in ocaml_interop::runtime::OCamlRuntime::init_persistent::_$u7b$u7b$closure$u7d$u7d$::h66aef86dfb8a1cc1 /rust/registry/src/github.com-1ecc6299db9ec823/ocaml-interop-0.8.7/src/runtime.rs:40:21 | #7 0x55ceef1eaf2b in std::sync::once::Once::call_once::_$u7b$u7b$closure$u7d$u7d$::h49a51688d2d874b0 /rustc/8769f4ef2fe1efddd1f072485f97f568e7328f79/library/std/src/sync/once.rs:269:41 | #8 0x55ceeeebcba9 in std::sync::once::Once::call_inner::h53ef098615f1b57f /rustc/8769f4ef2fe1efddd1f072485f97f568e7328f79/library/std/src/sync/once.rs:426:21
Oddly enough I wasn't able to reproduce this locally but it turns out I was using ocaml 4.12 and after upgrading locally to 4.13 I was indeed able to reproduce this. I don't know how to "fix" this so I opted instead to simply ignore leaks in this fuzzer which will hopefully get it really fuzzing after that merges and we get another build tomorrow.
conrad-watt commented on issue #3843:
I wonder if this is related to this issue (the amount leaked is the same!).
conrad-watt commented on issue #3843:
Follow-up thought: maybe it would make sense to pin a specific version of OCaml (I'm personally using 4.11.2 locally), since this appears to be the second gotcha introduced because the fuzz setup is grabbing the latest release. It looks like the opam install script the Dockerfile is using can be configured with a
--version
flag.
conrad-watt edited a comment on issue #3843:
Follow-up thought: maybe it would make sense to pin a specific version of OCaml (I'm personally using 4.11.2 locally), since this appears to be the second gotcha introduced because the fuzz setup is grabbing the latest release. It looks like the opam install script the Dockerfile is using can be configured with a
--version
flag.EDIT: I should add that OCaml will sometimes introduce (usually minor) breaking changes on version jumps, which might catch us out in future.
alexcrichton commented on issue #3843:
Ah yeah that looks like precisely the issue, shame on me for not doing a google search... Otherwise though we are clearly novices in working with ocaml on the Wasmtime side of things, and pinning to a single working version of ocaml sounds good to me. I'll see if I can frob the docker script to force 4.11.2
conrad-watt edited a comment on issue #3843:
Follow-up thought: maybe it would make sense to pin a specific version of OCaml (I'm personally using 4.11.2 locally), since this appears to be the second gotcha introduced because the fuzz setup is grabbing the latest release. It looks like the
install.sh
opam install script the Dockerfile is using can be configured with a--version
flag.EDIT: I should add that OCaml will sometimes introduce (usually minor) breaking changes on version jumps, which might catch us out in future.
Last updated: Dec 23 2024 at 12:05 UTC