Stream: git-wasmtime

Topic: wasmtime / issue #3843 Fuzzing against verified fork of s...


view this post on Zulip Wasmtime GitHub notifications bot (Feb 23 2022 at 17:16):

github-actions[bot] commented on issue #3843:

Subscribe to Label Action

cc @fitzgen

<details>
This issue or pull request has been labeled: "fuzzing"

Thus the following users have been cc'd because of the following labels:

To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.

Learn more.
</details>

view this post on Zulip Wasmtime GitHub notifications bot (Feb 23 2022 at 18:21):

abrown commented on issue #3843:

Nice work! Looks like there's just a few annoying bits to fix. I suggest that however we solve the "ocamlfind is not present" error (and the gmp paths as well), we make it easy on ourselves when we set up the environment for this to run on oss-fuzz. I had to tweak this Dockerfile a few times to get the OCaml bits in place; maybe we should just have a script in Wasmtime to do this for us?

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 12:08):

conrad-watt commented on issue #3843:

I'm not sure how to tackle the fuzzing CI failures. Should I be looking at one of the scripts in the ci folder?

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 15:09):

alexcrichton commented on issue #3843:

The fuzz CI failure looks to be:

  make: ocamlfind: Command not found

The CI configuration lives here and this may be as simple as installing ocamlfind as a package, but I don't know much about building ocaml myself. In any case though editing those lines around ocaml installation is where updates would go to get this building.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 16:27):

conrad-watt commented on issue #3843:

The fuzz targets CI seems to be working now (fingers crossed).

BTW, the tests I fixed in my latest commit don't appear to be run as part of the CI. Is this intended? They can be run manually with cargo test --features build-libinterpret in the crates/fuzzing/wasm-spec-interpreter subdir.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 16:31):

conrad-watt edited a comment on issue #3843:

The fuzz targets CI seems to be working now (fingers crossed).

BTW, the tests I fixed in my latest commit don't appear to be run as part of the CI. Is this intended? They can be run manually with cargo test --features build-libinterpret in the crates/fuzzing/wasm-spec-interpreter subdir.

EDIT: if I'm understanding correctly, this means that the default value of LIBGMP_PATHS in my tweaked makefile is actually successfully pointing to the place where the worker installs that dependency.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 16:32):

conrad-watt edited a comment on issue #3843:

The fuzz targets CI seems to be working now (fingers crossed).

EDIT: if I'm understanding correctly, this means that the default value of LIBGMP_PATHS in my tweaked makefile is actually successfully pointing to the place where the worker installs that dependency.

BTW, the tests I fixed in my latest commit don't appear to be run as part of the CI. Is this intended? They can be run manually with cargo test --features build-libinterpret in the crates/fuzzing/wasm-spec-interpreter subdir.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 20:57):

alexcrichton commented on issue #3843:

To get this fuzzed on oss-fuzz we also need to update oss-fuzz's Dockerfile for wasmtime. I attempted a patch that adds libgmp-dev and then uses opam install for ocamlfind as well as zarith (and I had to tweak the CFLAGS for zarith to get it to compile right since it shouldn't use the default fuzzing flags since we're using the system compiler at the docker-image-build-time rather than the llvm compiler). When building wasmtime though I get:

error: failed to run custom build command for `wasm-spec-interpreter v0.1.0 (/src/wasmtime/crates/fuzzing/wasm-spec-interpreter)`

Caused by:
  process didn't exit successfully: `/src/wasmtime/target/release/build/wasm-spec-interpreter-2c2fa92e1c9017fb/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=ocaml/interpret.ml
  cargo:rerun-if-changed=ocaml/Makefile
  make -C spec/interpreter libopt
  make[1]: Entering directory '/src/wasmtime/crates/fuzzing/wasm-spec-interpreter/ocaml/spec/interpreter'
  ls binary/decode.ml binary/decode.mli binary/encode.ml binary/encode.mli binary/utf8.ml binary/utf8.mli exec/eval.ml exec/eval.mli exec/eval_num.ml exec/eval_num.mli exec/eval_vec.ml exec/eval_vec.mli exec/f32.ml exec/f32_convert.ml exec/f32_convert.mli exec/f64.ml exec/f64_convert.ml exec/f64_convert.mli exec/fxx.ml exec/i16.ml exec/i32.ml exec/i32_convert.ml exec/i32_convert.mli exec/i64.ml exec/i64_convert.ml exec/i64_convert.mli exec/i8.ml exec/ixx.ml exec/v128.ml exec/v128.mli host/env.ml host/spectest.ml host/spectest_isa.ml main/flags.ml main/main.ml runtime/func.ml runtime/func.mli runtime/global.ml runtime/global.mli runtime/instance.ml runtime/memory.ml runtime/memory.mli runtime/table.ml runtime/table.mli script/import.ml script/import.mli script/js.ml script/js.mli script/run.ml script/run.mli script/script.ml syntax/ast.ml syntax/free.ml syntax/free.mli syntax/operators.ml syntax/types.ml syntax/values.ml tests/smallint.ml text/arrange.ml text/arrange.mli text/lexer.mli text/lexer.mll text/parse.ml text/parse.mli text/parser.mly text/print.ml text/print.mli util/error.ml util/error.mli util/lib.ml util/lib.mli util/sexpr.ml util/sexpr.mli util/source.ml util/source.mli valid/valid.ml valid/valid.mli verified/WasmRef_Isa_m.ml verified_wrapper/ast_convert.ml verified_wrapper/f32Wrapper.ml verified_wrapper/f32Wrapper_convert.ml verified_wrapper/f64Wrapper.ml verified_wrapper/f64Wrapper_convert.ml verified_wrapper/i32Wrapper_convert.ml verified_wrapper/i64Wrapper_convert.ml verified_wrapper/implWrapper.ml verified_wrapper/implWrapperTypes.ml verified_wrapper/libAux.ml \
        | sed 's:\(.*/\)\{0,1\}\(.*\)\.[^\.]*:\2:' \
        | grep -v main \
        | sort | uniq \
        >wasm.mlpack
  echo >_tags "true: bin_annot"
  echo >>_tags "true: debug"
  echo >>_tags "true: optimize(classic)"
  echo >>_tags "<{util,syntax,binary,text,valid,runtime,exec,script,host,main,tests,verified,verified_wrapper}/*.cmx>: for-pack(Wasm)"
  ocamlbuild -use-ocamlfind -lexflags -ml -cflags '-w +a-3-4-8-11-20-26-27-32-39-42-44-45 -warn-error +a-3 -unboxed-types' -I util -I syntax -I binary -I text -I valid -I runtime -I exec -I script -I host -I main -I tests -I verified -I verified_wrapper -pkgs zarith -libs bigarray -quiet wasm.cmx

  + ocamlfind ocamlc -c -w +a-3-4-8-11-20-26-27-32-39-42-44-45 -warn-error +a-3 -unboxed-types -g -bin-annot -package zarith -I exec -I text -I binary -I runtime -I host -I util -I script -I verified_wrapper -I syntax -I valid -I verified -I main -I tests -o exec/fxx.cmo exec/fxx.ml
  File "exec/fxx.ml", line 1:
  Error (warning 70 [missing-mli]): Cannot find interface file.
  Command exited with code 2.
  rm wasm.mlpack _tags
  make[1]: Leaving directory '/src/wasmtime/crates/fuzzing/wasm-spec-interpreter/ocaml/spec/interpreter'

  --- stderr
  Cloning into 'ocaml/spec'...
  make[1]: *** [Makefile:101: _build/wasm.cmx] Error 10
  make: *** [Makefile:33: spec/interpreter/_build/wasm.cmxa] Error 2
  thread 'main' panicked at 'Failed to build the OCaml library using 'make'.', crates/fuzzing/wasm-spec-interpreter/build.rs:58:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I'm not familiar enough with ocaml builds to know what's going on here, but @conrad-watt do you know what the error is here? If you'd like to reproduce locally you can check out https://github.com/google/oss-fuzz, apply the patch above, and then run python infra/helper.py build_fuzzers wasmtime

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 21:10):

conrad-watt commented on issue #3843:

I think this is a new warning (promoted to error by -warn-error) specific to Ocaml 4.13, so this environment must be building with a very new version of OCaml. Since the error is occuring in the repo I have control over, I've just pushed a small change to the makefile that should get rid of it.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 21:11):

conrad-watt edited a comment on issue #3843:

I think this is a new warning (promoted to error by -warn-error) specific to OCaml 4.13, so this environment must be building with a very new version of OCaml. Since the error is occuring in the repo I have control over, I've just pushed a small change to the makefile that should get rid of it.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 21:12):

conrad-watt edited a comment on issue #3843:

I think this is a new warning (promoted to error by -warn-error) specific to OCaml 4.13, so this environment must be building with a very new version of OCaml. Since the error is occuring in the repo I have control over, I've just pushed a small change to the makefile that should get rid of it.

https://github.com/conrad-watt/spec/commit/7485eb0084b74871f96f261b9f916864596d8f1d

view this post on Zulip Wasmtime GitHub notifications bot (Mar 01 2022 at 23:22):

alexcrichton commented on issue #3843:

Ah that worked perfectly, thanks!

view this post on Zulip Wasmtime GitHub notifications bot (Mar 02 2022 at 17:06):

alexcrichton commented on issue #3843:

https://github.com/google/oss-fuzz/pull/7346 managed to get in in time and the latest build of wasmtime on 2022-03-02 succeeded and differential_spec is in the logs there, so we should start seeing fuzzing on oss-fuzz soon, although hopefully no crashes!

This reminds me actually, if we get a crash we'll reduce it and figure out what's causing it, but @conrad-watt if it's a bug in the spec interpreter where would you prefer issues to be filed? I think the two options are here with you cc'd or your spec repo, and either way should work for us

view this post on Zulip Wasmtime GitHub notifications bot (Mar 02 2022 at 17:41):

conrad-watt commented on issue #3843:

As a default, if it's a Wasm-semantics bug in the spec interpreter I'd prefer it as an issue in my repo - if it's a bug with the build or Rust FFI I guess it can go here (or as a new issue in this repo).

view this post on Zulip Wasmtime GitHub notifications bot (Mar 03 2022 at 16:25):

alexcrichton commented on issue #3843:

Some further updates! We discovered yesterday our fuzzers hadn't been running for almost a week due to completely unrelated issues to this PR. After making the fuzzer binaries smaller fuzzing is indeed running and we got our first "fuzz bug" on the spec interpreter:

Direct leak of 8192 byte(s) in 1 object(s) allocated from:
--
  | #0 0x55ceeef332be in malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
  | #1 0x55ceef1cc8e4 in caml_setup_stack_overflow_detection /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/signals_nat.c:300:15
  | #2 0x55ceef1cc8e4 in caml_init_signals /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/signals_nat.c:279:7
  | #3 0x55ceef1cae50 in caml_startup_common /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/startup_nat.c:140:3
  | #4 0x55ceef1caefa in caml_startup_exn /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/startup_nat.c:163:10
  | #5 0x55ceef1caefa in caml_startup /root/.opam/default/.opam-switch/build/ocaml-base-compiler.4.13.1/runtime/startup_nat.c:168:15
  | #6 0x55ceef1eaf2b in ocaml_interop::runtime::OCamlRuntime::init_persistent::_$u7b$u7b$closure$u7d$u7d$::h66aef86dfb8a1cc1 /rust/registry/src/github.com-1ecc6299db9ec823/ocaml-interop-0.8.7/src/runtime.rs:40:21
  | #7 0x55ceef1eaf2b in std::sync::once::Once::call_once::_$u7b$u7b$closure$u7d$u7d$::h49a51688d2d874b0 /rustc/8769f4ef2fe1efddd1f072485f97f568e7328f79/library/std/src/sync/once.rs:269:41
  | #8 0x55ceeeebcba9 in std::sync::once::Once::call_inner::h53ef098615f1b57f /rustc/8769f4ef2fe1efddd1f072485f97f568e7328f79/library/std/src/sync/once.rs:426:21

Oddly enough I wasn't able to reproduce this locally but it turns out I was using ocaml 4.12 and after upgrading locally to 4.13 I was indeed able to reproduce this. I don't know how to "fix" this so I opted instead to simply ignore leaks in this fuzzer which will hopefully get it really fuzzing after that merges and we get another build tomorrow.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 03 2022 at 16:35):

conrad-watt commented on issue #3843:

I wonder if this is related to this issue (the amount leaked is the same!).

view this post on Zulip Wasmtime GitHub notifications bot (Mar 03 2022 at 16:45):

conrad-watt commented on issue #3843:

Follow-up thought: maybe it would make sense to pin a specific version of OCaml (I'm personally using 4.11.2 locally), since this appears to be the second gotcha introduced because the fuzz setup is grabbing the latest release. It looks like the opam install script the Dockerfile is using can be configured with a --version flag.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 03 2022 at 16:47):

conrad-watt edited a comment on issue #3843:

Follow-up thought: maybe it would make sense to pin a specific version of OCaml (I'm personally using 4.11.2 locally), since this appears to be the second gotcha introduced because the fuzz setup is grabbing the latest release. It looks like the opam install script the Dockerfile is using can be configured with a --version flag.

EDIT: I should add that OCaml will sometimes introduce (usually minor) breaking changes on version jumps, which might catch us out in future.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 03 2022 at 16:50):

alexcrichton commented on issue #3843:

Ah yeah that looks like precisely the issue, shame on me for not doing a google search... Otherwise though we are clearly novices in working with ocaml on the Wasmtime side of things, and pinning to a single working version of ocaml sounds good to me. I'll see if I can frob the docker script to force 4.11.2

view this post on Zulip Wasmtime GitHub notifications bot (Mar 03 2022 at 16:53):

conrad-watt edited a comment on issue #3843:

Follow-up thought: maybe it would make sense to pin a specific version of OCaml (I'm personally using 4.11.2 locally), since this appears to be the second gotcha introduced because the fuzz setup is grabbing the latest release. It looks like the install.sh opam install script the Dockerfile is using can be configured with a --version flag.

EDIT: I should add that OCaml will sometimes introduce (usually minor) breaking changes on version jumps, which might catch us out in future.


Last updated: Dec 23 2024 at 12:05 UTC