Stream: wasmtime

Topic: fuzzing and `R_AARCH64_CALL26` limits


view this post on Zulip Alex Crichton (Mar 02 2022 at 15:45):

@Sam Parker or @Anton Kirilov I've got a question for y'all as our resident arm64-experts. When I build wasmtime's fuzzers myself on AArch64 I occasionally see linker errors that look like:

view this post on Zulip Alex Crichton (Mar 02 2022 at 15:45):

er, sorry, early send

view this post on Zulip Alex Crichton (Mar 02 2022 at 15:45):

          compile_maybe_invalid.e9aef4b8-cgu.0:(.text.symbol_nmae+0x6c): relocation truncated to fit: R_AARCH64_CALL26 against symbol `__sanitizer_cov_trace_const_cmp4' defined in .text.__sanitizer_cov_trace_const_cmp4 section in /path/to/libfoo.rlib(FuzzerTracePC.o)

view this post on Zulip Alex Crichton (Mar 02 2022 at 15:46):

basically: relocation truncated to fit: R_AARCH64_CALL26

I was wondering, do y'all know how to get around this? Is there a linker option to say "please insert some sort of trampoline to get me all the way there" or something like that?

view this post on Zulip Alex Crichton (Mar 02 2022 at 15:46):

note that this is the native linker, so not actually related to wasmtime itself, only the fact that our fuzz outputs are apparently quite large (I'm independently trying to go figure out why they're so large as well)

view this post on Zulip Sam Parker (Mar 02 2022 at 16:10):

What optimisation level is being used and have you tried using an optimization level (-Os) to reduce code size? If it's a static build you can also try passing -mcmodel=large to the compiler.

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:20):

this is an optimized fuzzed build and I haven't tried -Copt-level=s yet (this is all mostly rust code)

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:21):

I tried -Ccode-model=large but it printed out lots of warnings from llvm about the tls model not being supported

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:21):

something about ELF TLS only being supported with a "small" code model or a "local-exec" tls model, but when forcing the tls model to "local-exec" it printed the same warnings

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:22):

specifically:

LLVM ERROR: ELF TLS only supported in small memory model or in local exec TLS model

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:22):

and that's with:

RUSTFLAGS='-Ccode-model=large -Ztls-model=local-exec' CARGO_TARGET_DIR=large cargo +nightly fuzz build

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:23):

I built binutils 2.38 and it appears to not work out-of-the-box at least

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:24):

oh also for the "large" code model it may not be supported in Rust for one reason or another (or I don't know what I'm doing), if I rustc foo.rs -C code-model=large for fn main() {} and nothing else it prints:

  = note: /home/acrichto/root/bin/ld: foo.foo.938a9993-cgu.0.rcgu.o: relocation R_AARCH64_MOVW_UABS_G0_NC against `a local symbol' can not be used when making a shared object; recompile with -fPIC
          /home/acrichto/root/bin/ld: foo.foo.938a9993-cgu.1.rcgu.o: relocation R_AARCH64_MOVW_UABS_G0_NC against `a local symbol' can not be used when making a shared object; recompile with -fPIC
          foo.foo.938a9993-cgu.0.rcgu.o:(.data.rel.ro..L__unnamed_1+0x0): dangerous relocation: unsupported relocation
          foo.foo.938a9993-cgu.0.rcgu.o:(.data.rel.ro..L__unnamed_1+0x18): dangerous relocation: unsupported relocation
          foo.foo.938a9993-cgu.0.rcgu.o:(.data.rel.ro..L__unnamed_1+0x20): dangerous relocation: unsupported relocation
          foo.foo.938a9993-cgu.0.rcgu.o:(.data.rel.ro..L__unnamed_1+0x28): dangerous relocation: unsupported relocation
          collect2: error: ld returned 1 exit status

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:24):

I'm trying to figure out what's so darned big but I can't figure out a good way to determine that

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:26):

this hasn't been a persistent problem in the past but now I can't build the fuzzers at all on aarch64 unfortunately

view this post on Zulip Alex Crichton (Mar 02 2022 at 16:30):

ok looks like -Copt-level=s at least works for me for now, and wow the fuzzers are ~300mb apiece...

view this post on Zulip Alex Crichton (Mar 02 2022 at 17:00):

https://github.com/bytecodealliance/wasmtime/pull/3872 should probably also help with the size of cranelift itself

I frequently notice that the fuzz build of cranelift-codegen takes an extremely long time and recently realized that one issue is that when fuzzers are built we enable all of the backends in cranel...

view this post on Zulip Alex Crichton (Mar 02 2022 at 17:11):

alas no still too large to link with default settings...

view this post on Zulip Alex Crichton (Mar 02 2022 at 18:54):

Ok more investigation and thinking has revealed that cargo fuzz by default passes -Clink-dead-code to rustc. The cargo fuzz command also has --strip-dead-code as a parmeter, and when I pass that then all the fuzzers link locally because the binaries are ~100MB smaller.

It sounds, though, like there's no AArch64 linker option or thing like that to get this to "just work". I probably need to pass --strip-dead-code when I build the fuzzers myself.

@fitzgen (he/him) question for you as you ostensibly maintain cargo-fuzz -- back in https://github.com/rust-fuzz/cargo-fuzz/commit/86b4889e3e0f6faac8433659bcac08c4b32097c0 it was claimed that dead code caused fuzzers to print errors, but in https://github.com/rust-fuzz/cargo-fuzz/pull/260 an option was added to strip dead code and running locally that appears to work ok to me at least. Do you know if something about libfuzzer changed in the meantime, and if so if the "strip dead code" option could be turned on by default?

Without it, some optimized targets would fail to launch with the following error: ERROR: The size of coverage PC tables does not match the number of instrumented PCs. This might be a compiler bug,...
The -Clink-dead-code flag was added to fix an error with certain targets. However, it can also cause problems. Example: You depend on crate A which links to library B. A includes an unused referenc...

view this post on Zulip Alex Crichton (Mar 02 2022 at 18:56):

one other thing that came up is that https://github.com/bytecodealliance/wasmtime/pull/3836 is the "cause" for why I can't build the fuzzers on aarch64 any more. AFAIK wasm-mutate is a pretty hefty library, especially when fuzzing, but it's only being linked because dead code is linked. Otherwise that should only affect the one fuzz target

There's definitely more to be done here but I think this is a good starting point. The big open question for the future in my mind is how this will integrate with the way we heavily rely on swarm t...

view this post on Zulip fitzgen (he/him) (Mar 02 2022 at 19:50):

Yeah I don't know the details. libFuzzer is always a mess of linker errors if you look at it the wrong way, unfortunately :(

view this post on Zulip Alex Crichton (Mar 02 2022 at 20:59):

On a whim I attempted to use some different linkers here. I happened to be testing a cargo fuzz build --dev build which has actually never worked for me before on arm64, but the timings I get are:

I... may just switch to using LLD on this machine as a result

view this post on Zulip Sam Parker (Mar 03 2022 at 13:15):

Wow, that's quite the difference!


Last updated: Dec 23 2024 at 14:03 UTC