Alex Crichton (Jun 23 2022 at 15:50): Alex Crichton (Jun 23 2022 at 15:50):@Anton Kirilov I'm in the process of trying to move our aarch64.S file to a global_asm! block in Rust to avoid the need for an external asssembler, and I wanted to see at the same time if we could drop some bits and pieces from the file in case LLVM provides them by default. The appending to .note.gnu.property you recently added I'm not sure if LLVM handles or not. Is there a CLI command I can verify that the final binary has "BIT bits enabled"? I'd like to see if I drop the block what the difference in the final binary is
Alex Crichton (Jun 23 2022 at 18:14):apologies for the extra ping but I left a comment here as well -- https://github.com/bytecodealliance/wasmtime/pull/4306#discussion_r905321213
Anton Kirilov (Jun 23 2022 at 18:17):I was just about to answer :smile:.
Anton Kirilov (Jun 23 2022 at 18:19):I think that readelf -n should answer your question.
Anton Kirilov (Jun 23 2022 at 18:20):However, note that with respect to (static) linking BTI support is an all-or-nothing affair.
Alex Crichton (Jun 23 2022 at 18:21):ah ok so on main if I cargo test -p wasmtime-fiber which forces inclusion of the aarch64 bits the resulting binary shows:
$ readelf -n target/debug/deps/wasmtime_fiber-0da0f8ddb3bb19e6
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 5b8f1f636ce7db50b76cdd2f613d24f9ed007c60
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.7.0
which I think is missing? But I think that's expected where I didn't build the Rust code with bti support?
Anton Kirilov (Jun 23 2022 at 18:22):That is, if at least one of the object files that are linked into the final executable lacks the ELF note, then the linker is not going to annotate the executable (and will print a warning, I believe), thus disabling BTI.
Alex Crichton (Jun 23 2022 at 18:23):hm so if I build with RUSTFLAGS='-Zbranch-protection=bti' cargo +nightly test -p wasmtime-fiber it also doesn't have anything other than the above
Alex Crichton (Jun 23 2022 at 18:24):ah I think the -Z flag may not work because even an empty main.rs doesn't get anything else
Anton Kirilov (Jun 23 2022 at 18:24):Is that the original code or the one in the PR?
Alex Crichton (Jun 23 2022 at 18:24):the original code for the test above
Anton Kirilov (Jun 23 2022 at 18:25):OK, one more thing - could you try -mmark-bti-property?
Alex Crichton (Jun 23 2022 at 18:26):I think that's probably a clang/gcc flag, right? (if so I dunno the rustc equivalent)
Anton Kirilov (Jun 23 2022 at 18:26):This flag is supposed to instruct LLVM to add the note to all object files, including the ones generated from assembly code.
Alex Crichton (Jun 23 2022 at 18:26):I can confirm the LLVM-generated object files show:
File: main.main.84aa171d-cgu.6.rcgu.o
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0
Properties: AArch64 feature: BTI
I think it is a LLVM one.
Alex Crichton (Jun 23 2022 at 18:27):I suspect that rustc is passing the right flags to LLVM, but there's probably some C dependency in libstd or something which is disabling things by accident
Anton Kirilov (Jun 23 2022 at 18:27):Yes, a single object file that lacks the note is going to "contaminate" everything.
Anton Kirilov (Jun 23 2022 at 18:28):I believe you are supposed to see a warning from LLVM.
Anton Kirilov (Jun 23 2022 at 18:29):BTW:
Anton Kirilov said:
This flag is supposed to instruct LLVM to add the note to all object files, including the ones generated from assembly code.
This is only going to add the ELF note, though, not the BTI landing pads (i.e. instructions) themselves (critical for assembly files).
Alex Crichton (Jun 23 2022 at 18:29):for sure yeah I left in all the actual instructions
Anton Kirilov (Jun 23 2022 at 18:31):I can also check with our Rust team (that is, the people who added BTI support to the Rust compiler) tomorrow - it is quite late in the UK right now.
Alex Crichton (Jun 23 2022 at 18:31):oh no worries!
Alex Crichton (Jun 23 2022 at 18:32):sorry this is not urgent at all
Anton Kirilov (Jun 23 2022 at 18:34):No problem!
Alex Crichton (Jun 23 2022 at 18:50):Some other things I've tested. First rustc +nightly empty-main.rs -Z branch-protection=bti does not have the bti note, but the reason for this is that it's linked against the standard library which is precompiled and doesn't have bti notes.
Second I tried using cargo's -Zbuild-std to recompile the standard library. For unknown symbol-related reasons this also required -Ctarget-feature=+lse. The resulting binary still didn't have the bti note.
Next I passed -mmark-bti-property via CFLAGS during the build-std building-the-standard-library step because I know there's C code in the compiler-builtins crate. That still didn't have the bti note in the final binary.
One object file on the command line which doesn't have the note section is /tmp/rustcc7t9zx/symbols.o which I think was a relatively recent addition to rustc (used to force symbols to show up in the final binary if they're even not referenced from other files). All lib.rmeta files in rlibs also don't have the bti note but they also should never be read by the linker.
Alex Crichton (Jun 23 2022 at 18:50):Turns out that was all a red herring. I found the -z force-bti option to the linker and if I pass that it helpfully prints:
/home/acrichto/root/bin/ld: /usr/lib/gcc/aarch64-linux-gnu/8/../../../aarch64-linux-gnu/Scrt1.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/gcc/aarch64-linux-gnu/8/../../../aarch64-linux-gnu/crti.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/gcc/aarch64-linux-gnu/8/crtbeginS.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/aarch64-linux-gnu/libc_nonshared.a(elf-init.oS): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/aarch64-linux-gnu/libc_nonshared.a(stat64.oS): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/aarch64-linux-gnu/libc_nonshared.a(fstat64.oS): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/aarch64-linux-gnu/libc_nonshared.a(lstat64.oS): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/aarch64-linux-gnu/libc_nonshared.a(fstatat64.oS): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/gcc/aarch64-linux-gnu/8/crtendS.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
/home/acrichto/root/bin/ld: /usr/lib/gcc/aarch64-linux-gnu/8/../../../aarch64-linux-gnu/crtn.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
so my local toolchain is at fault (somewhat predictably now that I think about it)
Alex Crichton (Jun 23 2022 at 18:52):in any case this is all moot for my original question, I can confirm that -Z branch-protection=bti sets the note even in objects with global_asm!
Anton Kirilov (Jun 24 2022 at 11:02):Thanks a lot for this investigation, I'll pass this on to our Rust team because they wanted to do wider verification of the BTI support in Rust.
Last updated: Dec 23 2024 at 13:07 UTC