Nihal Pasham (Sep 29 2024 at 07:51): Nihal Pasham (Sep 29 2024 at 07:51):I'm attempting to translate a no_std Rust program into Cranelift IR using the rustc_codegen_cranelift backend but am encountering linker issues (undefined symbols on a m-series mac). Not sure why?
Compiling cranelift-ir-test v0.1.0 (/Users/nihal.pasham/devspace/compiler/cranelift-ir-test)
error: linking with `cc` failed: exit status: 1
|
= note: env -u IPHONEOS_DEPLOYMENT_TARGET -u TVOS_DEPLOYMENT_TARGET -u XROS_DEPLOYMENT_TARGET LC_ALL="C" PATH="/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.modular/pkg/packages.modular.com_mojo/bin:/Users/nihal.pasham/.wasmtime/bin:/opt/homebrew/opt/llvm/bin:/Users/nihal.pasham/.cargo/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/opt/make/libexec/gnubin:/Users/nihal.pasham/.local/bin:/opt/homebrew/opt/openjdk/bin:/Users/nihal.pasham/Library/xPacks/@xpack-dev-tools/arm-none-eabi-gcc/10.3.1-2.3.1/.content/bin:/opt/local/bin:/opt/local/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/nihal.pasham/.cargo/binexport" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "/var/folders/rj/5hqdh1tn6v314krprh453jd40000gn/T/rustcecz3Du/symbols.o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-86a91ad72b99b853.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcore-0d6279724923194b.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-108b1d39fdc49a06.rlib" "-o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55" "-Wl,-dead_strip" "-nodefaultlibs"
= note: ld: warning: no platform load command found in '/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o', assuming: macOS
Undefined symbols for architecture arm64:
"_main", referenced from:
<initial-undefines>
"dyld_stub_binder", referenced from:
<initial-undefines>
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: could not compile `cranelift-ir-test` (bin "cranelift-ir-test") due to 1 previous error
Nihal Pasham said:
I'm attempting to translate a
no_stdRust program into Cranelift IR using therustc_codegen_craneliftbackend but am encountering linker issues (undefined symbols on a m-series mac). Not sure why?Compiling cranelift-ir-test v0.1.0 (/Users/nihal.pasham/devspace/compiler/cranelift-ir-test) error: linking with `cc` failed: exit status: 1 | = note: env -u IPHONEOS_DEPLOYMENT_TARGET -u TVOS_DEPLOYMENT_TARGET -u XROS_DEPLOYMENT_TARGET LC_ALL="C" PATH="/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.modular/pkg/packages.modular.com_mojo/bin:/Users/nihal.pasham/.wasmtime/bin:/opt/homebrew/opt/llvm/bin:/Users/nihal.pasham/.cargo/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/opt/make/libexec/gnubin:/Users/nihal.pasham/.local/bin:/opt/homebrew/opt/openjdk/bin:/Users/nihal.pasham/Library/xPacks/@xpack-dev-tools/arm-none-eabi-gcc/10.3.1-2.3.1/.content/bin:/opt/local/bin:/opt/local/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/nihal.pasham/.cargo/binexport" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "/var/folders/rj/5hqdh1tn6v314krprh453jd40000gn/T/rustcecz3Du/symbols.o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-86a91ad72b99b853.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcore-0d6279724923194b.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-108b1d39fdc49a06.rlib" "-o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55" "-Wl,-dead_strip" "-nodefaultlibs" = note: ld: warning: no platform load command found in '/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o', assuming: macOS Undefined symbols for architecture arm64: "_main", referenced from: <initial-undefines> "dyld_stub_binder", referenced from: <initial-undefines> ld: symbol(s) not found for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) error: could not compile `cranelift-ir-test` (bin "cranelift-ir-test") due to 1 previous error
Also, is there a simpler standalone utility for translating Rust to Cranelift IR, ideally something that doesn't require writing a complete frontend parser?
Nihal Pasham (Sep 29 2024 at 12:39):For additional context: I'm simply trying to generate Cranelift IR for a pure function (i.e., very basic, without any dependencies) while avoiding the standard library or runtime.
#![no_std]
#![no_main]
#[allow(dead_code)]
fn test() {
let a = 3;
let b = 5;
let _c = a + b;
}
#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
loop {}
}
I'm personally not familiar with the details of cg-clif, and I don't think too many people that hang out here are, other than @bjorn3; I suspect you might get better support in the rust zulip, if they aren't around right now
bjorn3 (Sep 29 2024 at 19:27):The problem here is that you are trying to build an executable without linking libc. On macOS linking libc is mandatory for executables. You also haven't defined a main function for your executable.
bjorn3 (Sep 29 2024 at 19:28):If you want to get clif ir, you should pass --emit llvm-ir to get clif ir text files instead of compiling down to object files and linking.
bjorn3 (Sep 29 2024 at 19:28):Also what do you want to use the clif ir files for?
Nihal Pasham (Sep 30 2024 at 02:47):bjorn3 said:
Also what do you want to use the clif ir files for?
I'm trying to understand how Cranelift works under the hood, starting with its IR. I used the --emit=llvm-ir option along with the std-lib, which I believe produces a folder in the target directory that contains .clif IR files for the entire executable. That's a lot of output. Is there a way to generate the IR for just a single file?
I tried the following:
rustc --emit=llvm-ir src/main.rs
However, this only produces LLVM IR, not Cranelift IR.
Nihal Pasham (Sep 30 2024 at 02:56):I presume we need to force rustc to use the Cranelift backend while emitting, correct?
bjorn3 (Sep 30 2024 at 17:51):For each codegend function three files are emitted. The .unopt.clif file contains what cg_clif emits. The .opt.clif file contains the result of optimizing by Cranelift and the .vcode file contains the ir right before cranelift emits machine code bytes.
bjorn3 (Sep 30 2024 at 17:52):Nihal Pasham said:
I presume we need to force
rustcto use the Cranelift backend while emitting, correct?
If you want to avoid cargo, you can use rustc-clif instead of cargo-clif.
bjorn3 (Sep 30 2024 at 17:53):This will still emit clif ir for standard library functions if rustc decides to codegen them as part of the local crate however. There is no way to filter for just local functions.
Jubilee (Oct 01 2024 at 09:24):speaking of, is there a way to dump that clif to a text format?
Jubilee (Oct 01 2024 at 09:26):surely --emit=clif-ir will not Simply Work
Nihal Pasham (Oct 01 2024 at 11:41):I believe
bjorn3 said:
This will still emit clif ir for standard library functions if rustc decides to codegen them as part of the local crate however. There is no way to filter for just local functions.
Got it .
My goal is to document my learning journey with Cranelift, potentially through a series of videos for future reference. I'm focusing on understanding the process from IR generation for a simple program to how Cranelift ultimately produces machine code. The steps I’m aiming to cover are:
CLIF -> legalization -> mid-end egraph rewrites (if enabled, using ISLE rules) -> lowering to backend-specific VCode (using ISLE rules) -> regalloc -> binary emission
I believe I've gained a basic understanding of egraphs and ISLE, but I still don’t feel confident enough to walk through an entire example covering all the stages. Is there a way to dynamically step through each of these phases at runtime to observe how they work? Any suggestions or guidance would be greatly appreciated.
Nihal Pasham (Oct 01 2024 at 12:09):Jubilee said:
speaking of, is there a way to dump that clif to a text format?
To dump the IR for the entire program, I used the following command:
../rustc_codegen_cranelift/dist/rustc-clif --emit=llvm-ir src/main.rs
This creates a directory named main.clif that contains all the artifacts.
bjorn3 (Oct 01 2024 at 13:54):Is there a way to dynamically step through each of these phases at runtime to observe how they work?
Cranelift emits logs which contain the clif ir at each step, but cg_clif doesn't currently expose a way to enable logs.
Jubilee (Oct 01 2024 at 17:26):@Nihal Pasham I must warn you that if you try to inspect the details of how the compiler works at runtime and document based on that, instead of from the external boundaries of the program, your documentation will age much more rapidly.
Nihal Pasham (Oct 03 2024 at 02:57):I believe I've found what I was looking for. Since functions in Cranelift are compiled individually, I can simply add the #[no_mangle] attribute to the function I'm interested in (i.e., the one for which we want to see the generated IR). This approach produces the three different versions (.opt, .unopt, .vcode). While this isn’t exactly a straightforward "Rust function in, IR out," it does make it easier to locate the IR for a specific function.
Another observation: I noticed that the IR generated by cg_clif differs slightly from what's in Cranelift's IR reference documentation.
For example, the function signature syntax produced by cg_clif looks like this:
function u0:9(i8, i8) -> i8 apple_aarch64
whereas Cranelift IR uses a % like so:
function %average(i32, i32) -> f32 system_v
Just to confirm—does cg_clif produce a more specialized IR tailored for the Rust compiler's use?
fitzgen (he/him) (Oct 03 2024 at 03:12):those are just two different ways to name functions, either the ith namespace and jth function within it, or some user-provided string
Nihal Pasham (Oct 03 2024 at 03:32):So, it seems like u0 is the identifier for the "compilation unit," and 9 represents the function's unique identifier within that unit—correct?
Also, I noticed this in one of the blocks:
block0(v0: i8, v1: i8):
v2 -> v0
v5 -> v0
v11 -> v0
v3 -> v1
v6 -> v1
v12 -> v1
nop
; write_cvalue: Var(_1, var1): u8 <- ByVal(v0): u8
; write_cvalue: Var(_2, var2): u8 <- ByVal(v1): u8
jump block1
Are these simply assignments, or should I interpret them differently?
bjorn3 (Oct 03 2024 at 06:47):cranelift-module uses u0:X to refer to function X within the current module (can be an import) and u1:Y to refer to data object Y.
bjorn3 (Oct 03 2024 at 06:49):The v2 -> v0 are aliases. So in this case all references to v2 are implicitly replaced with v0 during compilation. They mostly exist to make the SSA lowering in cranelift-frontend easier.
Amanieu (Oct 03 2024 at 12:18):@bjorn3 --emit llvm-ir doesn't seem to work with -O to get clif IR with optimzations enabled. It just doesn't create the directory.
bjorn3 (Oct 03 2024 at 12:19):It works fine for me.
bjorn3 (Oct 03 2024 at 12:20):$ dist/rustc-clif -O - --emit llvm-ir
fn main() {}
^D
$ ls -l | grep rust_out
-rw-rw-r-- 1 gh-bjorn3 gh-bjorn3 2264 Oct 3 12:20 rust_out.allocator_shim.rcgu.o
drwxrwxr-x 2 gh-bjorn3 gh-bjorn3 4096 Oct 3 12:19 rust_out.clif
Hmm maybe it's because my nightly is a month old
Amanieu (Oct 03 2024 at 12:24):I'm using rustc-codegen-cranelift-preview
Amanieu (Oct 03 2024 at 12:24):from rustup
bjorn3 (Oct 03 2024 at 12:25):Should have worked fine back for the last couple of years.
bjorn3 (Oct 03 2024 at 12:25):Are you passing any -o or --out-dir?
Amanieu (Oct 03 2024 at 12:25):Here's the command I'm using:
rustc test.rs --crate-type rlib -Zcodegen-backend=cranelift --emit llvm-ir
If I add -O then no test.clif directory is created.
Amanieu (Oct 03 2024 at 12:26):Same with -Copt-level with any value except 0.
bjorn3 (Oct 03 2024 at 12:26):That is weird...
Amanieu (Oct 03 2024 at 12:27):Hmm maybe the whole function is optimized away?
Amanieu (Oct 03 2024 at 12:27):Oh yea that was it.
Amanieu (Oct 03 2024 at 12:27):#[no_mangle] fixed it
bjorn3 (Oct 03 2024 at 12:27):That explains it.
Amanieu (Oct 03 2024 at 12:28):It still has set opt_level=none even with -O, is that intended?
bjorn3 (Oct 03 2024 at 12:30):-O is equivalent to -Copt-level=2, which is internally called OptLevel::Default for whatever reason. For this and -Copt-level=1 I don't set the opt_level flag at all, so Cranelift defaults to opt_level=none. Will fix in a bit. In any case -Copt-level=3 should work for now.
bjorn3 (Oct 03 2024 at 12:36):Pushed a fix
Last updated: Nov 22 2024 at 17:03 UTC