Stream: cranelift

Topic: Translating Rust code to Cranelift IR


view this post on Zulip Nihal Pasham (Sep 29 2024 at 07:51):

I'm attempting to translate a no_std Rust program into Cranelift IR using the rustc_codegen_cranelift backend but am encountering linker issues (undefined symbols on a m-series mac). Not sure why?

   Compiling cranelift-ir-test v0.1.0 (/Users/nihal.pasham/devspace/compiler/cranelift-ir-test)
error: linking with `cc` failed: exit status: 1
  |
  = note: env -u IPHONEOS_DEPLOYMENT_TARGET -u TVOS_DEPLOYMENT_TARGET -u XROS_DEPLOYMENT_TARGET LC_ALL="C" PATH="/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.modular/pkg/packages.modular.com_mojo/bin:/Users/nihal.pasham/.wasmtime/bin:/opt/homebrew/opt/llvm/bin:/Users/nihal.pasham/.cargo/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/opt/make/libexec/gnubin:/Users/nihal.pasham/.local/bin:/opt/homebrew/opt/openjdk/bin:/Users/nihal.pasham/Library/xPacks/@xpack-dev-tools/arm-none-eabi-gcc/10.3.1-2.3.1/.content/bin:/opt/local/bin:/opt/local/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/nihal.pasham/.cargo/binexport" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "/var/folders/rj/5hqdh1tn6v314krprh453jd40000gn/T/rustcecz3Du/symbols.o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-86a91ad72b99b853.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcore-0d6279724923194b.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-108b1d39fdc49a06.rlib" "-o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55" "-Wl,-dead_strip" "-nodefaultlibs"
  = note: ld: warning: no platform load command found in '/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o', assuming: macOS
          Undefined symbols for architecture arm64:
            "_main", referenced from:
                <initial-undefines>
            "dyld_stub_binder", referenced from:
                <initial-undefines>
          ld: symbol(s) not found for architecture arm64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: could not compile `cranelift-ir-test` (bin "cranelift-ir-test") due to 1 previous error

view this post on Zulip Nihal Pasham (Sep 29 2024 at 07:54):

Nihal Pasham said:

I'm attempting to translate a no_std Rust program into Cranelift IR using the rustc_codegen_cranelift backend but am encountering linker issues (undefined symbols on a m-series mac). Not sure why?

   Compiling cranelift-ir-test v0.1.0 (/Users/nihal.pasham/devspace/compiler/cranelift-ir-test)
error: linking with `cc` failed: exit status: 1
  |
  = note: env -u IPHONEOS_DEPLOYMENT_TARGET -u TVOS_DEPLOYMENT_TARGET -u XROS_DEPLOYMENT_TARGET LC_ALL="C" PATH="/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/nihal.pasham/.modular/pkg/packages.modular.com_mojo/bin:/Users/nihal.pasham/.wasmtime/bin:/opt/homebrew/opt/llvm/bin:/Users/nihal.pasham/.cargo/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/opt/make/libexec/gnubin:/Users/nihal.pasham/.local/bin:/opt/homebrew/opt/openjdk/bin:/Users/nihal.pasham/Library/xPacks/@xpack-dev-tools/arm-none-eabi-gcc/10.3.1-2.3.1/.content/bin:/opt/local/bin:/opt/local/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/nihal.pasham/.cargo/binexport" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "/var/folders/rj/5hqdh1tn6v314krprh453jd40000gn/T/rustcecz3Du/symbols.o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-86a91ad72b99b853.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcore-0d6279724923194b.rlib" "/Users/nihal.pasham/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-108b1d39fdc49a06.rlib" "-o" "/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55" "-Wl,-dead_strip" "-nodefaultlibs"
  = note: ld: warning: no platform load command found in '/Users/nihal.pasham/devspace/compiler/cranelift-ir-test/target/debug/deps/cranelift_ir_test-06cd2efd590c2c55.9zu9dm6yolguzepxzrn2yw7xs.rcgu.o', assuming: macOS
          Undefined symbols for architecture arm64:
            "_main", referenced from:
                <initial-undefines>
            "dyld_stub_binder", referenced from:
                <initial-undefines>
          ld: symbol(s) not found for architecture arm64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: could not compile `cranelift-ir-test` (bin "cranelift-ir-test") due to 1 previous error

Also, is there a simpler standalone utility for translating Rust to Cranelift IR, ideally something that doesn't require writing a complete frontend parser?

view this post on Zulip Nihal Pasham (Sep 29 2024 at 12:39):

For additional context: I'm simply trying to generate Cranelift IR for a pure function (i.e., very basic, without any dependencies) while avoiding the standard library or runtime.

#![no_std]
#![no_main]

#[allow(dead_code)]
fn test() {
    let a = 3;
    let b = 5;
    let _c = a + b;
}

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
    loop {}
}

view this post on Zulip fitzgen (he/him) (Sep 29 2024 at 17:18):

I'm personally not familiar with the details of cg-clif, and I don't think too many people that hang out here are, other than @bjorn3; I suspect you might get better support in the rust zulip, if they aren't around right now

view this post on Zulip bjorn3 (Sep 29 2024 at 19:27):

The problem here is that you are trying to build an executable without linking libc. On macOS linking libc is mandatory for executables. You also haven't defined a main function for your executable.

view this post on Zulip bjorn3 (Sep 29 2024 at 19:28):

If you want to get clif ir, you should pass --emit llvm-ir to get clif ir text files instead of compiling down to object files and linking.

view this post on Zulip bjorn3 (Sep 29 2024 at 19:28):

Also what do you want to use the clif ir files for?

view this post on Zulip Nihal Pasham (Sep 30 2024 at 02:47):

bjorn3 said:

Also what do you want to use the clif ir files for?

I'm trying to understand how Cranelift works under the hood, starting with its IR. I used the --emit=llvm-ir option along with the std-lib, which I believe produces a folder in the target directory that contains .clif IR files for the entire executable. That's a lot of output. Is there a way to generate the IR for just a single file?

I tried the following:

rustc --emit=llvm-ir src/main.rs

However, this only produces LLVM IR, not Cranelift IR.

view this post on Zulip Nihal Pasham (Sep 30 2024 at 02:56):

I presume we need to force rustc to use the Cranelift backend while emitting, correct?

view this post on Zulip bjorn3 (Sep 30 2024 at 17:51):

For each codegend function three files are emitted. The .unopt.clif file contains what cg_clif emits. The .opt.clif file contains the result of optimizing by Cranelift and the .vcode file contains the ir right before cranelift emits machine code bytes.

view this post on Zulip bjorn3 (Sep 30 2024 at 17:52):

Nihal Pasham said:

I presume we need to force rustc to use the Cranelift backend while emitting, correct?

If you want to avoid cargo, you can use rustc-clif instead of cargo-clif.

view this post on Zulip bjorn3 (Sep 30 2024 at 17:53):

This will still emit clif ir for standard library functions if rustc decides to codegen them as part of the local crate however. There is no way to filter for just local functions.

view this post on Zulip Jubilee (Oct 01 2024 at 09:24):

speaking of, is there a way to dump that clif to a text format?

view this post on Zulip Jubilee (Oct 01 2024 at 09:26):

surely --emit=clif-ir will not Simply Work

view this post on Zulip Nihal Pasham (Oct 01 2024 at 11:41):

I believe

bjorn3 said:

This will still emit clif ir for standard library functions if rustc decides to codegen them as part of the local crate however. There is no way to filter for just local functions.

Got it .

My goal is to document my learning journey with Cranelift, potentially through a series of videos for future reference. I'm focusing on understanding the process from IR generation for a simple program to how Cranelift ultimately produces machine code. The steps I’m aiming to cover are:

CLIF -> legalization -> mid-end egraph rewrites (if enabled, using ISLE rules) -> lowering to backend-specific VCode (using ISLE rules) -> regalloc -> binary emission

I believe I've gained a basic understanding of egraphs and ISLE, but I still don’t feel confident enough to walk through an entire example covering all the stages. Is there a way to dynamically step through each of these phases at runtime to observe how they work? Any suggestions or guidance would be greatly appreciated.

view this post on Zulip Nihal Pasham (Oct 01 2024 at 12:09):

Jubilee said:

speaking of, is there a way to dump that clif to a text format?

To dump the IR for the entire program, I used the following command:

../rustc_codegen_cranelift/dist/rustc-clif --emit=llvm-ir src/main.rs

This creates a directory named main.clif that contains all the artifacts.

view this post on Zulip bjorn3 (Oct 01 2024 at 13:54):

Is there a way to dynamically step through each of these phases at runtime to observe how they work?

Cranelift emits logs which contain the clif ir at each step, but cg_clif doesn't currently expose a way to enable logs.

view this post on Zulip Jubilee (Oct 01 2024 at 17:26):

@Nihal Pasham I must warn you that if you try to inspect the details of how the compiler works at runtime and document based on that, instead of from the external boundaries of the program, your documentation will age much more rapidly.

view this post on Zulip Nihal Pasham (Oct 03 2024 at 02:57):

I believe I've found what I was looking for. Since functions in Cranelift are compiled individually, I can simply add the #[no_mangle] attribute to the function I'm interested in (i.e., the one for which we want to see the generated IR). This approach produces the three different versions (.opt, .unopt, .vcode). While this isn’t exactly a straightforward "Rust function in, IR out," it does make it easier to locate the IR for a specific function.

Another observation: I noticed that the IR generated by cg_clif differs slightly from what's in Cranelift's IR reference documentation.

For example, the function signature syntax produced by cg_clif looks like this:

function u0:9(i8, i8) -> i8 apple_aarch64

whereas Cranelift IR uses a % like so:

function %average(i32, i32) -> f32 system_v

Just to confirm—does cg_clif produce a more specialized IR tailored for the Rust compiler's use?

view this post on Zulip fitzgen (he/him) (Oct 03 2024 at 03:12):

those are just two different ways to name functions, either the ith namespace and jth function within it, or some user-provided string

view this post on Zulip Nihal Pasham (Oct 03 2024 at 03:32):

So, it seems like u0 is the identifier for the "compilation unit," and 9 represents the function's unique identifier within that unit—correct?

Also, I noticed this in one of the blocks:

block0(v0: i8, v1: i8):
    v2 -> v0
    v5 -> v0
    v11 -> v0
    v3 -> v1
    v6 -> v1
    v12 -> v1
    nop
; write_cvalue: Var(_1, var1): u8 <- ByVal(v0): u8
; write_cvalue: Var(_2, var2): u8 <- ByVal(v1): u8
    jump block1

Are these simply assignments, or should I interpret them differently?

view this post on Zulip bjorn3 (Oct 03 2024 at 06:47):

cranelift-module uses u0:X to refer to function X within the current module (can be an import) and u1:Y to refer to data object Y.

view this post on Zulip bjorn3 (Oct 03 2024 at 06:49):

The v2 -> v0 are aliases. So in this case all references to v2 are implicitly replaced with v0 during compilation. They mostly exist to make the SSA lowering in cranelift-frontend easier.

view this post on Zulip Amanieu (Oct 03 2024 at 12:18):

@bjorn3 --emit llvm-ir doesn't seem to work with -O to get clif IR with optimzations enabled. It just doesn't create the directory.

view this post on Zulip bjorn3 (Oct 03 2024 at 12:19):

It works fine for me.

view this post on Zulip bjorn3 (Oct 03 2024 at 12:20):

$ dist/rustc-clif -O - --emit llvm-ir
fn main() {}
^D
$ ls -l | grep rust_out
-rw-rw-r--  1 gh-bjorn3 gh-bjorn3  2264 Oct  3 12:20 rust_out.allocator_shim.rcgu.o
drwxrwxr-x  2 gh-bjorn3 gh-bjorn3  4096 Oct  3 12:19 rust_out.clif

view this post on Zulip Amanieu (Oct 03 2024 at 12:24):

Hmm maybe it's because my nightly is a month old

view this post on Zulip Amanieu (Oct 03 2024 at 12:24):

I'm using rustc-codegen-cranelift-preview

view this post on Zulip Amanieu (Oct 03 2024 at 12:24):

from rustup

view this post on Zulip bjorn3 (Oct 03 2024 at 12:25):

Should have worked fine back for the last couple of years.

view this post on Zulip bjorn3 (Oct 03 2024 at 12:25):

Are you passing any -o or --out-dir?

view this post on Zulip Amanieu (Oct 03 2024 at 12:25):

Here's the command I'm using:

rustc test.rs --crate-type rlib -Zcodegen-backend=cranelift --emit llvm-ir

view this post on Zulip Amanieu (Oct 03 2024 at 12:26):

If I add -O then no test.clif directory is created.

view this post on Zulip Amanieu (Oct 03 2024 at 12:26):

Same with -Copt-level with any value except 0.

view this post on Zulip bjorn3 (Oct 03 2024 at 12:26):

That is weird...

view this post on Zulip Amanieu (Oct 03 2024 at 12:27):

Hmm maybe the whole function is optimized away?

view this post on Zulip Amanieu (Oct 03 2024 at 12:27):

Oh yea that was it.

view this post on Zulip Amanieu (Oct 03 2024 at 12:27):

#[no_mangle] fixed it

view this post on Zulip bjorn3 (Oct 03 2024 at 12:27):

That explains it.

view this post on Zulip Amanieu (Oct 03 2024 at 12:28):

It still has set opt_level=none even with -O, is that intended?

view this post on Zulip bjorn3 (Oct 03 2024 at 12:30):

-O is equivalent to -Copt-level=2, which is internally called OptLevel::Default for whatever reason. For this and -Copt-level=1 I don't set the opt_level flag at all, so Cranelift defaults to opt_level=none. Will fix in a bit. In any case -Copt-level=3 should work for now.

view this post on Zulip bjorn3 (Oct 03 2024 at 12:36):

Pushed a fix


Last updated: Dec 23 2024 at 12:05 UTC