Stream: cranelift

Topic: `wasmtime compile` for `riscv64-ima` target

Andrew Borg (aborg-dev) (Nov 21 2024 at 08:47):

Hey folks!

I've been experimenting with wasmtime compile to AOT-compile WASM modules for the RISC-V target.

Specifically, I'm interested in riscv64-ima (note no "C" extension for compressed instructions).

I was able to successfully compile code with:

cargo run --bin wasmtime --features all-arch -- compile --target riscv64 fibonacci.wasm

Unfortunately it does contain compressed (16 bit) instructions in some functions, e.g. ones related to type_id: gist.

I tried some hackery to disable the compressed instructions on the backend level by removing try_emit_compressed in riscv64/inst/emit.rs but that didn't have any effect on the generated bytecode for the functions above.

Is there any hope I can achieve this with low effort (I'm contemplating an alternative of implementing +C extension in our RISC-V interpreter which will probably take a week or two)? I'm fine with hacky one-off solutions as only need this to do some prototype measurements.

Thanks!

Afonso Bordado (Nov 21 2024 at 08:48):

Hey, by default we target riscv64gc, but you should be able to disable compressed extensions by specifying the has_c=false flag

Afonso Bordado (Nov 21 2024 at 08:49):

I don't know exactly the wasmtime flag for that, I rememeber it used to be something along the lines of --cranelift-flag ..., but not quite sure

Afonso Bordado (Nov 21 2024 at 08:55):

Okay, looking at the code you should be using -Ccranelift-has_zca=false -Ccranelift-has_zcd=false. We do have has_c which is defined as a combination of those two flags, but for some reason the wasmtime cli isn't recognizing that

Andrew Borg (aborg-dev) (Nov 21 2024 at 09:48):

@Afonso Bordado , thank you! I was trying to tweak these flags in the code, bug passing them through CLI is much nicer.

As it stands though, I don't see any effect when disabling them, they seemed to be off already. Specifically, here is the experiment that I've tried:

1. Patch this code https://github.com/aborg-dev/wasmtime/commit/0c1c2f612bb73a5c56eae98fee89e05f8f18f4a0 to make sure we are not generating compressed instructions in the riscv64 backend.
2. Compile without has_* flags:

cargo run --bin wasmtime --features all-arch -- compile --target riscv64 data/fibonacci.wasm
md5sum fibonacci.cwasm
> dd5a06834687e5af0c8f7d01e5cef81c  fibonacci.cwasm

The panic in the code did not fire.

3. Compile with has_* flags:

rm fibonacci.cwasm
cargo run --bin wasmtime --features all-arch -- compile -Ccranelift-has_zca=false -Ccranelift-has_zcd=false --target riscv64 data/fibonacci.wasm
md5sum fibonacci.cwasm
> dd5a06834687e5af0c8f7d01e5cef81c  fibonacci.cwasm

Again, the same checksum and the panic didn't fire

4. Compile for imac target

rm fibonacci.cwasm
cargo run --bin wasmtime --features all-arch -- compile --target riscv64imac data/fibonacci.wasm
md5sum fibonacci.cwasm
38e8a91b709a5ae8f2fdfc4dee91fb9d  fibonacci.cwasm

The checksum changed (I suspect just because the target is encoded in the ELF file), the panic didn't fire

5. Finally, manually enable has_* flags:

cargo run --bin wasmtime --features all-arch -- compile  -Ccranelift-has_zca=true -Ccranelift-has_zcd=true --target riscv64 data/fibonacci.wasm

Now the panic actually triggers.

I've included the data/fibonacci.wasm in the commit if anyone wants to try to reproduce this. It has been generated using rustc.

Overall, it seems that these compressed instructions are generated somewhere outside of riscv64 codegen? I've tracked their source to array_to_wasm_trampoline function, but couldn't find anything suspicious there.

Afonso Bordado (Nov 21 2024 at 10:09):

Oh right, if those functions are generated in the trampoline then we don't compile them using cranelift. I think they get compiled at wasmtime compile time and then copied to the final binary. But @Alex Crichton might be able to explain this better.

Alex Crichton (Nov 21 2024 at 15:13):

From the gist in the OP of this thread, I don't think those are actually compressed instructions. I think that's data after the final ret which happens to disassemble as compressed instructions but I don't believe they're actually executed at runtime (or at least they shouldn't be).

Otherwise I think this all makes sense:

• I don't think we actually enable the compressed extension by default?
• That would explain why --target riscv64 doesn't trigger the panic
• If --target riscv64 doesn't trigger the panic then manually disabling extensions also won't change much
• With --target riscv64imac that's parsed by target-lexicon and we don't actually infer target features from that, just the target architecture. Arguably a bug in Wasmtime but that won't actually enable the c extension
• With -Ccranelift-has_zca=true -Ccranelift-has_zcd=true that's the first time things were enabled, so that would explain things.

Overall, it seems that these compressed instructions are generated somewhere outside of riscv64 codegen? I've tracked their source to array_to_wasm_trampoline function, but couldn't find anything suspicious there.

I'm not sure I fully understand this? When you say you've tracked it down to there, is this in a debugger for example (e.g. you ran something and got SIGILL?) Or are you inspecting codegen manually?

I think they get compiled at wasmtime compile time and then copied to the final binary

Oh I don't believe that this should be happening, all compiled code is generated by Cranelift

Afonso Bordado (Nov 21 2024 at 15:21):

Oh right, I thought we had some pieces that were compiled outside of cranelift.

IIRC, by default we use riscv64gc (even if the target is riscv64) that has compressed instructions enabled by default

The extra data being interpreted as instructions seems very likely, I've run into that a lot with our internal disassembly tests as well

Alex Crichton (Nov 21 2024 at 15:22):

I thought as well we had c enabled by default, but given that --target isn't emitting compressed instructions I think that we don't? (perhaps due to a bug?)

Afonso Bordado (Nov 21 2024 at 15:24):

Yeah that's strange, I'll try to look into it

Andrew Borg (aborg-dev) (Nov 22 2024 at 13:55):

@Alex Crichton , ahhh, very good catch, that explains a lot!

For a bit more context, I'm prototyping an ahead-of-time binary translator from RISC-V bytecode into a custom ISA and it sounds like I need to disambiguate between the actual bytecode and some sort of "payload" that Cranelift codegen stores after the function bytecode? What is this payload? I tried to look at the disasm/vcode of the function, but it ends on the ret statement and actually does not include these additional few bytes that I see in the objdump.

Assuming my understanding is correct, I see two ways to accomplish this:

1. Have a way to find the boundary between the bytecode and Cranelift "payload", e.g. a length of functions stored somewhere, or some special market at the end of a function. Can I rely on "ret" instruction for this?
2. Have a way to disable the "payload" generation - but I haven't found the place in code where it is added

Alex Crichton (Nov 22 2024 at 17:38):

We have an issue about moving these data bits out of the .text section but otherwise AFAIK there's no great way to detect this right now with our current object files

Alex Crichton (Nov 22 2024 at 17:38):

we could perhaps update our object files with new symbols or such to help outline these sections though

Last updated: Dec 23 2024 at 12:05 UTC