wasmtime / PR #11418 Fix the QEMU command in the cross-co... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / PR #11418 Fix the QEMU command in the cross-co...

Wasmtime GitHub notifications bot (Aug 12 2025 at 08:01):

yomaytk opened PR #11418 from yomaytk:fix-docs-cross-compiling to bytecodealliance:main:

cargo test command of cross compilation target causes the following error in the many tests because the setting runner = "qemu-aarch64 -L /usr/aarch64-linux-gnu -E LD_LIBRARY_PATH=/usr/aarch64-linux-gnu/lib -E WASMTIME_TEST_NO_HOG_MEMORY=1" of .cargo/config.toml cannot pass environment variables to the qemu-user executed by binfmt.
# aarch64
aarch64-binfmt-P: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory
This PR fixes the error by specifying env QEMU_LD_PREFIX=... to the .cargo/config.toml.

Wasmtime GitHub notifications bot (Aug 12 2025 at 08:01):

yomaytk requested wasmtime-default-reviewers for a review on PR #11418.

Wasmtime GitHub notifications bot (Aug 12 2025 at 08:01):

yomaytk requested fitzgen for a review on PR #11418.

Wasmtime GitHub notifications bot (Aug 12 2025 at 16:10):

alexcrichton requested alexcrichton for a review on PR #11418.

Wasmtime GitHub notifications bot (Aug 12 2025 at 16:12):

alexcrichton commented on PR #11418:

Could you detail a bit more about the differences? I'm mostly curious because I've been using the previously-documented form for as long as I can remember. Additionally sticking to just a qemu-* binary feels nice in the sense that it doesn't also require env necessarily.

Wasmtime GitHub notifications bot (Aug 12 2025 at 16:22):

cfallin commented on PR #11418:

It appears that -L and the QEMU_LD_PREFIX environment variable should be completely equivalent -- from the output of qemu-aarch64 -h, I see
Options and associated environment variables:

Argument             Env-variable      Description
[ ... ]
-L path              QEMU_LD_PREFIX    set the elf interpreter prefix to 'path'
I'm surprised that the existing command didn't work for you. Do you have the named file (from your error message) at /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1?

If so, could you show your complete Cargo config, the command you ran, and share info about your environment (Linux distro, Rust compiler version)?

Wasmtime GitHub notifications bot (Aug 13 2025 at 05:11):

yomaytk commented on PR #11418:

@alexcrichton @cfallin Thank you for the review!

According to the error message, the path to ld-linux-aarch64.so.1 was missing /usr/aarch64-linux-gnu, so it seemed that binfmt was being launched without setting /usr/aarch64-linux-gnu as the sysroot. Therefore, I thought that during the execution of qemu-aarch64, the binfmt process launched as a other process could not receive the original -L option, leaving the sysroot empty, so I specified it via env command.

However, since the original configuration passes your tests, I’m not sure if my understanding is correct…

Do you have the named file (from your error message) at /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1?

Yes, I do.
$ ls -l /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1
-rwxr-xr-x 1 root root 202784 Apr 15  2024 /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1
If so, could you show your complete Cargo config, the command you ran, and share info about your environment (Linux distro, Rust compiler version)?

Environment is as follows:
$ uname -m; lsb_release -d
x86_64
No LSB modules are available.
Description:    Ubuntu 24.04.2 LTS
$ rustc --version
rustc 1.89.0 (29483883e 2025-08-04)
.cargo/config.toml is as follows (before the changes in this PR).
[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
runner = "qemu-aarch64 -L /usr/aarch64-linux-gnu -E LD_LIBRARY_PATH=/usr/aarch64-linux-gnu/lib -E WASMTIME_TEST_NO_HOG_MEMORY=1"
Also, I attached the Cargo.lock file (the Cargo.toml is the same as in the latest commit) and the log file from running cargo build --target aarch64-unknown-linux-gnu.

aarch64-test.log
Cargo.lock.txt

Wasmtime GitHub notifications bot (Aug 13 2025 at 05:13):

yomaytk edited a comment on PR #11418:

@alexcrichton @cfallin Thank you for the review!

According to the error message, the path to ld-linux-aarch64.so.1 was missing /usr/aarch64-linux-gnu, so it seemed that binfmt was being launched without setting /usr/aarch64-linux-gnu as the sysroot. Therefore, I thought that during the execution of qemu-aarch64, the binfmt process launched as a other process could not receive the original -L option, leaving the sysroot empty, so I specified it via env command.

However, since the original configuration passes your tests, I’m not sure if my understanding is correct…

Do you have the named file (from your error message) at /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1?

Yes, I do.
$ ls -l /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1
-rwxr-xr-x 1 root root 202784 Apr 15  2024 /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1
If so, could you show your complete Cargo config, the command you ran, and share info about your environment (Linux distro, Rust compiler version)?

Environment is as follows:
$ uname -m; lsb_release -d
x86_64
No LSB modules are available.
Description:    Ubuntu 24.04.2 LTS
$ rustc --version
rustc 1.89.0 (29483883e 2025-08-04)
.cargo/config.toml is as follows (before the changes in this PR).
[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
runner = "qemu-aarch64 -L /usr/aarch64-linux-gnu -E LD_LIBRARY_PATH=/usr/aarch64-linux-gnu/lib -E WASMTIME_TEST_NO_HOG_MEMORY=1"
Also, I attached the Cargo.lock file (the Cargo.toml is the same as in the latest commit) and the log file from running cargo test --target aarch64-unknown-linux-gnu.

aarch64-test.log
Cargo.lock.txt

Wasmtime GitHub notifications bot (Aug 13 2025 at 05:23):

cfallin commented on PR #11418:

The aarch64-binfmt-P is suspect -- that seems to imply to me that some other configuration is causing Linux's binfmt support (i.e., automatic registration of qemu to allow exec() of foreign-architecture binaries directly) to run instead.

Question: do you have any other Cargo config, perhaps in your ~/.cargo? And @alexcrichton would certainly know better than me about how Cargo finds configuration and whether there are any other gotchas here...

Wasmtime GitHub notifications bot (Aug 13 2025 at 06:34):

yomaytk commented on PR #11418:

The aarch64-binfmt-P is suspect -- that seems to imply to me that some other configuration is causing Linux's binfmt support (i.e., automatic registration of qemu to allow exec() of foreign-architecture binaries directly) to run instead.

Yes, I agree. So I disabled the aarch64 binfmt and reran the tests, but they failed in the same tests. The error details have changed, but when I set the environment variables with env and enable binfmt, all the tests pass, so it looks like I need to review the configuration related to binfmt.

The results of some tests are as follows.
$ cargo test --target aarch64-unknown-linux-gnu big_table_in_pooling_allocator
...
running 1 test
test cli_tests::big_table_in_pooling_allocator ... FAILED

failures:

---- cli_tests::big_table_in_pooling_allocator stdout ----
Error: Failed to execute wasmtime with: ["tests/all/cli_tests/big_table.wat"]
status: exit status: 127

$ cargo test --target aarch64-unknown-linux-gnu bad_text_syntax
...
running 1 test
test cli_tests::bad_text_syntax ... FAILED

failures:

---- cli_tests::bad_text_syntax stdout ----

thread 'cli_tests::bad_text_syntax' panicked at tests/all/cli_tests.rs:816:5:
bad stderr:
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Just in case, I attached the full log file.

aarch64-test-nobinfmt.log

do you have any other Cargo config, perhaps in your ~/.cargo?

No, just .cargo/config.toml.

Wasmtime GitHub notifications bot (Aug 13 2025 at 17:35):

alexcrichton commented on PR #11418:

Aha ok this makes sense now. Some tests that we have will execute the wasmtime binary itself (e.g. CLI tests). This gets tricky when the test suite has a configure runner (e.g. cargo wrapped it in qemu) because the test doesn't know that it needs to also wrap the invocation of the wasmtime binary itself. Effectively the wrapper that Cargo uses may or may not be communicated to the test itself to further execute the cross-architecture binary.

Currently the test itself detecting a runner only works if the runner is in an env var, not via Cargo configuration. This is what our CI runners configure, for example. If you're using ~/.cargo/config.toml then Cargo won't set env vars and the test won't know it's running under emulation so it won't know to use the emulation when re-executing wasmtime.

Now if you have binfmt support enabled on Linux that's why the env vars appear to fix things for you. With binfmt support the test doesn't need to know to wrap the wasmtime invocation in QEMU, the kernel does that. The reason the env vars then work is that they're propagated automatically vs with CLI flags they're not propagated automatically.

So, in essence, I think this is a confluence of events where it's less to do with Cargo and more to do with how our test suite executes the wasmtime CLI and tries to do that correctly when under emulation. Unfortunately both before/after this PR suffer from not being 100% accurate:

Before this PR the documented way of configuring a runner technically doesn't work because the env vars aren't set for the test suite to re-execute wasmtime. All parts of our test suite that don't execute commands (most of them) will still work however.

After this PR the documented way of configuring will only work if binfmt support is configured. While your PR fixes systems with binfmt support setup, those systems without it still won't wrap the wasmtime execution with qemu-* (as you've seen)

The only "real" way to fix this is to export CARGO_*_RUNNER env vars like CI does, but that's a bit of a bummer. Ideally we could detect Cargo's runner configuration and/or detect we're under QEMU and just skip the tests.

Wasmtime GitHub notifications bot (Aug 14 2025 at 11:04):

yomaytk requested wasmtime-core-reviewers for a review on PR #11418.

Wasmtime GitHub notifications bot (Aug 14 2025 at 11:04):

yomaytk updated PR #11418.

Last updated: Dec 06 2025 at 07:03 UTC