wasmtime / issue #12778 `Illegal instruction` on arm64 An... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / issue #12778 `Illegal instruction` on arm64 An...

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:11):

Test Case

Use the original bytecodealliance/sample-wasi-http-rust's /echo API is enough to reproduce.

Environment

Android arm64 device or emulator.
Latest release of wasmtime-v42.0.1-aarch64-android.tar.xz and bytecodealliance/sample-wasi-http-rust.

Steps to Reproduce
adb shell mkdir -p /data/local/tmp
adb push wasmtime sample-wasi-http-rust.wasm /data/local/tmp
adb forward tcp:8080 tcp:8080
adb shell HOME=/data/local/tmp \
  /data/local/tmp/wasmtime serve \
  -Scli -Shttp \
  /data/local/tmp/sample-wasi-http-rust.wasm
Open a new terminal:
curl -d "$(printf 'data%.0s' {1..5000})"  http://127.0.0.1:8080/echo > /dev/null
output:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 28038    0  8038  100 20000   197k   491k --:--:-- --:--:-- --:--:--  702k
curl: (18) transfer closed with outstanding read data remaining
And the wasmtime process crashed and showed:
Serving HTTP on http://0.0.0.0:8080/
Illegal instruction
https://github.com/user-attachments/assets/d86ec46d-ccdf-4421-a879-21fb0797e762

Also tried on Termux, same results.

Expected Results

wasmtime process should not exit

Actual Results

Illegal instruction

Versions and Environment

Wasmtime version or commit: wasmtime-v42.0.1-aarch64-android

Operating system: Android

Architecture: arm64

Extra Info

Maybe related to #11295

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:11):

Heap-Hop added the bug label to Issue #12778.

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:14):

Heap-Hop edited issue #12778:

Test Case

Use the original bytecodealliance/sample-wasi-http-rust's /echo API is enough to reproduce.

Environment

Android arm64 device or emulator.
Latest release of wasmtime-v42.0.1-aarch64-android.tar.xz and bytecodealliance/sample-wasi-http-rust.

Steps to Reproduce
adb shell mkdir -p /data/local/tmp
adb push wasmtime sample-wasi-http-rust.wasm /data/local/tmp
adb forward tcp:8080 tcp:8080
adb shell HOME=/data/local/tmp \
  /data/local/tmp/wasmtime serve \
  -Scli -Shttp \
  /data/local/tmp/sample-wasi-http-rust.wasm
Open a new terminal:
curl -d "$(printf 'data%.0s' {1..5000})"  http://127.0.0.1:8080/echo > /dev/null
output:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 28038    0  8038  100 20000   197k   491k --:--:-- --:--:-- --:--:--  702k
curl: (18) transfer closed with outstanding read data remaining
And the wasmtime process crashed and showed:
Serving HTTP on http://0.0.0.0:8080/
Illegal instruction
https://github.com/user-attachments/assets/d86ec46d-ccdf-4421-a879-21fb0797e762

Also tried on Termux, same results.

Expected Results

wasmtime process should not exit

Actual Results

Illegal instruction

Versions and Environment

Wasmtime version or commit: wasmtime-v42.0.1-aarch64-android

Operating system: Android 16

Architecture: arm64

Extra Info

Maybe related to #11295

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:29):

Heap-Hop commented on issue #12778:

lldb:

(lldb) c
Process 8704 resuming
Process 8704 stopped
* thread #2, name = 'tokio-runtime-w', stop reason = signal SIGILL: illegal operand
    frame #0: 0x00000059d5fa3c30 wasmtime`wasmtime_internal_fiber::stackswitch::aarch64::wasmtime_fiber_switch_::h5b6d4250d8a8bee0 + 100
wasmtime`wasmtime_internal_fiber::stackswitch::aarch64::wasmtime_fiber_switch_::h5b6d4250d8a8bee0:
->  0x59d5fa3c30 <+100>: autiasp
    0x59d5fa3c34 <+104>: ret

wasmtime`wasmtime_internal_fiber::FiberStack::from_custom::h1d78d71118b22fbd:
    0x59d5fa3c38 <+0>:   sub    sp, sp, #0x50
    0x59d5fa3c3c <+4>:   stp    x29, x30, [sp, #0x30]
Target 0: (wasmtime) stopped.

Wasmtime GitHub notifications bot (Mar 14 2026 at 12:56):

Heap-Hop commented on issue #12778:

Just tried through Copilot, GPT-5.4.
https://github.com/Heap-Hop/wasmtime/commit/c5245c1960cf4e712e3f9dde838b2898ee30011b

Using the artifact bins-aarch64-android from Artifacts.
It works fine now.

This is the summary by AI:
(I'm not an expert of this area, please review these changes)

Pointer Authentication (PAuth) and Android SIGILL crash

Background

Wasmtime uses a small assembly stub to perform stack switching for lightweight fibers.
On AArch64 this stub historically emitted the ARM Pointer Authentication (PAuth) instructions
paciasp / autiasp to protect return addresses as a form of control-flow integrity.

On many Android arm64 devices the CPU does not support the Pointer Authentication extension.
In that case, executing autiasp causes a fatal SIGILL (Illegal Instruction) and crashes the Wasmtime process.

This is the issue reported in:

https://github.com/bytecodealliance/wasmtime/issues/12778

https://github.com/bytecodealliance/wasmtime/issues/11295

What changed

The fiber stack-switch implementation in crates/fiber/src/stackswitch/aarch64.rs
now **only emits the PAuth instructions if the Rust compiler is configured with one of the
PAuth target features**:

+paca (Armv8.3-A Pointer Authentication)

+pacg (Armv8.3-A Pointer Authentication)

When neither feature is enabled (the default for aarch64-linux-android), the generated
code does not include paciasp / autiasp, avoiding the SIGILL crash.

Security implications

:check: Safe for devices without PAuth support

If the CPU does not support PAuth, the CPU cannot execute the instructions anyway.
This change prevents Wasmtime from crashing on such devices, which is a correctness fix.

:secure: CFI is preserved when PAuth is enabled

If a build enables -C target-feature=+paca / +pacg, Wasmtime will still emit the
PAuth instructions and gain the same control-flow integrity protection as before.

In other words:

No PAuth target feature → No PAuth instructions → No crash (but no CFI)

PAuth target feature enabled → PAuth instructions emitted → CFI is intact

Relevant files

crates/fiber/src/stackswitch/aarch64.rs (the assembly stub that was modified)

Wasmtime GitHub notifications bot (Mar 14 2026 at 13:14):

Heap-Hop commented on issue #12778:

https://github.com/Heap-Hop/wasmtime/commit/c5245c1960cf4e712e3f9dde838b2898ee30011b causes a new failure:
https://github.com/Heap-Hop/wasmtime/actions/runs/23087440874/job/67066363560#step:22:849
This might not be a complete fix yet.

Wasmtime GitHub notifications bot (Mar 15 2026 at 10:42):

Heap-Hop commented on issue #12778:

https://github.com/Heap-Hop/wasmtime/actions/runs/23099865150
CI passed and it works fine on Android.

The changes were suggested by AI, and I'm not very familiar with this area, so they definitely need review.

Should I open a PR for this, or would someone else like to start one based on this attempt?

Wasmtime GitHub notifications bot (Mar 15 2026 at 17:54):

cfallin commented on issue #12778:

Thanks for filing the issue. Looking at the diff you liked, the fundamental bug appears to be that we don't conditionalize use of pointer-authentication instructions on target features, as you say. Cranelift-compiled code will work fine because we have cranelift-native that does feature detection; but the fiber-switch routines are hardcoded.

That said, the diff has kind of made a mess of the code -- it removes the cfg_if entirely and messes up the logic. Could you (human, not LLM) refactor the patch so that it retains the cfg_if, and splits the else branch into else if (pac features) { existing non-Apple defs } else { empty defs }? I think we'd be happy to review a patch with that. Thanks!

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:36):

alexcrichton commented on issue #12778:

This might also be something we need an expert/someone knowledgable about Android to weigh-in. My impression was that the pointer authentication instructions are noops on CPUs that don't support them, which means that I don't know what the issue is here. What pointer authentication is expected to happen on Android?

Basically I'm not sure why we'd want a duplicate path without pointer authentication. I'd expect the pointer-authentication bits to be self-contained in this file where if the external systems uses them then it respects the ABI, but if the external system has pointer authentication disabled then this would be a bunch of noops.

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:36):

alexcrichton added the wasmtime:platform-support label to Issue #12778.

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:41):

cfallin commented on issue #12778:

Basically I'm not sure why we'd want a duplicate path without pointer authentication. I'd expect the pointer-authentication bits to be self-contained in this file where if the external systems uses them then it respects the ABI, but if the external system has pointer authentication disabled then this would be a bunch of noops.

My understanding is that it's a little more subtle than that: the instructions may be present on the CPU (it's part of ARMv8.3+) but the OS might not configure the secret keys, etc. That configuration is part of the target triple aka Rust target features. So we really do need the handwritten assembly to follow the same conventions that rustc itself follows in codegen'ing the runtime (for example), which means we conditionally emit the instructions.

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:44):

cfallin commented on issue #12778:

(To confirm why Cranelift-generated code gets this right, in cranelift-native we have a check for the target feature paca which configures use of the feature.)

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:45):

cfallin commented on issue #12778:

(Or, actually, I'm not sure if that tests the CPU-level feature or a feature vector provided by the OS; I don't have an Android device handy to test. In any case it seems like having our inline assembly follow the same prologue/epilogue conventions as the rest of the runtime seems sensible.)

Wasmtime GitHub notifications bot (Mar 16 2026 at 16:29):

alexcrichton commented on issue #12778:

Do you know where exactly the illegal instruction here is coming from? Can these instructions trap if the OS has them disabled? Are we using the wrong key since it was detected wrong? Are we in a situation where the fiber code is doing pointer authentication, the rest of the system isn't, and that causes issues?

Wasmtime GitHub notifications bot (Mar 16 2026 at 16:58):

cfallin commented on issue #12778:

In #11295 there's a debugger trace showing the fault on autiasp (authenticate return address before return). These instructions can certainly trap (that's their purpose!) -- the OS doesn't "disable" them, it fails to set up the kernel state that configures the secret keys used for pointer signing.

Are we in a situation where the fiber code is doing pointer authentication, the rest of the system isn't, and that causes issues?

Yes, exactly. The target triple appears not to have the paca feature, which means that rustc codegens the rest of the runtime (and all of userspace is compiled) without these instructions. The path I suggested above is to make our inline assembly match what rustc codegens for any other function.

Wasmtime GitHub notifications bot (Mar 16 2026 at 17:38):

alexcrichton commented on issue #12778:

What I don't understand is that the fiber code should work if pointer authentication is supported/implemented (aka not a nop) and the rest of the system isn't using it (e.g. Rust compiled without it). My understanding is that the inline asm we have encodes lr at the start of the function, and then decodes it at the end of the function (e.g. paciasp at the start, autiasp at the end). The trapping instruction in #11295 (and this issue) is autiasp, but the input to autiasp is, I thought, entirely controlled by wasmtime-fiber. Given that I don't understand why something needs to be conditional, but I must be missing something...

Wasmtime GitHub notifications bot (Mar 25 2026 at 11:50):

Heap-Hop commented on issue #12778:

I made two versions for this issue.

Fix Android pauth crash (minimal workaround)
It’s a simple workaround: simply skip all PAC instructions on Android.

Fix Android pauth crash
The changes are still generated by AI, reusing the cfg_if and trying to be reasonable.
It would be great if it was an inspiration for someone else can continue working on.

Wasmtime GitHub notifications bot (Apr 06 2026 at 01:00):

GamePad64 commented on issue #12778:

Encounered this issue on Android, too. @Heap-Hop your "maximal" fix works like a charm :+1:

Wasmtime GitHub notifications bot (Apr 16 2026 at 00:42):

alexcrichton commented on issue #12778:

Ok I think I've figured out what's going on here, but I don't know what to do about it. With this test:
    #[test]
    fn what() {
        let a = 3;
        let b = 3;
        let a = &a as *const _ as usize;
        let b = &b as *const _ as usize;
        eprintln!();
        eprintln!("pacia1716({a:#x}, {b:#x}) = {:#x}", pacia1716(a, b));
        std::thread::scope(|s| {
            s.spawn(move || {
                eprintln!("pacia1716({a:#x}, {b:#x}) = {:#x}", pacia1716(a, b));
            });
        });
        eprintln!("pacib1716({a:#x}, {b:#x}) = {:#x}", pacib1716(a, b));
        std::thread::scope(|s| {
            s.spawn(move || {
                eprintln!("pacib1716({a:#x}, {b:#x}) = {:#x}", pacib1716(a, b));
            });
        });
    }

    fn pacia1716(mut r17: usize, r16: usize) -> usize {
        unsafe {
            core::arch::asm!(
                "pacia1716\n",
                inout("x17") r17,
                in("x16") r16,
            );
            r17
        }
    }
    fn pacib1716(mut r17: usize, r16: usize) -> usize {
        unsafe {
            core::arch::asm!(
                "pacib1716\n",
                inout("x17") r17,
                in("x16") r16,
            );
            r17
        }
    }
the goal here is to investigate pointer authentication across threads.

On aarch64-unknown-linux-gnu running through qemu-aarch64, I see:
pacia1716(0x76338433de28, 0x76338433de2c) = 0x6976338433de28
pacia1716(0x76338433de28, 0x76338433de2c) = 0x6976338433de28
pacib1716(0x76338433de28, 0x76338433de2c) = 0x1c76338433de28
pacib1716(0x76338433de28, 0x76338433de2c) = 0x1c76338433de28
On aarch64-apple-darwin natively I see:
pacia1716(0x16f2a2538, 0x16f2a253c) = 0x16f2a2538
pacia1716(0x16f2a2538, 0x16f2a253c) = 0x16f2a2538
pacib1716(0x16f2a2538, 0x16f2a253c) = 0x16f2a2538
pacib1716(0x16f2a2538, 0x16f2a253c) = 0x16f2a2538
On aarch64-linux-android running through an emulator I see:
pacia1716(0x7464888218, 0x746488821c) = 0x3cd4f464888218
pacia1716(0x7464888218, 0x746488821c) = 0x6ad3f464888218
pacib1716(0x7464888218, 0x746488821c) = 0x6e407464888218
pacib1716(0x7464888218, 0x746488821c) = 0x6e407464888218
Notably Linux does indeed have pointer authentication instructions and they're enabled, and the keys are different. Turns out on macOS the pointer authentication is disabled, although I'm not sure why. On Android what's curious is that the "A" key has different values on different threads and the "B" key has the same value on different threads.

Through this I've got a simple test which crashes on Android and passes everywhere else:
    #[test]
    fn cross_thread_fiber() {
        let fiber = Fiber::<(), (), ()>::new(fiber_stack(1024 * 1024), move |_, s| {
            s.suspend(());
        })
        .unwrap();
        assert!(fiber.resume(()).is_err());
        std::thread::spawn(move || {
            assert!(fiber.resume(()).is_ok());
        })
        .join()
        .unwrap();
    }
Ok, so what to do about this. I do not think that looking at #[cfg(target_feature)] is correct primarily. That's not set on Rust for any of these targets and, in my understanding, does not reflect reality. Thoughts and questions I would still have are:

I can't find documentation of whether the "A" key or the "B" key should be used.

Linux kernel documentation indicates that the keys have the same values for all threads in a process, but the behvaior on Android is contradicting that.

I can't find any documentation on Android's implementation and support of pointer authentication (e.g. if they mention they have per-thread keys somewhere).

I don't know why we have pointer authentication in wasmtime-fiber in the first place. It's a nice defense-in-depth measure of sorts but all Rust code doesn't have pointer authentication by default anyway. I'm not sure why we want our handwritten assembly to have it, and my understanding is that if we didn't have it then the consequences wouldn't be all that much.

I don't know if we should just switch to using the "B" key everywhere. That seems to work on Android and it's already hardcoded to be used for macOS. I don't know why it's used on macOS and nowhere else. I cannot find ABI documentation for which key to use.

The "easy" path forward is to just jettison pointer authentication here, either entirely or just for Android targets. The not-as-easy-but-still-pretty-trivial path forward would be to switch Android to using the "B" key like macOS. The harder path forward is actually bottoming out answers to any of the questions I pose here. I suspect none of us are really equipped to answer these questions other than an aarch64 Android expert of which those that exist are probably all locked up at Google.

Looking at some history here it appears that #4195 is the genesis of our inline assembly here. The macOS implementation later came in #4720. These unfortunately don't answer my questions here, e.g. I don't know why macOS uses the "B" key and I also don't know why it's required in fibers in general.

Personally I'm tempted to just switch the keys from "A" to "B" for Android. That at least keeps the spirit of the change and I believe should work. I would continue to not understand why "A" doesn't work for Android, nor why all this is necessary in the first place.

Wasmtime GitHub notifications bot (Apr 16 2026 at 00:51):

cfallin commented on issue #12778:

Thanks for the investigation here!

I guess this is consistent with my theory above: the Android hardware (or emulator in this case) has the instructions, but the keys aren't configured as we would expect. "Keys differ for different threads, contra Linux's docs" is consistent with "target actually does not have this feature enabled at the ABI level, and config registers have random/leftover state", no? I guess I'm curious why "do what the Rust compiler does, i.e., do not do pauth" is not the right answer here?

Said another way: if we don't have Android ABI docs stating that the B key is guaranteed by the ABI to be set properly and set the same on all threads, then I don't think we should write code that assumes that.

If the target feature is disabled on all these targets, then actually we should be not be doing pauth on any of these targets, IMHO...

Wasmtime GitHub notifications bot (Apr 16 2026 at 01:05):

alexcrichton commented on issue #12778:

That's a good question, but I'm not sure how much we can use Rust as precedent here. The paca and pacg target features seem to have no effect on the output, and the -Zbranch-protection argument, which I think would have an effect, is unstable and nightly-only at this time. Even rustc +nightly --target aarch64-unknown-linux-gnu -Zbranch-protection=pac-ret --print cfg does not print the target features.

Locally using RUSTFLAGS=-Zbranch-protection=pac-ret cargo +nightly run --target aarch64-unknown-linux-gnu -Z build-std this builds a runnable binary (in QEMU) where I see it using the "A" key (matching wasmtime-fiber). When compiling for Android I also see it using the "A" key as well. Ironically when compiling for macOS it also uses the "A" key.

Effectively I don't think the Rust compilation environment provides any reasonable vector by which to learn about (a) whether pointer authentication is enabled and (b) which key is in use. Overall this is clearly still unstable to use for Rust code which is a much bigger target for a theoretical attack than our tiny bit of inline asm. To me this makes me rethink and agree with you that we should just jettison all of this. I think everything is too experimental at this time in Rust and I don't think the precedent is clear enough. One day if Rust provides #[cfg] to tell us what it's doing we can read that and use the appropriate key, but until then I think it's reasonable if both Rust and our inline asm doesn't have pointer authentication.

Wasmtime GitHub notifications bot (Apr 16 2026 at 22:03):

alexcrichton closed issue #12778:

Test Case

Use the original bytecodealliance/sample-wasi-http-rust's /echo API is enough to reproduce.

Environment

Android arm64 device or emulator.
Latest release of wasmtime-v42.0.1-aarch64-android.tar.xz and bytecodealliance/sample-wasi-http-rust.

Steps to Reproduce
adb shell mkdir -p /data/local/tmp
adb push wasmtime sample-wasi-http-rust.wasm /data/local/tmp
adb forward tcp:8080 tcp:8080
adb shell HOME=/data/local/tmp \
  /data/local/tmp/wasmtime serve \
  -Scli -Shttp \
  /data/local/tmp/sample-wasi-http-rust.wasm
Open a new terminal:
curl -d "$(printf 'data%.0s' {1..5000})"  http://127.0.0.1:8080/echo > /dev/null
output:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 28038    0  8038  100 20000   197k   491k --:--:-- --:--:-- --:--:--  702k
curl: (18) transfer closed with outstanding read data remaining
And the wasmtime process crashed and showed:
Serving HTTP on http://0.0.0.0:8080/
Illegal instruction
https://github.com/user-attachments/assets/d86ec46d-ccdf-4421-a879-21fb0797e762

Also tried on Termux, same results.

Expected Results

wasmtime process should not exit

Actual Results

Illegal instruction

Versions and Environment

Wasmtime version or commit: wasmtime-v42.0.1-aarch64-android

Operating system: Android 16

Architecture: arm64

Extra Info

Maybe related to #11295

Last updated: May 03 2026 at 21:15 UTC

Stream: git-wasmtime

Topic: wasmtime / issue #12778 `Illegal instruction` on arm64 An...

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:11):

Test Case

Environment

Steps to Reproduce

Expected Results

Actual Results

Versions and Environment

Extra Info

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:11):

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:14):

Test Case

Environment

Steps to Reproduce

Expected Results

Actual Results

Versions and Environment

Extra Info

Wasmtime GitHub notifications bot (Mar 14 2026 at 11:29):

Wasmtime GitHub notifications bot (Mar 14 2026 at 12:56):

Pointer Authentication (PAuth) and Android SIGILL crash

Background

What changed

Security implications

:check: Safe for devices without PAuth support

:secure: CFI is preserved when PAuth is enabled

Relevant files

Wasmtime GitHub notifications bot (Mar 14 2026 at 13:14):

Wasmtime GitHub notifications bot (Mar 15 2026 at 10:42):

Wasmtime GitHub notifications bot (Mar 15 2026 at 17:54):

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:36):

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:36):

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:41):

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:44):

Wasmtime GitHub notifications bot (Mar 16 2026 at 15:45):

Wasmtime GitHub notifications bot (Mar 16 2026 at 16:29):

Wasmtime GitHub notifications bot (Mar 16 2026 at 16:58):

Wasmtime GitHub notifications bot (Mar 16 2026 at 17:38):

Wasmtime GitHub notifications bot (Mar 25 2026 at 11:50):

Wasmtime GitHub notifications bot (Apr 06 2026 at 01:00):

Wasmtime GitHub notifications bot (Apr 16 2026 at 00:42):

Wasmtime GitHub notifications bot (Apr 16 2026 at 00:51):

Wasmtime GitHub notifications bot (Apr 16 2026 at 01:05):

Wasmtime GitHub notifications bot (Apr 16 2026 at 22:03):

Test Case

Environment

Steps to Reproduce

Expected Results

Actual Results

Versions and Environment

Extra Info

Pointer Authentication (PAuth) and Android `SIGILL` crash