Stream: git-wasmtime

Topic: wasmtime / issue #13857 Unix: sigaltstack is munmapped on...

view this post on Zulip Wasmtime GitHub notifications bot (Jul 10 2026 at 01:34):

tiandiwonder opened issue #13857:

Summary

On Unix, lazy_per_thread_init (crates/wasmtime/src/runtime/vm/sys/unix/signals.rs) registers a custom alternate signal stack per thread. Its TLS destructor deallocates it like this:

impl Drop for Stack {
    fn drop(&mut self) {
        unsafe {
            // Deallocate the stack memory.
            let r = rustix::mm::munmap(self.mmap_ptr, self.mmap_size);
            debug_assert!(r.is_ok(), "munmap failed during thread shutdown");
        }
    }
}

The stack is munmapped without first calling sigaltstack with SS_DISABLE (or restoring the previous stack), so the kernel keeps the freed range registered as the thread's alternate signal stack for the remainder of thread teardown.

Consequences

  1. If any SA_ONSTACK signal is delivered to the thread after this destructor runs (other TLS destructors, sanitizer teardown, etc.), the kernel pushes the signal frame onto unmapped memory.

  2. Worse, for AddressSanitizer embedders: ASan's thread-destruction hook calls UnsetAlternateSignalStack() (compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp), which queries the currently registered altstack and unconditionally UnmapOrDie(oldstack.ss_sp, oldstack.ss_size). With the dangling registration left by Stack::drop, ASan re-munmaps a 256 KiB range that was already freed — and possibly already reused by the allocator — silently destroying pages of live, unrelated heap allocations. The eventual crash appears at a random later point in whatever code owned the reused memory, which makes it extremely hard to attribute.

This looks like the actual mechanism behind the long-standing mystery documented in the comment at the top of lazy_per_thread_init (the ASan fuzzing crashes reproduced in https://gist.github.com/alexcrichton/6815a5d57a3c5ca94a8d816a9fcc91af): it is not TLS-destructor ordering corrupting ASan state — it is the dangling sigaltstack registration making ASan's UnsetAlternateSignalStack unmap a stale (reused) range. The existing cfg!(asan) guard only protects builds where the Rust code is compiled with -Zsanitizer=address; embedders that instrument only their C/C++ side with ASan and link wasmtime built without it (a common setup) still hit the bug.

Real-world impact

We (ClickHouse) chased sporadic ASan CI crashes for two months, where random large read buffers were found unmapped mid-use. Forensics of a core dump showed the faulted buffer was still registered as live in ASan's secondary allocator (never freed) and the destroyed page range measured exactly one wasmtime sigaltstack allocation (guard page + MIN_STACK_SIZE = 64×4096); a WASM module had executed on a pool thread minutes earlier, and the crash fired when that thread was reaped. Details: https://github.com/ClickHouse/ClickHouse/issues/104692, fixed on our side by re-enabling cfg(asan) via https://github.com/ClickHouse/ClickHouse/pull/109950.

Suggested fix

In Stack::drop, disable (or restore the previously registered) alternate signal stack before unmapping:

impl Drop for Stack {
    fn drop(&mut self) {
        unsafe {
            let disable = libc::stack_t {
                ss_sp: ptr::null_mut(),
                ss_flags: libc::SS_DISABLE,
                ss_size: MIN_STACK_SIZE,
            };
            let r = libc::sigaltstack(&disable, ptr::null_mut());
            debug_assert_eq!(r, 0, "sigaltstack(SS_DISABLE) failed during thread shutdown");
            let r = rustix::mm::munmap(self.mmap_ptr, self.mmap_size);
            debug_assert!(r.is_ok(), "munmap failed during thread shutdown");
        }
    }
}

With that in place the cfg!(asan) opt-out may even become unnecessary, since ASan's UnsetAlternateSignalStack would then see a disabled stack instead of a dangling one (worth verifying — UnmapOrDie(NULL, ...) on a disabled stack would fail loudly rather than corrupt, so ASan-instrumented-Rust builds may still want the guard).

view this post on Zulip Wasmtime GitHub notifications bot (Jul 10 2026 at 14:53):

alexcrichton commented on issue #13857:

Thanks for the report! I'm not certain there's much we can do about this though other than recommending that asan builds indeed compile Rust with asan as well to disable the sigaltstack. This is a much better explanation to the current hand-wavy one of why we can't use a sigaltstack with asan, however.

What I'd be worried about is that I basically don't think we're equipped to reset the thread's sigaltstack back to the original on drop. While we could preserve whatever was leftover there we still don't have any guarantee about the order of TLS destructors so we don't know if we'd reset the stack from Wasmtime's stack to the prior or if the stack has already been reset by something else. Additionally I don't think we can just unset the sigaltstack because historical versions of LLVM seem to always call munmap on the return value of sigaltstack when unsetting.

Looking at LLVM tip-of-tree, however, it looks like this behavior has changed and might mean that this is now compatible? I think that the tip-of-tree behavior would be compatible with Wasmtime as-is, and it'd also be compatible with unsetting the sigaltstack as well in the handler.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2026 at 09:33):

tiandiwonder commented on issue #13857:

Re the historical-LLVM concern: I checked the released compiler-rt implementations, and UnmapOrDie starts with if (!addr || !size) return; (sanitizer_posix.cpp; it is the same at least as far back as the clang-21 runtime where we hit this). After sigaltstack(SS_DISABLE), Linux reports ss_sp = NULL, ss_size = 0 for the disabled stack, so a historical UnsetAlternateSignalStack running after the disable does UnmapOrDie(NULL, 0) — a harmless no-op rather than a Die. So plain SS_DISABLE in Stack::drop (no restoring of the previous stack, agreed that is not robust under TLS-destructor ordering) appears compatible with both historical and tip-of-tree runtimes. Darwin is worth double-checking, but per POSIX a disabled stack reports SS_DISABLE and sanitizers only act on the pointer.

To be fair about coverage: SS_DISABLE does not help the opposite TLS-destructor ordering — if ASan's thread teardown runs before wasmtime's TLS destructor, a pre-llvm/llvm-project#179000 runtime still unmaps the then-live wasmtime stack. That side can only be fixed in the sanitizer runtime, and #179000 (merged 2026-05-11) ships in LLVM 23 at the earliest; every released LLVM (≤ 22) has the unconditional unmap.

Still, SS_DISABLE in Stack::drop looks like a strict improvement with no downside on any runtime version: it fixes the dangling kernel registration for all embedders (consequence 1 — signal frames pushed onto unmapped memory during the rest of thread teardown), and it closes the ordering we actually hit in production (wasmtime destructor first, ASan teardown second). Keeping the cfg!(asan) guard on top for Rust-instrumented builds remains reasonable.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 20 2026 at 19:57):

alexcrichton closed issue #13857:

Summary

On Unix, lazy_per_thread_init (crates/wasmtime/src/runtime/vm/sys/unix/signals.rs) registers a custom alternate signal stack per thread. Its TLS destructor deallocates it like this:

impl Drop for Stack {
    fn drop(&mut self) {
        unsafe {
            // Deallocate the stack memory.
            let r = rustix::mm::munmap(self.mmap_ptr, self.mmap_size);
            debug_assert!(r.is_ok(), "munmap failed during thread shutdown");
        }
    }
}

The stack is munmapped without first calling sigaltstack with SS_DISABLE (or restoring the previous stack), so the kernel keeps the freed range registered as the thread's alternate signal stack for the remainder of thread teardown.

Consequences

  1. If any SA_ONSTACK signal is delivered to the thread after this destructor runs (other TLS destructors, sanitizer teardown, etc.), the kernel pushes the signal frame onto unmapped memory.

  2. Worse, for AddressSanitizer embedders: ASan's thread-destruction hook calls UnsetAlternateSignalStack() (compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp), which queries the currently registered altstack and unconditionally UnmapOrDie(oldstack.ss_sp, oldstack.ss_size). With the dangling registration left by Stack::drop, ASan re-munmaps a 256 KiB range that was already freed — and possibly already reused by the allocator — silently destroying pages of live, unrelated heap allocations. The eventual crash appears at a random later point in whatever code owned the reused memory, which makes it extremely hard to attribute.

This looks like the actual mechanism behind the long-standing mystery documented in the comment at the top of lazy_per_thread_init (the ASan fuzzing crashes reproduced in https://gist.github.com/alexcrichton/6815a5d57a3c5ca94a8d816a9fcc91af): it is not TLS-destructor ordering corrupting ASan state — it is the dangling sigaltstack registration making ASan's UnsetAlternateSignalStack unmap a stale (reused) range. The existing cfg!(asan) guard only protects builds where the Rust code is compiled with -Zsanitizer=address; embedders that instrument only their C/C++ side with ASan and link wasmtime built without it (a common setup) still hit the bug.

Real-world impact

We (ClickHouse) chased sporadic ASan CI crashes for two months, where random large read buffers were found unmapped mid-use. Forensics of a core dump showed the faulted buffer was still registered as live in ASan's secondary allocator (never freed) and the destroyed page range measured exactly one wasmtime sigaltstack allocation (guard page + MIN_STACK_SIZE = 64×4096); a WASM module had executed on a pool thread minutes earlier, and the crash fired when that thread was reaped. Details: https://github.com/ClickHouse/ClickHouse/issues/104692, fixed on our side by re-enabling cfg(asan) via https://github.com/ClickHouse/ClickHouse/pull/109950.

Suggested fix

In Stack::drop, disable (or restore the previously registered) alternate signal stack before unmapping:

impl Drop for Stack {
    fn drop(&mut self) {
        unsafe {
            let disable = libc::stack_t {
                ss_sp: ptr::null_mut(),
                ss_flags: libc::SS_DISABLE,
                ss_size: MIN_STACK_SIZE,
            };
            let r = libc::sigaltstack(&disable, ptr::null_mut());
            debug_assert_eq!(r, 0, "sigaltstack(SS_DISABLE) failed during thread shutdown");
            let r = rustix::mm::munmap(self.mmap_ptr, self.mmap_size);
            debug_assert!(r.is_ok(), "munmap failed during thread shutdown");
        }
    }
}

With that in place the cfg!(asan) opt-out may even become unnecessary, since ASan's UnsetAlternateSignalStack would then see a disabled stack instead of a dangling one (worth verifying — UnmapOrDie(NULL, ...) on a disabled stack would fail loudly rather than corrupt, so ASan-instrumented-Rust builds may still want the guard).

Last updated: Jul 29 2026 at 05:03 UTC