Stream: git-wasmtime

Topic: wasmtime / issue #10099 Crash in macOS when using wasmtim...

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 11:18):

plusvic added the bug label to Issue #10099.

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 11:19):

plusvic opened issue #10099:

This is a very esoteric issue related to the use of Mach ports for exception handling when wasmtime is running inside a Python 3.9 process. I'm the author of a multi-platform Rust library that embeds wasmtime for running WASM code, the library supports Linux, Windows and macOS. This library also have Python bindings that allow using the library from Python.

A user of the library reported a crash that was happening in a very specific environment: macOS 15.2 (both Intel and ARM are affected, and probably other macOS versions too), with Python 3.9.6 (the version that comes pre-installed in macOS). The process was being SIGKILLed when executing code deep inside wasmtime, the same code worked fine in other platforms, or in the same macOS machine with Python 3.11.

After some investigation everything pointed to the Mach ports exception handling mechanism when wasmtime is run inside Python 3.9. In fact, if I turn off the use of Mach ports in wasmtime everything works fine.

I don't have a clue on why this is happening only with Python 3.9. My guess is that it is related to Python's exception handling mechanism that somewhat is interfering with wasmtime's exception handling, and perhaps something changed in later Python versions. It could be a Python's bug that was fixed, it could a design change. For the time being I can turn off Mach ports as a workaround, but I would like to understand what's happening here.

wasmtime-python.zip

Test Case

As my library is a quite complicated project, I've created a simple Rust project that reproduces the issue (attached to this issue). It consists in a Python extension written in Rust using PyO3, this extension exports a function that

Steps to Reproduce

Expected Results

The process should finish without crashes.

Actual Results

The process gets killed:

/usr/bin/python3 crash.py                                                                                                                    ✔
[1]    34279 killed     /usr/bin/python3 crash.py

The crash log is included below, notice that the exception type is EXC_GUARD and the code is GUARD_TYPE_MACH_PORT. According to Apple's documentation, the process is killed with EXC_GUARD when it detects a violation in some guarded resourced, usually a file descriptor or Mach port.

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               Python [34279]
Path:                  /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Identifier:            com.apple.python3
Version:               3.9.6 (3.9.6)
Build Info:            python3-141000000000000~2583
Code Type:             X86-64 (Native)
Parent Process:        zsh [1143]
Responsible:           iTerm2 [911]
User ID:               178948

Date/Time:             2025-01-24 11:53:27.1988 +0100
OS Version:            macOS 15.2 (24C101)
Report Version:        12
Bridge OS Version:     9.2 (22P2093)
Anonymous UUID:        5FA73C73-360A-C399-E0CC-3B9EAB5492D0

Sleep/Wake UUID:       CA017020-8C6C-41A1-AE24-D06480AE16DB

Time Awake Since Boot: 80000 seconds
Time Since Wake:       5718 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_GUARD (SIGKILL)
Exception Codes:       GUARD_TYPE_MACH_PORT
Exception Codes:       0x0000000000000000, 0x0000000000000000

Termination Reason:    Namespace GUARD, Code 2305843034983497728

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib              0x7ff80b5daf0e mach_msg2_trap + 10
1   libsystem_kernel.dylib              0x7ff80b5e95fa mach_msg2_internal + 84
2   libsystem_kernel.dylib              0x7ff80b5dfffb thread_set_exception_ports + 134
3   wasmtime_python.abi3.so                0x10d421005 wasmtime::runtime::vm::sys::unix::machports::lazy_per_thread_init::h995e1f3306bad314 + 69
4   wasmtime_python.abi3.so                0x10ce54cd5 wasmtime::runtime::vm::traphandlers::tls::raw::replace::h7b1ea496b7d61dcd + 85
5   wasmtime_python.abi3.so                0x10cd6cc95 wasmtime::runtime::vm::traphandlers::catch_traps::haad6173518c65209 + 213
6   wasmtime_python.abi3.so                0x10cd66c62 wasmtime::runtime::func::typed::TypedFunc$LT$Params$C$Results$GT$::call::h4ad4a8810ccdb7a1 + 354
7   wasmtime_python.abi3.so                0x10cd6e575 wasmtime_python::_$LT$impl$u20$wasmtime_python..sum..MakeDef$GT$::_PYO3_DEF::trampoline::hfe78f2643f499858 + 1077
8   Python3                                0x10c0a6ae8 0x10c024000 + 535272
9   Python3                                0x10c066830 _PyObject_MakeTpCall + 384
10  Python3                                0x10c143850 0x10c024000 + 1177680
11  Python3                                0x10c140557 _PyEval_EvalFrameDefault + 23527
12  Python3                                0x10c144693 0x10c024000 + 1181331
13  Python3                                0x10c13a871 PyEval_EvalCode + 81
14  Python3                                0x10c181ea1 0x10c024000 + 1433249
15  Python3                                0x10c182013 0x10c024000 + 1433619
16  Python3                                0x10c1802a4 PyRun_SimpleFileExFlags + 708
17  Python3                                0x10c19d6f0 Py_RunMain + 1840
18  Python3                                0x10c19db80 0x10c024000 + 1547136
19  Python3                                0x10c19dbdb Py_BytesMain + 43
20  dyld                                0x7ff80b28d2cd start + 1805

Thread 1:
0   libsystem_kernel.dylib              0x7ff80b5daf0e mach_msg2_trap + 10
1   libsystem_kernel.dylib              0x7ff80b5e95fa mach_msg2_internal + 84
2   libsystem_kernel.dylib              0x7ff80b5e2016 mach_msg_overwrite + 649
3   libsystem_kernel.dylib              0x7ff80b5db1ff mach_msg + 19
4   wasmtime_python.abi3.so                0x10ce55ef5 wasmtime::runtime::vm::sys::unix::machports::handler_thread::hbfc195b7bc426a10 + 277
5   wasmtime_python.abi3.so                0x10ce08da9 std::sys::backtrace::__rust_begin_short_backtrace::h9512e42a7b3024a7 + 9
6   wasmtime_python.abi3.so                0x10cdec98a core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hd93b48b54e95ca2f + 138
7   wasmtime_python.abi3.so                0x10d3f5c5b std::sys::pal::unix::thread::Thread::new::thread_start::hcfbe36d5799581a6 + 43
8   libsystem_pthread.dylib             0x7ff80b61c253 _pthread_start + 99
9   libsystem_pthread.dylib             0x7ff80b617bef thread_start + 15

Thread 2:
0   libsystem_kernel.dylib              0x7ff80b5ddaaa __psynch_cvwait + 10
1   libsystem_pthread.dylib             0x7ff80b61c7a8 _pthread_cond_wait + 1193
2   wasmtime_python.abi3.so                0x10d44518f rayon_core::sleep::Sleep::sleep::h0f8e31d6649cc8dc + 511
3   wasmtime_python.abi3.so                0x10d444b54 rayon_core::registry::WorkerThread::wait_until_cold::h5e480e7580c0b42b + 276
4   wasmtime_python.abi3.so                0x10d1e5d14 rayon_core::registry::ThreadBuilder::run::h1666e03950a6d606 + 340
5   wasmtime_python.abi3.so                0x10d1ebbf2 std::sys::backtrace::__rust_begin_short_backtrace::h84248f773d70d50b + 34
6   wasmtime_python.abi3.so                0x10d1ea2e5 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h2bcc168af3d264f4 + 373
7   wasmtime_python.abi3.so                0x10d3f5c5b std::sys::pal::unix::thread::Thread::new::thread_start::hcfbe36d5799581a6 + 43
8   libsystem_pthread.dylib             0x7ff80b61c253 _pthread_start + 99
9   libsystem_pthread.dylib             0x7ff80b617bef thread_start + 15

Thread 3:
0   libsystem_kernel.dylib              0x7ff80b5ddaaa __psynch_cvwait + 10
1   libsystem_pthread.dylib             0x7ff80b61c7a8 _pthread_cond_wait + 1193
2   wasmtime_python.abi3.so                0x10d44518f rayon_core::sleep::Sleep::sleep::h0f8e31d6649cc8dc + 511
3   wasmtime_python.abi3.so                0x10d444b54 rayon_core::registry::WorkerThread::wait_until_cold::h5e480e7580c0b42b + 276
4   wasmtime_python.abi3.so                0x10d1e5d14 rayon_core::registry::ThreadBuilder::run::h1666e03950a6d606 + 340
5   wasmtime_python.abi3.so                0x10d1ebbf2 std::sys::backtrace::__rust_begin_short_backtrace::h84248f773d70d50b + 34
6   wasmtime_python.abi3.so                0x10d1ea2e5 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h2bcc168af3d264f4 + 373
7   wasmtime_python.abi3.so                0x10d3f5c5b std::sys::pal::unix::thread::Thread::new::thread_start::hcfbe36d5799581a6 + 43
8   libsystem_pthread.dylib             0x7ff80b61c253 _pthread_start + 99
9   libsystem_pthread.dylib             0x7ff80b617bef thread_start + 15

Thread 4:
0   libsystem_kernel.dylib              0x7ff80b5ddaaa __psynch_cvwait + 10
1   libsystem_pthread.dylib             0x7ff80b61c7a8 _pthread_cond_wait + 1193
2   wasmtime_python.abi3.so                0x10d44518f rayon_core::sleep::Sleep::sleep::h0f8e31d6649cc8dc + 511
3   wasmtime_python.abi3.so                0x10d444b54 rayon_core::registry::WorkerThread::wait_until_cold::h5e480e7580c0b42b + 276
4   wasmtime_python.abi3.so                0x10d1e5d14 rayon_core::registry::ThreadBuilder::run::h1666e03950a6d606 + 340
5   wasmtime_python.abi3.so                0x10d1e
[message truncated]

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 11:20):

plusvic edited issue #10099:

This is a very esoteric issue related to the use of Mach ports for exception handling when wasmtime is running inside a Python 3.9 process. I'm the author of a multi-platform Rust library that embeds wasmtime for running WASM code, the library supports Linux, Windows and macOS. This library also have Python bindings that allow using the library from Python.

A user of the library reported a crash that was happening in a very specific environment: macOS 15.2 (both Intel and ARM are affected, and probably other macOS versions too), with Python 3.9.6 (the version that comes pre-installed in macOS). The process was being SIGKILLed when executing code deep inside wasmtime, the same code worked fine in other platforms, or in the same macOS machine with Python 3.11.

After some investigation everything pointed to the Mach ports exception handling mechanism when wasmtime is run inside Python 3.9. In fact, if I turn off the use of Mach ports in wasmtime everything works fine.

I don't have a clue on why this is happening only with Python 3.9. My guess is that it is related to Python's exception handling mechanism that somewhat is interfering with wasmtime's exception handling, and perhaps something changed in later Python versions. It could be a Python's bug that was fixed, it could a design change. For the time being I can turn off Mach ports as a workaround, but I would like to understand what's happening here.

wasmtime-python.zip

Test Case

As my library is a quite complicated project, I've created a simple Rust project that reproduces the issue (attached to this issue). It consists in a Python extension written in Rust using PyO3.

Steps to Reproduce

Expected Results

The process should finish without crashes.

Actual Results

The process gets killed:

/usr/bin/python3 crash.py                                                                                                                    ✔
[1]    34279 killed     /usr/bin/python3 crash.py

The crash log is included below, notice that the exception type is EXC_GUARD and the code is GUARD_TYPE_MACH_PORT. According to Apple's documentation, the process is killed with EXC_GUARD when it detects a violation in some guarded resourced, usually a file descriptor or Mach port.

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               Python [34279]
Path:                  /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Identifier:            com.apple.python3
Version:               3.9.6 (3.9.6)
Build Info:            python3-141000000000000~2583
Code Type:             X86-64 (Native)
Parent Process:        zsh [1143]
Responsible:           iTerm2 [911]
User ID:               178948

Date/Time:             2025-01-24 11:53:27.1988 +0100
OS Version:            macOS 15.2 (24C101)
Report Version:        12
Bridge OS Version:     9.2 (22P2093)
Anonymous UUID:        5FA73C73-360A-C399-E0CC-3B9EAB5492D0

Sleep/Wake UUID:       CA017020-8C6C-41A1-AE24-D06480AE16DB

Time Awake Since Boot: 80000 seconds
Time Since Wake:       5718 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_GUARD (SIGKILL)
Exception Codes:       GUARD_TYPE_MACH_PORT
Exception Codes:       0x0000000000000000, 0x0000000000000000

Termination Reason:    Namespace GUARD, Code 2305843034983497728

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib              0x7ff80b5daf0e mach_msg2_trap + 10
1   libsystem_kernel.dylib              0x7ff80b5e95fa mach_msg2_internal + 84
2   libsystem_kernel.dylib              0x7ff80b5dfffb thread_set_exception_ports + 134
3   wasmtime_python.abi3.so                0x10d421005 wasmtime::runtime::vm::sys::unix::machports::lazy_per_thread_init::h995e1f3306bad314 + 69
4   wasmtime_python.abi3.so                0x10ce54cd5 wasmtime::runtime::vm::traphandlers::tls::raw::replace::h7b1ea496b7d61dcd + 85
5   wasmtime_python.abi3.so                0x10cd6cc95 wasmtime::runtime::vm::traphandlers::catch_traps::haad6173518c65209 + 213
6   wasmtime_python.abi3.so                0x10cd66c62 wasmtime::runtime::func::typed::TypedFunc$LT$Params$C$Results$GT$::call::h4ad4a8810ccdb7a1 + 354
7   wasmtime_python.abi3.so                0x10cd6e575 wasmtime_python::_$LT$impl$u20$wasmtime_python..sum..MakeDef$GT$::_PYO3_DEF::trampoline::hfe78f2643f499858 + 1077
8   Python3                                0x10c0a6ae8 0x10c024000 + 535272
9   Python3                                0x10c066830 _PyObject_MakeTpCall + 384
10  Python3                                0x10c143850 0x10c024000 + 1177680
11  Python3                                0x10c140557 _PyEval_EvalFrameDefault + 23527
12  Python3                                0x10c144693 0x10c024000 + 1181331
13  Python3                                0x10c13a871 PyEval_EvalCode + 81
14  Python3                                0x10c181ea1 0x10c024000 + 1433249
15  Python3                                0x10c182013 0x10c024000 + 1433619
16  Python3                                0x10c1802a4 PyRun_SimpleFileExFlags + 708
17  Python3                                0x10c19d6f0 Py_RunMain + 1840
18  Python3                                0x10c19db80 0x10c024000 + 1547136
19  Python3                                0x10c19dbdb Py_BytesMain + 43
20  dyld                                0x7ff80b28d2cd start + 1805

Thread 1:
0   libsystem_kernel.dylib              0x7ff80b5daf0e mach_msg2_trap + 10
1   libsystem_kernel.dylib              0x7ff80b5e95fa mach_msg2_internal + 84
2   libsystem_kernel.dylib              0x7ff80b5e2016 mach_msg_overwrite + 649
3   libsystem_kernel.dylib              0x7ff80b5db1ff mach_msg + 19
4   wasmtime_python.abi3.so                0x10ce55ef5 wasmtime::runtime::vm::sys::unix::machports::handler_thread::hbfc195b7bc426a10 + 277
5   wasmtime_python.abi3.so                0x10ce08da9 std::sys::backtrace::__rust_begin_short_backtrace::h9512e42a7b3024a7 + 9
6   wasmtime_python.abi3.so                0x10cdec98a core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hd93b48b54e95ca2f + 138
7   wasmtime_python.abi3.so                0x10d3f5c5b std::sys::pal::unix::thread::Thread::new::thread_start::hcfbe36d5799581a6 + 43
8   libsystem_pthread.dylib             0x7ff80b61c253 _pthread_start + 99
9   libsystem_pthread.dylib             0x7ff80b617bef thread_start + 15

Thread 2:
0   libsystem_kernel.dylib              0x7ff80b5ddaaa __psynch_cvwait + 10
1   libsystem_pthread.dylib             0x7ff80b61c7a8 _pthread_cond_wait + 1193
2   wasmtime_python.abi3.so                0x10d44518f rayon_core::sleep::Sleep::sleep::h0f8e31d6649cc8dc + 511
3   wasmtime_python.abi3.so                0x10d444b54 rayon_core::registry::WorkerThread::wait_until_cold::h5e480e7580c0b42b + 276
4   wasmtime_python.abi3.so                0x10d1e5d14 rayon_core::registry::ThreadBuilder::run::h1666e03950a6d606 + 340
5   wasmtime_python.abi3.so                0x10d1ebbf2 std::sys::backtrace::__rust_begin_short_backtrace::h84248f773d70d50b + 34
6   wasmtime_python.abi3.so                0x10d1ea2e5 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h2bcc168af3d264f4 + 373
7   wasmtime_python.abi3.so                0x10d3f5c5b std::sys::pal::unix::thread::Thread::new::thread_start::hcfbe36d5799581a6 + 43
8   libsystem_pthread.dylib             0x7ff80b61c253 _pthread_start + 99
9   libsystem_pthread.dylib             0x7ff80b617bef thread_start + 15

Thread 3:
0   libsystem_kernel.dylib              0x7ff80b5ddaaa __psynch_cvwait + 10
1   libsystem_pthread.dylib             0x7ff80b61c7a8 _pthread_cond_wait + 1193
2   wasmtime_python.abi3.so                0x10d44518f rayon_core::sleep::Sleep::sleep::h0f8e31d6649cc8dc + 511
3   wasmtime_python.abi3.so                0x10d444b54 rayon_core::registry::WorkerThread::wait_until_cold::h5e480e7580c0b42b + 276
4   wasmtime_python.abi3.so                0x10d1e5d14 rayon_core::registry::ThreadBuilder::run::h1666e03950a6d606 + 340
5   wasmtime_python.abi3.so                0x10d1ebbf2 std::sys::backtrace::__rust_begin_short_backtrace::h84248f773d70d50b + 34
6   wasmtime_python.abi3.so                0x10d1ea2e5 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h2bcc168af3d264f4 + 373
7   wasmtime_python.abi3.so                0x10d3f5c5b std::sys::pal::unix::thread::Thread::new::thread_start::hcfbe36d5799581a6 + 43
8   libsystem_pthread.dylib             0x7ff80b61c253 _pthread_start + 99
9   libsystem_pthread.dylib             0x7ff80b617bef thread_start + 15

Thread 4:
0   libsystem_kernel.dylib              0x7ff80b5ddaaa __psynch_cvwait + 10
1   libsystem_pthread.dylib             0x7ff80b61c7a8 _pthread_cond_wait + 1193
2   wasmtime_python.abi3.so                0x10d44518f rayon_core::sleep::Sleep::sleep::h0f8e31d6649cc8dc + 511
3   wasmtime_python.abi3.so                0x10d444b54 rayon_core::registry::WorkerThread::wait_until_cold::h5e480e7580c0b42b + 276
4   wasmtime_python.abi3.so                0x10d1e5d14 rayon_core::registry::ThreadBuilder::run::h1666e03950a6d606 + 340
5   wasmtime_python.abi3.so                0x10d1ebbf2 std::sys::backtrace::__rust_begin_s
[message truncated]

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 11:30):

plusvic commented on issue #10099:

I added the bug label and can't remove it now. This is may not be a wasmtime.

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 11:30):

plusvic edited a comment on issue #10099:

I added the bug label and can't remove it now. This is may not be a wasmtime bug.

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 11:35):

bjorn3 commented on issue #10099:

The crash happens when thread_set_exception_ports is called by Wasmtime: https://github.com/bytecodealliance/wasmtime/blob/887e5c93500c6f85e2570046788761a9f4498123/crates/wasmtime/src/runtime/vm/sys/unix/machports.rs#L483

with Python 3.9.6 (the version that comes pre-installed in macOS)

Do you happen to know the exact set of entitlements this executable was compiled with? Maybe one of the hardening entitlements prevents using exception ports?

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 12:38):

plusvic commented on issue #10099:

The crash happens when thread_set_exception_ports is called by Wasmtime:

wasmtime/crates/wasmtime/src/runtime/vm/sys/unix/machports.rs

Line 483 in 887e5c9

let kret = thread_set_exception_ports(

with Python 3.9.6 (the version that comes pre-installed in macOS)

Do you happen to know the exact set of entitlements this executable was compiled with? Maybe one of the hardening entitlements prevents using exception ports?

The Python interpreter executable doesn't seem to specify any entitlement:

codesign -d --entitlements - /usr/bin/python3
Executable=/usr/bin/python3

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 12:59):

plusvic commented on issue #10099:

The crash happens when thread_set_exception_ports is called by Wasmtime:
wasmtime/crates/wasmtime/src/runtime/vm/sys/unix/machports.rs
Line 483 in 887e5c9
let kret = thread_set_exception_ports(

with Python 3.9.6 (the version that comes pre-installed in macOS)

Do you happen to know the exact set of entitlements this executable was compiled with? Maybe one of the hardening entitlements prevents using exception ports?

The Python interpreter executable doesn't seem to specify any entitlement:

codesign -d --entitlements - /usr/bin/python3 Executable=/usr/bin/python3

With this other path I get a different result, but nothing that prevents the use of exception ports.

codesign -d --entitlements - /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Executable=/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
[Dict]
    [Key] com.apple.application-identifier
    [Value]
        [String] com.apple.python3

view this post on Zulip Wasmtime GitHub notifications bot (Jan 24 2025 at 19:41):

alexcrichton commented on issue #10099:

Thanks for the thorough report and reproduction steps! I took a look at this but alas don't have much to add. It deifnitely seems like macOS doesn't like the call to thread_set_exception_ports but I'm not really sure why. Searching around isn't turning up much that seems relevant, and I'm definitely no macOS expert myself to know what's going on here...

view this post on Zulip Wasmtime GitHub notifications bot (Jan 27 2025 at 13:13):

plusvic commented on issue #10099:

To make this issue even more strange, everything is OK with a Python binary downloaded from python.org. Exactly the same version (Python 3.9.6), it crashes with the Python binary provided by Apple, but works correctly with the one from python.org.

It looks like Apple is making some modifications to the Python binary they distribute.

Last updated: Feb 28 2025 at 01:30 UTC