Hi there! I am getting the following assertion failure on my project's CI (macos) with wasmtime 0.27.0 on module instantiation. Will bump to 0.28.0 and investigate further, but I thought I'd report it here first just in case:
# thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
# left: `268435466`,
# right: `0`: failed to set thread exception port', crates/runtime/src/traphandlers/macos.rs:436:9
Ping on this - this still happens everyday on macOS (GitHub actions) with wasmtime v0.28.0; does nobody else use wasmtime on macos and encounter this?
macOS os tested on CI.
Just a guess, but is your CI restricted in any way? Or running an older macOS version?
The error code is 0x1000000a by the way.
This corresponds to MACH_SEND_INVALID_RIGHT
@Thibault Charbonnier sorry I don't think I saw this when it was originally posted! Can you describe your setup with macOS and CI a bit more? It looks like the thread_set_exception_ports
call is failing, but I'm not sure why such a call would fail myself (I'm not super familiar with mach)
We lifted most of our mach bits from spidermonkey and looking around there if this function call fails it basically disables wasm execution in firefox. In that sense this seems like it's not really expected to fail, so I'm not sure why it's failing!
As mentioned, though, we do run CI on macOS, so everything should work. If previous versions of Wasmtime work for you do you think you'd be able to run a bisection to figure out the failing commit?
No, it is macos-latest
and the official release artifacts for v0.28.0 + clang
hm ok so this is on GitHub Actions CI? Can you describe how you're embedding Wasmtime? (e.g. language and runtime and such)
This is wasmtime inside of Nginx, running the proxy-wasm Rust SDK
is the code for this public? (e.g. could I repro locally?)
No it is not, sorry
ah ok no worries, are you able to help out in narrowing this down and/or debugging it? Without a reproduction locally I'm not sure how this could happen so I'm not sure what's going on :(
This is how nginx is built/link to wasmtime on macos when this assertion failure happens:
nginx version: nginx/1.21.1 (ngx_wasm_module [dev debug wasmtime])
built by clang 12.0.0 (clang-1200.0.32.29)
configure arguments: --build='ngx_wasm_module [dev debug wasmtime]' --builddir=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/work/buildroot --prefix=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/t/servroot --with-cc-opt='-O0 -ggdb3 -gdwarf' --with-ld-opt=' -Wl,-rpath,/Users/runner/hostedtoolcache/wasmtime/0.28.0/x64/lib' --with-debug --add-module=/Users/runner/work/ngx_wasm_module/ngx_wasm_module --add-dynamic-module=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/work/downloads/echo-nginx-module --add-dynamic-module=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/work/downloads/headers-more-nginx-module
The same building options have worked fine on all other platforms and still do, only the macos-latest target has this assertion failure
are you able to test this out locally and reproduce with a custom build of wasmtime?
(e.g. can you test out changes to see if the error reproduces?)
one wild guess is that a patch like this may change the output depending on whether TLS in nginx works differently or not
I don't have a mac handy but later on will do so
oh does nginx do any sort of stack switching?
and/or coroutines?
coroutines: it is entirely single-threaded, async I/O (by way of select/epoll/kqueue...); but for now we compile modules when worker process start, and enter/leave any and all wasm stacks before nginx does any sort of yielding - there's no pausing needed of any sort; unless there are other things wasmtime does I am not aware of
so there is stack switching somewhere in nginx? (or your embedding/moudule within nginx)
without async enabled (which it isn't in the C API) wasmtime itself does no stack switching
no stack switching of any sort that I can think of no; only level-triggered events and each connection/fd gets it own wasmtime instance at the moment and it seems to have worked well so far
Oh, that may have something to do with the proxy-wasm SDK now that I think of it... :thinking: (It uses thread_local!
, not sure if that is a clue...)
Last updated: Dec 23 2024 at 13:07 UTC