Thibault Charbonnier (Jul 07 2021 at 17:41): Thibault Charbonnier (Jul 07 2021 at 17:41):Hi there! I am getting the following assertion failure on my project's CI (macos) with wasmtime 0.27.0 on module instantiation. Will bump to 0.28.0 and investigate further, but I thought I'd report it here first just in case:
# thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
# left: `268435466`,
# right: `0`: failed to set thread exception port', crates/runtime/src/traphandlers/macos.rs:436:9
Ping on this - this still happens everyday on macOS (GitHub actions) with wasmtime v0.28.0; does nobody else use wasmtime on macos and encounter this?
bjorn3 (Jul 26 2021 at 20:41):macOS os tested on CI.
bjorn3 (Jul 26 2021 at 20:42):Just a guess, but is your CI restricted in any way? Or running an older macOS version?
bjorn3 (Jul 26 2021 at 20:47):The error code is 0x1000000a by the way.
bjorn3 (Jul 26 2021 at 20:49):This corresponds to MACH_SEND_INVALID_RIGHT
bjorn3 (Jul 26 2021 at 20:49): Alex Crichton (Jul 26 2021 at 21:04):@Thibault Charbonnier sorry I don't think I saw this when it was originally posted! Can you describe your setup with macOS and CI a bit more? It looks like the thread_set_exception_ports call is failing, but I'm not sure why such a call would fail myself (I'm not super familiar with mach)
We lifted most of our mach bits from spidermonkey and looking around there if this function call fails it basically disables wasm execution in firefox. In that sense this seems like it's not really expected to fail, so I'm not sure why it's failing!
As mentioned, though, we do run CI on macOS, so everything should work. If previous versions of Wasmtime work for you do you think you'd be able to run a bisection to figure out the failing commit?
Thibault Charbonnier (Jul 26 2021 at 21:07):No, it is macos-latest and the official release artifacts for v0.28.0 + clang
Alex Crichton (Jul 26 2021 at 21:09):hm ok so this is on GitHub Actions CI? Can you describe how you're embedding Wasmtime? (e.g. language and runtime and such)
Thibault Charbonnier (Jul 26 2021 at 21:11):This is wasmtime inside of Nginx, running the proxy-wasm Rust SDK
Alex Crichton (Jul 26 2021 at 21:12):is the code for this public? (e.g. could I repro locally?)
Thibault Charbonnier (Jul 26 2021 at 21:15):No it is not, sorry
Alex Crichton (Jul 26 2021 at 21:16):ah ok no worries, are you able to help out in narrowing this down and/or debugging it? Without a reproduction locally I'm not sure how this could happen so I'm not sure what's going on :(
Thibault Charbonnier (Jul 26 2021 at 21:26):This is how nginx is built/link to wasmtime on macos when this assertion failure happens:
Thibault Charbonnier (Jul 26 2021 at 21:26):nginx version: nginx/1.21.1 (ngx_wasm_module [dev debug wasmtime])
built by clang 12.0.0 (clang-1200.0.32.29)
configure arguments: --build='ngx_wasm_module [dev debug wasmtime]' --builddir=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/work/buildroot --prefix=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/t/servroot --with-cc-opt='-O0 -ggdb3 -gdwarf' --with-ld-opt=' -Wl,-rpath,/Users/runner/hostedtoolcache/wasmtime/0.28.0/x64/lib' --with-debug --add-module=/Users/runner/work/ngx_wasm_module/ngx_wasm_module --add-dynamic-module=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/work/downloads/echo-nginx-module --add-dynamic-module=/Users/runner/work/ngx_wasm_module/ngx_wasm_module/work/downloads/headers-more-nginx-module
The same building options have worked fine on all other platforms and still do, only the macos-latest target has this assertion failure
Alex Crichton (Jul 26 2021 at 21:28):are you able to test this out locally and reproduce with a custom build of wasmtime?
Alex Crichton (Jul 26 2021 at 21:28):(e.g. can you test out changes to see if the error reproduces?)
Alex Crichton (Jul 26 2021 at 21:30):one wild guess is that a patch like this may change the output depending on whether TLS in nginx works differently or not
Thibault Charbonnier (Jul 26 2021 at 21:31):I don't have a mac handy but later on will do so
Alex Crichton (Jul 26 2021 at 21:32):oh does nginx do any sort of stack switching?
Alex Crichton (Jul 26 2021 at 21:32):and/or coroutines?
Thibault Charbonnier (Jul 26 2021 at 21:38):coroutines: it is entirely single-threaded, async I/O (by way of select/epoll/kqueue...); but for now we compile modules when worker process start, and enter/leave any and all wasm stacks before nginx does any sort of yielding - there's no pausing needed of any sort; unless there are other things wasmtime does I am not aware of
Alex Crichton (Jul 26 2021 at 21:39):so there is stack switching somewhere in nginx? (or your embedding/moudule within nginx)
without async enabled (which it isn't in the C API) wasmtime itself does no stack switching
Thibault Charbonnier (Jul 26 2021 at 21:56):no stack switching of any sort that I can think of no; only level-triggered events and each connection/fd gets it own wasmtime instance at the moment and it seems to have worked well so far
Thibault Charbonnier (Jul 26 2021 at 21:58):Oh, that may have something to do with the proxy-wasm SDK now that I think of it... :thinking: (It uses thread_local!, not sure if that is a clue...)
Last updated: Nov 22 2024 at 17:03 UTC