Stream: git-wasmtime

Topic: wasmtime / issue #10000 Spurious (?) CI failures on s390x

view this post on Zulip Wasmtime GitHub notifications bot (Jan 13 2025 at 20:34):

alexcrichton opened issue #10000:

I've seen two CI builds fail today on the s390x tests of the wasmtime-cli crate:

The first was re-queued and merged and the second is currently back in the queue. The first didn't touch anything in wasmtime-cli and the second does touch quite a bit that could affect it so it's currently an assumption of mine that the s390x-failure there was specific to s390x.

I tried running the test suite in a loop locally for a bit using s390x/qemu and so far it hasn't failed in the same way. I figured I'd open an issue to keep track of things if they happen in the future.

cc @uweigand

view this post on Zulip Wasmtime GitHub notifications bot (Jan 16 2025 at 13:34):

iii-i commented on issue #10000:

In my local runs I've seen segfaults due to an ABI problem in the capstone wrapper: https://github.com/capstone-rust/capstone-rs/pull/166. I wonder if this can be related?

view this post on Zulip Wasmtime GitHub notifications bot (Jan 16 2025 at 17:38):

abrown added the cranelift:area:s390x label to Issue #10000.

view this post on Zulip Wasmtime GitHub notifications bot (Feb 05 2025 at 00:04):

abrown commented on issue #10000:

Not sure, but I believe I saw a similar failure over in #10110: https://github.com/bytecodealliance/wasmtime/actions/runs/13124822832/job/36618948362#step:18:987.

view this post on Zulip Wasmtime GitHub notifications bot (Feb 19 2025 at 20:38):

alexcrichton commented on issue #10000:

Another failure at https://github.com/bytecodealliance/wasmtime/actions/runs/13421426149/job/37494891337

view this post on Zulip Wasmtime GitHub notifications bot (Apr 02 2025 at 16:18):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (May 30 2025 at 21:34):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Jun 02 2025 at 14:21):

alexcrichton added the ci label to Issue #10000.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 06 2025 at 20:59):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Mar 30 2026 at 18:08):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Apr 06 2026 at 14:20):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Apr 06 2026 at 16:43):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Apr 07 2026 at 12:28):

uweigand commented on issue #10000:

Hi @alexcrichton , these seems to be qemu - are we now running CI on qemu again vs. the native GitHub actions? Just wondering whether you've seen any of these segfaults on native as well ...

These qemu failures are hard to reproduce - can we at least try to capture core dumps?

view this post on Zulip Wasmtime GitHub notifications bot (Apr 07 2026 at 17:40):

alexcrichton commented on issue #10000:

Yeah I believe these are all through QEMU, not native. After the incident awhile back we switched back to QEMU and haven't gone back to native after that so far.

I'd capture core dumps if I knew how to, but I'm not sure how to capture/upload core dumps from github actions :(

I mostly just want to be sure to log these over time to track incident rate, no need to urgently fix or such.

view this post on Zulip Wasmtime GitHub notifications bot (Apr 13 2026 at 19:04):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Apr 13 2026 at 20:08):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Apr 16 2026 at 19:56):

alexcrichton commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (May 08 2026 at 15:43):

fitzgen commented on issue #10000:

another

view this post on Zulip Wasmtime GitHub notifications bot (Jun 02 2026 at 16:24):

alexcrichton commented on issue #10000:

@uweigand if you're interested and have some spare cycles, I pointed an LLM at this in the hopes that it might be able to find something and it ... sort of did. I've attached 3 different core files here I captured locally as well as the LLM summary. No real progress was made in creating a reproduction and it still takes ~20 minutes to reproduce locally with very heavy load. I was able to reproduce with the tip-of-tree QEMU in addition to the one we're using in CI to at least verify that this hasn't been fixed in the meantime.

I realize that this is a big dump of info and you're by no means obligated to sift through it. The current conclusion is that this is related to signal handlers so I'm going to go and try to disable signal handlers in CI for s390x and see if that helps. Unfortunately not much else is actionable at this time AFAIK -- I can't really make heads or tails of this summary myself. Without a standalone reproducer it's very difficult to reason about.

s390x-flaky-issue.tar.gz

core-C-qemu-master-11.0.50-not-fixed.core.gz

core-B-info-none-crosswire.core.gz

core-A-register-corruption.core.gz

view this post on Zulip Wasmtime GitHub notifications bot (Jun 02 2026 at 17:00):

uweigand commented on issue #10000:

@uweigand if you're interested and have some spare cycles, I pointed an LLM at this in the hopes that it might be able to find something and it ... sort of did.

Well, this is interesting. The report itself appears to simply conclude that if QEMU were to randomly corrupt guest register values, then that could explain the crashes ... which is true but not particularly helpful. I couldn't find any specific clue as to where and why such corruption might happen.

However, the good news is that there at least appears to be a somewhat reliable reproducer. That in itself would be quite helpful and allow me to start having a look. Do you have the scripts to reproduce - according to REPORT.md they're supposed to be in the reproducers/ and tools/ directories? If not, I guess I can build my own.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 02 2026 at 17:41):

alexcrichton commented on issue #10000:

The reproduction is effectively cargo test --test wast in a loop. CI has some settings to mirror (e.g. opt-level=2, no incremental, etc), and there's some bits about basically running a bunch of them to try to get more interesting scheduler interleavings. It still took 10-30 minutes to reproduce, however, so knowing whether something is fixed or not is unfortunately not easy.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 12 2026 at 03:57):

alexcrichton commented on issue #10000:

The hack added in #13537 didn't work as surfaced through failures on https://github.com/bytecodealliance/wasmtime/pull/13613. Reverting that in https://github.com/bytecodealliance/wasmtime/pull/13618.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 16 2026 at 01:49):

alexcrichton commented on issue #10000:

@uweigand would you be able to help hunt this down in a possibly architecture-agnostic area of QEMU? We got a different failure today which while on s390x doesn't seem to be s390x-specific. Specifically this program seems to either reproduce the CI failure or indicate something separate. Namely this reads into a guaranteed-valid buffer and gets EFAULT from qemu-user. If you're not otherwise already familiar with qemu or don't have prior experience I can also try reporting this upstream to them as well.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 24 2026 at 15:50):

uweigand commented on issue #10000:

@uweigand would you be able to help hunt this down in a possibly architecture-agnostic area of QEMU? We got a different failure today which while on s390x doesn't seem to be s390x-specific. Specifically this program seems to either reproduce the CI failure or indicate something separate. Namely this reads into a guaranteed-valid buffer and gets EFAULT from qemu-user. If you're not otherwise already familiar with qemu or don't have prior experience I can also try reporting this upstream to them as well.

Hi @alexcrichton , I've tried to reproduce this but without success so far. Can you share more data about your host environment (host architecture, OS/distro, qemu version, kernel version)?

In general, I'd be happy to have a look. But given that it seems to be a general problem it would of course also be good to report upstream.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 29 2026 at 13:52):

alexcrichton commented on issue #10000:

I'm on x86_64 Linux with 7.0.0-22-generic and Ubuntu 26.04 using qemu-aarch64 version 9.1.2 as a reproduction personally.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 30 2026 at 14:06):

uweigand commented on issue #10000:

OK, I was able to reproduce in that environment, but it still takes multiple iterations of the test. It does occur every 5 minutes or so. I've also been able to reproduce on Ubuntu 24.04 and with various qemu versions.

However, this was all on x86_64 Linux as host. On s390x Linux as host I have not been able to reproduce the problem at all.

@iii-i you mentioned you were seeing a similar issue in the past. Can you have a look at the mmrace.c test case linked above (https://github.com/bytecodealliance/wasmtime/issues/10000#issuecomment-4714102951) and verify that this is the same problem?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 01 2026 at 11:01):

iii-i commented on issue #10000:

I wrote something similar a few years back; shortly thereafter it started failing spuriously in CI and was disabled; unfortunately nobody had the time to investigate.

This seems to be a race between lockless reads in page_check_range() and pageflags_set_clear(). I have a dirty fix now and will try to make something upstreamable out of it.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 06 2026 at 12:56):

uweigand commented on issue #10000:

The reproduction is effectively cargo test --test wast in a loop. CI has some settings to mirror (e.g. opt-level=2, no incremental, etc), and there's some bits about basically running a bunch of them to try to get more interesting scheduler interleavings. It still took 10-30 minutes to reproduce, however, so knowing whether something is fixed or not is unfortunately not easy.

Getting back to the s390x specific failure, I've tried to reproduce it like you describe here, but without success even after running many hours. Either I'm doing something wrong or the timing behavior of my system is just different ...

Is there any way to extract core file plus associated executable from one of the crashes you were seeing? Maybe that would at least allow me to seeing something ...

view this post on Zulip Wasmtime GitHub notifications bot (Jul 06 2026 at 14:52):

alexcrichton commented on issue #10000:

IIRC it also took ~hours to reproduce locally and also required a ton of load/tuning to get something to reproduce due to what I presume being a really small race window. I don't have cores/executables on-hand myself, but I can try to poke a bit more once the fix for the other issue lands. I'll attempt to re-enable signals on CI and see if we got lucky and it's the same issue or if there's a second one

view this post on Zulip Wasmtime GitHub notifications bot (Jul 08 2026 at 16:36):

uweigand commented on issue #10000:

@uweigand if you're interested and have some spare cycles, I pointed an LLM at this in the hopes that it might be able to find something and it ... sort of did.

Well, this is interesting. The report itself appears to simply conclude that if QEMU were to randomly corrupt guest register values, then that could explain the crashes ... which is true but not particularly helpful. I couldn't find any specific clue as to where and why such corruption might happen.

Since I'm still unable to reproduce on my own, I went back to this and tried to extract as much info as possible from just the core files without the associated executable. At least I was able to confirm that the "randomly corrupt guest register values" theory is definitely incorrect. The AI report states:

Crashing thread PC in JIT code at l %r3,16(%r2) with r2 = 0 → load from 0x10 → SIGSEGV. Preceding instructions:

lg %r3,0(%r3) ; r3 = pointer lgr %r2,%r3 ; r2 = r3 nill %r2,65534 ; r2 &= ~1 (r3 = 0x…f8c8 is even -> no-op) cghi %r3,0 / jgnlh ; null-check r3 (non-null -> fall through) l %r3,16(%r2) ; FAULT, r2 == 0

At the fault r3 = 0x…f8c8 (even), so r2 must equal r3 (a live, mapped pointer) — but r2 = 0. Nothing rewrites r2 between its assignment and the load. On a correct CPU the load would succeed; the fault exists only because a guest register was spuriously zeroed. (This is the rarer signature.)

But this overlooks the fact that there is a second control flow path via a branch into the faulting l %r3,16(%r2) instruction, where r2 can actually be legitimately zero. This code sequence was generated from table_get_funcref, which has a cold path calling into the table_get_lazy_init_func_ref, which may return NULL.

Therefore, the l %r3,16(%r2) is supposed to be marked with a TRAP_INDIRECT_CALL_TO_NULL marker, and the SIGSEGV should be handled that way - but it isn't.

This now actually matches the second example in the AI report, where we also have a SIGSEGV on an instruction that is properly marked as potentially trapping, so this could very well be the same underlying root cause. I have not been able to track this down so far, unfortunately.

Last updated: Jul 29 2026 at 05:03 UTC