Stream: git-wasmtime

Topic: wasmtime / issue #3203 Mac M1 Docker crash

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 16:37):

otaviopace labeled issue #3203:

Test Case

I've created a repo just for this: https://github.com/otaviopace/mac-m1-wasmtime-docker.

To reproduce the error, one should build the docker image in a linux computer, than try to run it on a Mac M1 computer.

For moving the image from one computer to the other, Docker Hub can be used, but this works the same.

Steps to Reproduce

Expected Results

wasmtime shouldn't crash from running amd64 on a Mac M1 computer.

Actual Results

wasmtime crashes on the traphandlers file, like this:

thread 'cool thread name' panicked at 'assertion failed: `(left == right)`
  left: `-1`,
 right: `0`: registering new sigaltstack failed', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.27.0/src/traphandlers/unix.rs:238:9

Versions and Environment

Wasmtime version or commit: 0.27.0

Operating system: Mac M1

Architecture: arm64

Extra Info

This issue has been created because we're having this problem on graph-node: https://github.com/graphprotocol/graph-node/issues/2325.

The weird thing is that we had no issues on the M1 running a Docker image built in Linux with wasmtime before. We have an automated process to generate the images for graph-node, and at some point in time, new images started to break without us changing this Docker automation, wasmtime version, or Docker for Mac version.

To isolate the problem, I've created a repository that has a simple example with the same issue.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 16:37):

otaviopace opened issue #3203:

Test Case

I've created a repo just for this: https://github.com/otaviopace/mac-m1-wasmtime-docker.

To reproduce the error, one should build the docker image in a linux computer, than try to run it on a Mac M1 computer.

For moving the image from one computer to the other, Docker Hub can be used, but this works the same.

Steps to Reproduce

Expected Results

wasmtime shouldn't crash from running amd64 on a Mac M1 computer.

Actual Results

wasmtime crashes on the traphandlers file, like this:

thread 'cool thread name' panicked at 'assertion failed: `(left == right)`
  left: `-1`,
 right: `0`: registering new sigaltstack failed', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.27.0/src/traphandlers/unix.rs:238:9

Versions and Environment

Wasmtime version or commit: 0.27.0

Operating system: Mac M1

Architecture: arm64

Extra Info

This issue has been created because we're having this problem on graph-node: https://github.com/graphprotocol/graph-node/issues/2325.

The weird thing is that we had no issues on the M1 running a Docker image built in Linux with wasmtime before. We have an automated process to generate the images for graph-node, and at some point in time, new images started to break without us changing this Docker automation, wasmtime version, or Docker for Mac version.

To isolate the problem, I've created a repository that has a simple example with the same issue.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 16:51):

cfallin commented on issue #3203:

Interesting -- it looks like this is an x86-64 binary running (via Rosetta 2) on M1 hardware? Does Docker on M1 run an x86-64 Linux kernel in a VM (as I think it works on Intel Mac hardware) or something else?

@bnjbvr is our resident has-M1-hardware-and-knows-how-to-use-it person for M1 issues -- Ben, what do you think?

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 16:55):

alexcrichton commented on issue #3203:

If you can test out a custom build, https://github.com/bytecodealliance/wasmtime/pull/3204 may help by providing the error code that sigaltstack is failing with (although I don't think it will help too much here in determining a cause...)

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 17:11):

otaviopace commented on issue #3203:

@cfallin on the questions you've pointed out

it looks like this is an x86-64 binary running (via Rosetta 2) on M1 hardware?

Yes, it the problem seems to be related to Rosetta 2 translation.

Does Docker on M1 run an x86-64 Linux kernel in a VM (as I think it works on Intel Mac hardware) or something else?

I don't think Docker runs a VM for linux, I think it probably just uses Rosetta for the translation, but I'm not completly sure.

@alexcrichton Thank you so much for the debug build, I'll try it and post the results here :smile:

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 17:15):

otaviopace edited a comment on issue #3203:

@cfallin on the questions you've pointed out:

it looks like this is an x86-64 binary running (via Rosetta 2) on M1 hardware?

Yes, the problem seems to be related to Rosetta 2 translation.

Does Docker on M1 run an x86-64 Linux kernel in a VM (as I think it works on Intel Mac hardware) or something else?

I don't think Docker runs a VM for Linux, I think it probably just uses Rosetta for the translation, but I'm not completly sure.

@alexcrichton Thank you so much for the debug build, I'll try it and post the results here :smile:

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 18:42):

peterhuene commented on issue #3203:

From Docker:

If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash.
We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally
fails to run the container. Where possible we recommend sticking to arm64 containers on M1
machines; they will also be faster.

From some Google searches, it seems that QEMU might have issues with signaltstack on M1 for x86-64 containers (seeing references to EINVAL for other projects).

I think this might not be a Wasmtime issue currently.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:03):

peterhuene edited a comment on issue #3203:

From Docker:

If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash.
We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally
fails to run the container. Where possible we recommend sticking to arm64 containers on M1
machines; they will also be faster.

and

However, attempts to run Intel-based containers on Apple Silicon machines under emulation can
crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs
(inotify) do not work under qemu emulation. Even when the containers do run correctly under
emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as
“best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever
possible, and encouraging container authors to produce arm64, or multi-arch, versions of their
containers. We expect this issue to become less common over time, as more and more images are
rebuilt supporting multiple architectures.

From some Google searches, it seems that QEMU might have issues with signaltstack on M1 for x86-64 containers (seeing references to EINVAL for other projects, e.g. running Chrome in an amd64 container).

I think this might not be an actionable Wasmtime issue; we should see if an arm64 container has similar problems.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:06):

peterhuene edited a comment on issue #3203:

From Docker:

If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash.
We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally
fails to run the container. Where possible we recommend sticking to arm64 containers on M1
machines; they will also be faster.

and

However, attempts to run Intel-based containers on Apple Silicon machines under emulation can
crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs
(inotify) do not work under qemu emulation. Even when the containers do run correctly under
emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as
“best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever
possible, and encouraging container authors to produce arm64, or multi-arch, versions of their
containers. We expect this issue to become less common over time, as more and more images are
rebuilt supporting multiple architectures.

From some Google searches, it seems that QEMU might have issues with sigaltstack on M1 for x86-64 containers (seeing references to EINVAL for other projects, e.g. running Chrome in an amd64 container).

I think this might not be an actionable Wasmtime issue; we should see if an arm64 container has similar problems.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:08):

peterhuene edited a comment on issue #3203:

From Docker:

If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash.
We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally
fails to run the container. Where possible we recommend sticking to arm64 containers on M1
machines; they will also be faster.

and

However, attempts to run Intel-based containers on Apple Silicon machines under emulation can
crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs
(inotify) do not work under qemu emulation. Even when the containers do run correctly under
emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as
“best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever
possible, and encouraging container authors to produce arm64, or multi-arch, versions of their
containers. We expect this issue to become less common over time, as more and more images are
rebuilt supporting multiple architectures.

From some Google searches, it seems that QEMU might have issues with sigaltstack on M1 for x86-64 containers (seeing references to EINVAL for other projects, e.g. running Chrome in an amd64 container or Rabbit MQ).

I think this might not be an actionable Wasmtime issue; we should see if an arm64 container has similar problems.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:32):

otaviopace commented on issue #3203:

@alexcrichton , with your debug change the log went from:

registering new sigaltstack failed

to this:

registering new sigaltstack failed: Invalid argument (os error 22)

Does this mean the Rosetta 2 translation expects a different argument for sigalstack?

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:34):

otaviopace edited a comment on issue #3203:

@alexcrichton , with your debug change the log went from:

registering new sigaltstack failed

to this:

registering new sigaltstack failed: Invalid argument (os error 22)

Does this mean the Rosetta 2 translation expects a different argument for sigalstack?

Edit: Just looked into similar issues @peterhuene linked and they get the same Invalid argument (22) result :disappointed:

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:36):

otaviopace commented on issue #3203:

Unfortunetally this really isn't a problem with wasmtime itself, I'll close this for now, thanks a lot for the help :heart:️

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:36):

otaviopace closed issue #3203:

Test Case

I've created a repo just for this: https://github.com/otaviopace/mac-m1-wasmtime-docker.

To reproduce the error, one should build the docker image in a linux computer, than try to run it on a Mac M1 computer.

For moving the image from one computer to the other, Docker Hub can be used, but this works the same.

Steps to Reproduce

Expected Results

wasmtime shouldn't crash from running amd64 on a Mac M1 computer.

Actual Results

wasmtime crashes on the traphandlers file, like this:

thread 'cool thread name' panicked at 'assertion failed: `(left == right)`
  left: `-1`,
 right: `0`: registering new sigaltstack failed', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.27.0/src/traphandlers/unix.rs:238:9

Versions and Environment

Wasmtime version or commit: 0.27.0

Operating system: Mac M1

Architecture: arm64

Extra Info

This issue has been created because we're having this problem on graph-node: https://github.com/graphprotocol/graph-node/issues/2325.

The weird thing is that we had no issues on the M1 running a Docker image built in Linux with wasmtime before. We have an automated process to generate the images for graph-node, and at some point in time, new images started to break without us changing this Docker automation, wasmtime version, or Docker for Mac version.

To isolate the problem, I've created a repository that has a simple example with the same issue.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:37):

peterhuene commented on issue #3203:

22 is EINVAL.

My gut tells me that the bug fixed by this [QEMU commit](
I also found this QEMU commit seems relevant as something that could lead to misinterpretation of a field like ss_flags.

That fix went into QEMU 5.2; Docker For Mac seems to package QEMU 5.0.1.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:37):

peterhuene edited a comment on issue #3203:

22 is EINVAL.

My gut tells me that the bug fixed by this QEMU commit seems relevant as something that could lead to misinterpretation of a field like ss_flags.

That fix went into QEMU 5.2; Docker For Mac seems to package QEMU 5.0.1.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:37):

otaviopace edited a comment on issue #3203:

Unfortunetelly this really isn't a problem with wasmtime itself, I'll close this for now, thanks a lot for the help :heart:️

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:37):

otaviopace edited a comment on issue #3203:

Unfortunetely this really isn't a problem with wasmtime itself, I'll close this for now, thanks a lot for the help :heart:️

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:40):

otaviopace commented on issue #3203:

How do you know which version of QEMU does Docker for Mac use?

I would like to keep an eye where I can know this version bump happened.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:40):

otaviopace edited a comment on issue #3203:

How do you know which version of QEMU does Docker for Mac use?

I would like to keep an eye for when this version bump happens.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 18 2021 at 19:41):

peterhuene commented on issue #3203:

I pulled it from the Docker for Mac's release notes, last upgrade mentioning QEMU was 3.2.0, so I assume it hasn't been bumped since.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 19 2021 at 09:04):

bnjbvr commented on issue #3203:

(Glad I could help! thanks for the ping @cfallin, I'm happy to investigate any Mac M1 specific failures.)

view this post on Zulip Wasmtime GitHub notifications bot (Aug 19 2021 at 12:29):

otaviopace commented on issue #3203:

I've created an issue in Docker for Mac to know if they plan or not to do this version bump soon (of QEMU).

So if anyone wants to keep an eye/track this, here it is: https://github.com/docker/for-mac/issues/5919

Last updated: Nov 22 2024 at 16:03 UTC