otaviopace labeled issue #3203:
Test Case
I've created a repo just for this: https://github.com/otaviopace/mac-m1-wasmtime-docker.
To reproduce the error, one should build the docker image in a linux computer, than try to run it on a Mac M1 computer.
For moving the image from one computer to the other, Docker Hub can be used, but this works the same.
Steps to Reproduce
- Build docker image on Linux computer
- Load/pull built image on a Mac M1 computer
- Run the container
Expected Results
wasmtime
shouldn't crash from running amd64 on a Mac M1 computer.Actual Results
wasmtime
crashes on the traphandlers file, like this:thread 'cool thread name' panicked at 'assertion failed: `(left == right)` left: `-1`, right: `0`: registering new sigaltstack failed', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.27.0/src/traphandlers/unix.rs:238:9
Versions and Environment
Wasmtime version or commit: 0.27.0
Operating system: Mac M1
Architecture: arm64
Extra Info
This issue has been created because we're having this problem on
graph-node
: https://github.com/graphprotocol/graph-node/issues/2325.The weird thing is that we had no issues on the M1 running a Docker image built in Linux with
wasmtime
before. We have an automated process to generate the images forgraph-node
, and at some point in time, new images started to break without us changing this Docker automation,wasmtime
version, or Docker for Mac version.To isolate the problem, I've created a repository that has a simple example with the same issue.
otaviopace opened issue #3203:
Test Case
I've created a repo just for this: https://github.com/otaviopace/mac-m1-wasmtime-docker.
To reproduce the error, one should build the docker image in a linux computer, than try to run it on a Mac M1 computer.
For moving the image from one computer to the other, Docker Hub can be used, but this works the same.
Steps to Reproduce
- Build docker image on Linux computer
- Load/pull built image on a Mac M1 computer
- Run the container
Expected Results
wasmtime
shouldn't crash from running amd64 on a Mac M1 computer.Actual Results
wasmtime
crashes on the traphandlers file, like this:thread 'cool thread name' panicked at 'assertion failed: `(left == right)` left: `-1`, right: `0`: registering new sigaltstack failed', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.27.0/src/traphandlers/unix.rs:238:9
Versions and Environment
Wasmtime version or commit: 0.27.0
Operating system: Mac M1
Architecture: arm64
Extra Info
This issue has been created because we're having this problem on
graph-node
: https://github.com/graphprotocol/graph-node/issues/2325.The weird thing is that we had no issues on the M1 running a Docker image built in Linux with
wasmtime
before. We have an automated process to generate the images forgraph-node
, and at some point in time, new images started to break without us changing this Docker automation,wasmtime
version, or Docker for Mac version.To isolate the problem, I've created a repository that has a simple example with the same issue.
cfallin commented on issue #3203:
Interesting -- it looks like this is an x86-64 binary running (via Rosetta 2) on M1 hardware? Does Docker on M1 run an x86-64 Linux kernel in a VM (as I think it works on Intel Mac hardware) or something else?
@bnjbvr is our resident has-M1-hardware-and-knows-how-to-use-it person for M1 issues -- Ben, what do you think?
alexcrichton commented on issue #3203:
If you can test out a custom build, https://github.com/bytecodealliance/wasmtime/pull/3204 may help by providing the error code that
sigaltstack
is failing with (although I don't think it will help too much here in determining a cause...)
otaviopace commented on issue #3203:
@cfallin on the questions you've pointed out
it looks like this is an x86-64 binary running (via Rosetta 2) on M1 hardware?
Yes, it the problem seems to be related to Rosetta 2 translation.
Does Docker on M1 run an x86-64 Linux kernel in a VM (as I think it works on Intel Mac hardware) or something else?
I don't think Docker runs a VM for linux, I think it probably just uses Rosetta for the translation, but I'm not completly sure.
@alexcrichton Thank you so much for the debug build, I'll try it and post the results here :smile:
otaviopace edited a comment on issue #3203:
@cfallin on the questions you've pointed out:
it looks like this is an x86-64 binary running (via Rosetta 2) on M1 hardware?
Yes, the problem seems to be related to Rosetta 2 translation.
Does Docker on M1 run an x86-64 Linux kernel in a VM (as I think it works on Intel Mac hardware) or something else?
I don't think Docker runs a VM for Linux, I think it probably just uses Rosetta for the translation, but I'm not completly sure.
@alexcrichton Thank you so much for the debug build, I'll try it and post the results here :smile:
peterhuene commented on issue #3203:
From Docker:
If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash. We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally fails to run the container. Where possible we recommend sticking to arm64 containers on M1 machines; they will also be faster.
From some Google searches, it seems that QEMU might have issues with
signaltstack
on M1 for x86-64 containers (seeing references toEINVAL
for other projects).I think this might not be a Wasmtime issue currently.
peterhuene edited a comment on issue #3203:
From Docker:
If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash. We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally fails to run the container. Where possible we recommend sticking to arm64 containers on M1 machines; they will also be faster.
and
However, attempts to run Intel-based containers on Apple Silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent. In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.
From some Google searches, it seems that QEMU might have issues with
signaltstack
on M1 for x86-64 containers (seeing references toEINVAL
for other projects, e.g. running Chrome in an amd64 container).I think this might not be an actionable Wasmtime issue; we should see if an arm64 container has similar problems.
peterhuene edited a comment on issue #3203:
From Docker:
If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash. We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally fails to run the container. Where possible we recommend sticking to arm64 containers on M1 machines; they will also be faster.
and
However, attempts to run Intel-based containers on Apple Silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent. In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.
From some Google searches, it seems that QEMU might have issues with
sigaltstack
on M1 for x86-64 containers (seeing references toEINVAL
for other projects, e.g. running Chrome in an amd64 container).I think this might not be an actionable Wasmtime issue; we should see if an arm64 container has similar problems.
peterhuene edited a comment on issue #3203:
From Docker:
If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash. We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally fails to run the container. Where possible we recommend sticking to arm64 containers on M1 machines; they will also be faster.
and
However, attempts to run Intel-based containers on Apple Silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent. In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.
From some Google searches, it seems that QEMU might have issues with
sigaltstack
on M1 for x86-64 containers (seeing references toEINVAL
for other projects, e.g. running Chrome in an amd64 container or Rabbit MQ).I think this might not be an actionable Wasmtime issue; we should see if an arm64 container has similar problems.
otaviopace commented on issue #3203:
@alexcrichton , with your debug change the log went from:
registering new sigaltstack failed
to this:
registering new sigaltstack failed: Invalid argument (os error 22)
Does this mean the Rosetta 2 translation expects a different argument for
sigalstack
?
otaviopace edited a comment on issue #3203:
@alexcrichton , with your debug change the log went from:
registering new sigaltstack failed
to this:
registering new sigaltstack failed: Invalid argument (os error 22)
Does this mean the Rosetta 2 translation expects a different argument for
sigalstack
?Edit: Just looked into similar issues @peterhuene linked and they get the same
Invalid argument (22)
result :disappointed:
otaviopace commented on issue #3203:
Unfortunetally this really isn't a problem with
wasmtime
itself, I'll close this for now, thanks a lot for the help :heart:️
otaviopace closed issue #3203:
Test Case
I've created a repo just for this: https://github.com/otaviopace/mac-m1-wasmtime-docker.
To reproduce the error, one should build the docker image in a linux computer, than try to run it on a Mac M1 computer.
For moving the image from one computer to the other, Docker Hub can be used, but this works the same.
Steps to Reproduce
- Build docker image on Linux computer
- Load/pull built image on a Mac M1 computer
- Run the container
Expected Results
wasmtime
shouldn't crash from running amd64 on a Mac M1 computer.Actual Results
wasmtime
crashes on the traphandlers file, like this:thread 'cool thread name' panicked at 'assertion failed: `(left == right)` left: `-1`, right: `0`: registering new sigaltstack failed', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-runtime-0.27.0/src/traphandlers/unix.rs:238:9
Versions and Environment
Wasmtime version or commit: 0.27.0
Operating system: Mac M1
Architecture: arm64
Extra Info
This issue has been created because we're having this problem on
graph-node
: https://github.com/graphprotocol/graph-node/issues/2325.The weird thing is that we had no issues on the M1 running a Docker image built in Linux with
wasmtime
before. We have an automated process to generate the images forgraph-node
, and at some point in time, new images started to break without us changing this Docker automation,wasmtime
version, or Docker for Mac version.To isolate the problem, I've created a repository that has a simple example with the same issue.
peterhuene commented on issue #3203:
22 is
EINVAL
.My gut tells me that the bug fixed by this [QEMU commit](
I also found this QEMU commit seems relevant as something that could lead to misinterpretation of a field likess_flags
.That fix went into QEMU 5.2; Docker For Mac seems to package QEMU 5.0.1.
peterhuene edited a comment on issue #3203:
22 is
EINVAL
.My gut tells me that the bug fixed by this QEMU commit seems relevant as something that could lead to misinterpretation of a field like
ss_flags
.That fix went into QEMU 5.2; Docker For Mac seems to package QEMU 5.0.1.
otaviopace edited a comment on issue #3203:
Unfortunetelly this really isn't a problem with
wasmtime
itself, I'll close this for now, thanks a lot for the help :heart:️
otaviopace edited a comment on issue #3203:
Unfortunetely this really isn't a problem with
wasmtime
itself, I'll close this for now, thanks a lot for the help :heart:️
otaviopace commented on issue #3203:
How do you know which version of QEMU does Docker for Mac use?
I would like to keep an eye where I can know this version bump happened.
otaviopace edited a comment on issue #3203:
How do you know which version of QEMU does Docker for Mac use?
I would like to keep an eye for when this version bump happens.
peterhuene commented on issue #3203:
I pulled it from the Docker for Mac's release notes, last upgrade mentioning QEMU was 3.2.0, so I assume it hasn't been bumped since.
bnjbvr commented on issue #3203:
(Glad I could help! thanks for the ping @cfallin, I'm happy to investigate any Mac M1 specific failures.)
otaviopace commented on issue #3203:
I've created an issue in Docker for Mac to know if they plan or not to do this version bump soon (of QEMU).
So if anyone wants to keep an eye/track this, here it is: https://github.com/docker/for-mac/issues/5919
Last updated: Dec 23 2024 at 13:07 UTC