Stream: git-wasmtime

Topic: wasmtime / issue #7469 Wasmtime `cargo test` fails on mac...

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 06:25):

cfallin opened issue #7469:

On my M1 machine, with Rust 1.73.0, having removed target/ and starting from a fresh main (630f8e32d8921a08ead7d27955686342f9f6fa06), I see:

cfallin@min:~/work/wasmtime% cargo test
[ ...]
     Compiling object v0.32.0
     Compiling wasi v0.11.0+wasi-snapshot-preview1
     Compiling serde_derive v1.0.188
     Compiling wasi-preview1-component-adapter v15.0.0 (/Users/cfallin/work/wasmtime/crates/wasi-preview1-component-adapter)
  error: failed to run custom build command for `wasi-preview1-component-adapter v15.0.0 (/Users/cfallin/work/wasmtime/crates/wasi-preview1-component-adapter)`

  Caused by:
    process didn't exit successfully: `/Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build` (signal: 9, SIGKILL: kill)
  warning: build failed, waiting for other jobs to finish...
  thread 'main' panicked at crates/test-programs/artifacts/build.rs:122:5:
  assertion failed: status.success()
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

and if I try launching build-script-build manually, I see

cfallin@min:~/work/wasmtime% /Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build
zsh: killed

A little more digging indicates

cfallin@min:~/work/wasmtime% codesign -v /Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build
/Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build: code object is not signed at all
In architecture: arm64

which is wildly confusing, because macOS/aarch64 requires all binaries to be signed (even just locally), and the way that dev-tools normally handle this is by invoking codesign after linking to add an ad-hoc local signature. Lack of any signature would explain SIGKILL from the kernel.

I'm also fairly perplexed because I believe I'm not the only one here using macOS/aarch64 (cc @alexcrichton at the very least?). The one slightly non-mainstream-path thing I'm doing is using Nix/home-manager locally (rustup and cargo are both ~/.nix-profile/bin/cargo) but local Rust development works perfectly fine otherwise.

I'm not sure what to debug from here -- but is there some custom build invocation somewhere that is doing something manually and not signing the binary?

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 06:26):

cfallin edited issue #7469:

On my M1 machine, with Rust 1.73.0, having removed target/ and starting from a fresh main (630f8e32d8921a08ead7d27955686342f9f6fa06), I see:

cfallin@min:~/work/wasmtime% cargo test
[ ...]
     Compiling object v0.32.0
     Compiling wasi v0.11.0+wasi-snapshot-preview1
     Compiling serde_derive v1.0.188
     Compiling wasi-preview1-component-adapter v15.0.0 (/Users/cfallin/work/wasmtime/crates/wasi-preview1-component-adapter)
  error: failed to run custom build command for `wasi-preview1-component-adapter v15.0.0 (/Users/cfallin/work/wasmtime/crates/wasi-preview1-component-adapter)`

  Caused by:
    process didn't exit successfully: `/Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build` (signal: 9, SIGKILL: kill)
  warning: build failed, waiting for other jobs to finish...
  thread 'main' panicked at crates/test-programs/artifacts/build.rs:122:5:
  assertion failed: status.success()
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

and if I try launching build-script-build manually, I see

cfallin@min:~/work/wasmtime% /Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build
zsh: killed

A little more digging indicates

cfallin@min:~/work/wasmtime% codesign -v /Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build
/Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build: code object is not signed at all
In architecture: arm64

which is wildly confusing, because macOS/aarch64 requires all binaries to be signed (even just locally), and the way that dev-tools normally handle this is by invoking codesign after linking to add an ad-hoc local signature. Lack of any signature would explain SIGKILL from the kernel.

I'm also fairly perplexed because I believe I'm not the only one here using macOS/aarch64 (cc @alexcrichton at the very least?). The one slightly non-mainstream-path thing I'm doing is using Nix/home-manager locally (rustup and cargo are ~/.nix-profile/bin/{rustup,cargo}) but local Rust development works perfectly fine otherwise.

I'm not sure what to debug from here -- but is there some custom build invocation somewhere that is doing something manually and not signing the binary?

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 13:14):

alexcrichton commented on issue #7469:

What macOS version are you on? I haven't had any issues myself but I also haven't updated to the latest (sonoma I think?), I'm on 13.5.1.

I'm not aware of anything codesigning related that rustc does myself, so if this is a new requirement of sonoma then that may be what's happening. Otherwise does this reproduce in a "hello world" for example with cargo run? (e.g. can any rustc-generated executable run on your system?)

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 15:49):

cfallin commented on issue #7469:

I'm on Ventura (13.6) -- haven't updated to Sonoma yet.

Codesigning has actually been a platform requirement on macOS/aarch64 since Apple Silicon rolled out; it's handled mostly transparently in either the linker or compiler drivers (I'm not sure which) by adding an ad-hoc signature to any binary one compiles locally. That's part of why I'm so surprised this is coming up!

I can build and run a Hello World fine; I've been building cilf-util and wasmtime (CLI binary) and developing with both for the past few weeks with no problem; it's just now that I thought to do a cargo test locally in the root and discovered this.

I can reproduce both on personal (M1) and work (M2) laptops; I'll try to bisect further...

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 16:04):

alexcrichton commented on issue #7469:

Weird! I dunno if something changed in 13.6, but I at least have been running tests regularly on an M2 for months now with no hiccups. I'd be pretty surprised if Nix factored meaningfully into this, so I'm as confused as you are...

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 16:12):

cfallin commented on issue #7469:

I've bisected this to f4be360648e080aae8d856967e6d2c40cccf57da (#7182), in a clean checkout and removing target each time to avoid incremental compilation weirdness. I guess that's the PR I would have suspected as well. Trying to dig into the actual failure now...

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 16:19):

cfallin closed issue #7469:

On my M1 machine, with Rust 1.73.0, having removed target/ and starting from a fresh main (630f8e32d8921a08ead7d27955686342f9f6fa06), I see:

cfallin@min:~/work/wasmtime% cargo test
[ ...]
     Compiling object v0.32.0
     Compiling wasi v0.11.0+wasi-snapshot-preview1
     Compiling serde_derive v1.0.188
     Compiling wasi-preview1-component-adapter v15.0.0 (/Users/cfallin/work/wasmtime/crates/wasi-preview1-component-adapter)
  error: failed to run custom build command for `wasi-preview1-component-adapter v15.0.0 (/Users/cfallin/work/wasmtime/crates/wasi-preview1-component-adapter)`

  Caused by:
    process didn't exit successfully: `/Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build` (signal: 9, SIGKILL: kill)
  warning: build failed, waiting for other jobs to finish...
  thread 'main' panicked at crates/test-programs/artifacts/build.rs:122:5:
  assertion failed: status.success()
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

and if I try launching build-script-build manually, I see

cfallin@min:~/work/wasmtime% /Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build
zsh: killed

A little more digging indicates

cfallin@min:~/work/wasmtime% codesign -v /Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build
/Users/cfallin/work/wasmtime/target/debug/build/test-programs-artifacts-02bef33e1d1021a1/out/release/build/wasi-preview1-component-adapter-ced036fa4eed0f20/build-script-build: code object is not signed at all
In architecture: arm64

which is wildly confusing, because macOS/aarch64 requires all binaries to be signed (even just locally), and the way that dev-tools normally handle this is by invoking codesign after linking to add an ad-hoc local signature. Lack of any signature would explain SIGKILL from the kernel.

I'm also fairly perplexed because I believe I'm not the only one here using macOS/aarch64 (cc @alexcrichton at the very least?). The one slightly non-mainstream-path thing I'm doing is using Nix/home-manager locally (rustup and cargo are ~/.nix-profile/bin/{rustup,cargo}) but local Rust development works perfectly fine otherwise.

I'm not sure what to debug from here -- but is there some custom build invocation somewhere that is doing something manually and not signing the binary?

view this post on Zulip Wasmtime GitHub notifications bot (Nov 03 2023 at 16:19):

cfallin commented on issue #7469:

I've worked it out: my home-manager setup was shadowing the system clang/LLVM with a Nix-provided clang/LLVM, and that one apparently knows how to create macOS/aarch64 binaries but does not sign them by default. (As one does?) And because home-manager is so neat and good at applying the same config everywhere, I had applied this to my work and personal laptops. Removing that shadowing and going back to system clang fixes the build. Sorry for the noise!

Last updated: Dec 23 2024 at 12:05 UTC