Stream: git-wasmtime

Topic: wasmtime / issue #8898 Unable to link a binary with wasmt...

Wasmtime GitHub notifications bot (Jul 03 2024 at 16:15):

pimeys added the bug label to Issue #8898.

Wasmtime GitHub notifications bot (Jul 03 2024 at 16:15):

pimeys opened issue #8898:

Test Case

I'm sorry I cannot find an isolated example for you to test with a small dependency tree, but in our repo, when we updated wasmtime from 21.0.1 to 22.0.0 we suddenly started having issues linking the binary when compiling against musl target.

Here's the branch:

https://github.com/grafbase/grafbase/pull/1840

And the error we see when linking:

  = note: /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(e_pow.o): warning: relocation against `_dl_x86_cpu_features' in read-only section `.text'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_ceil.o): in function `__ceil_ifunc':
          (.text+0x6): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_trunc.o): in function `__trunc_ifunc':
          (.text+0x6): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_fma.o): in function `__fma_ifunc':
          (.text+0xafd): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: (.text+0xb06): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(w_fmod.o): in function `__fmod':
          (.text+0x41): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(w_pow.o): in function `__pow':
          (.text+0x75): undefined reference to `errno'
          /usr/bin/ld: (.text+0xab): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_truncf.o): in function `__truncf_ifunc':
          (.text+0x6): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(math_err.o): in function `with_errno':
          (.text+0x3): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(math_err.o): in function `with_errno.constprop.0':
          (.text+0x13): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(e_pow.o): in function `__ieee754_pow_ifunc':
          (.text+0x746): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: (.text+0x74f): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: (.text+0x75f): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: read-only segment has dynamic relocations
          collect2: error: ld returned 1 exit status

We can currently work with the previous version, but this issue prevents us from updating to the latest runtime.

Steps to Reproduce

• fetch the branch linked in the previous chapter
• Make sure to have musl target installed from rustup
• Install musl-gcc if needed
• cargo build -p grafbase-gateway --release --target x86_64-unknown-linux-musl

Expected Results

The project compiles correctly into a working binary.

Actual Results

Linking fails as explained in the first chapter.

Versions and Environment

Wasmtime version or commit: 22.0.0

Operating system: NixOS unstable and Debian Bookworm

Architecture: x86_64-unknown-linux-musl

Extra Info

This is most probably some issue with wasmtime and some other crate together. I could not reduce a nice and clean repro as a new project. Been trying to get that the whole afternoon... But something definitely broke in the latest update of wasmtime.

Wasmtime GitHub notifications bot (Jul 03 2024 at 16:27):

bjorn3 commented on issue #8898:

For one reason or another the linker is using glibc's version of libm rather than musl's version of libm. You shouldn't need musl installed on the host system, nor use musl-gcc. Rustc already ships with a copy of musl's libc.a which includes libm.

Wasmtime GitHub notifications bot (Jul 03 2024 at 16:28):

bjorn3 edited a comment on issue #8898:

For one reason or another the linker is using glibc's version of libm rather than musl's version of libm. I'm not sure why that happens,. In any case you shouldn't need musl installed on the host system, nor use musl-gcc. Rustc already ships with a copy of musl's libc.a which includes libm.

Wasmtime GitHub notifications bot (Jul 03 2024 at 16:54):

pimeys commented on issue #8898:

This might be it. I installed a clean debian container to test this out and some dependency didn't build due to missing musl-gcc... I will get back to this when I'm on my computer again.

Wasmtime GitHub notifications bot (Jul 04 2024 at 10:08):

pimeys commented on issue #8898:

So, this is what I found out:

• wasmtime depends on ittapi, which depends on ittapi-sys
• ittapi-sys depends on cc crate, and fails to compile on a system without musl-gcc
• if installing musl-tools, the crate compiles but with version 22 of wasmtime the linking stage fails to the error described in the issue

I wonder how common these kind of issues are with musl? We do have several sys dependencies which contain C code and which compile just fine against musl. But why did the wasmtime 22 update break the linking?

Wasmtime GitHub notifications bot (Jul 04 2024 at 13:19):

pimeys commented on issue #8898:

I did a git-bisect today trying to find which commit broke the build and I think I found it:

https://github.com/bytecodealliance/wasmtime/commit/cc93e8f7f0476c89e1a2ca3a27518e0a7c4177aa#diff-0edccdae9479a6e6d11da47ad26d419b88045e14c5a6086ce7cdd8e236132ed4R38-R41

Ping @alexcrichton

Wasmtime GitHub notifications bot (Jul 04 2024 at 13:24):

pimeys edited a comment on issue #8898:

I did a git-bisect today trying to find which commit broke the build and I think I found it:

https://github.com/bytecodealliance/wasmtime/commit/cc93e8f7f0476c89e1a2ca3a27518e0a7c4177aa#diff-0edccdae9479a6e6d11da47ad26d419b88045e14c5a6086ce7cdd8e236132ed4R38-R41

Ping @alexcrichton

Removing these lines from build.rs fixes the issue. But this probably breaks something else...

Wasmtime GitHub notifications bot (Jul 08 2024 at 13:44):

alexcrichton commented on issue #8898:

I agree with @bjorn3 that this error is probably a misconfigured linker error rather than anything else. Are you configuring the linker as used by Cargo for the musl target? For example CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=musl-gcc. Usage of musl-gcc is auto-configured by the cc crate but the Rust compiler itself does not leverage the same decision making process so the Rust compiler uses cc by default which doesn't work with musl objects since then it crosses the wires by accident.

Wasmtime GitHub notifications bot (Jul 09 2024 at 13:14):

pimeys commented on issue #8898:

Hey @alexcrichton, is there some documentation how exactly the static musl compilation of a Rust project with C dependencies should look like? In general everybody seems to say you _just_ add a target with rustup and add the target to the cargo cli params. This has worked in the past seven years I've been working with Rust quite well. Now, when changing the CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER to musl-gcc, immediately when getting a crate depending on libc we get:

error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/pimeys/code/grafbase/grafbase/cli/.cargo/bin:/home/pimeys/code/grafbase/grafbase/node_modules/.bin:/nix/store/628y2lfy59bz200rcpbrh9sc2p9v83dm-cargo-insta-1.33.0/bin:/nix/store/vcpa0lb46lyk6afvaj1bqk6z3sayz4ak-cargo-nextest-0.9.72/bin:/nix/store/pxnxx19lr32mqgxkpsz4h7qixzyd84q8-cargo-component-0.13.2/bin:/nix/store/9dna4y1s34nv8glc6d7pkp3a0a9v6ijr-hey-0.1.4/bin:/nix/store/z8lzlvwbygb865qx25wrvxpfa9x8k81i-cargo-bloat-0.12.1/bin:/nix/store/fdg2c6w444j2718mi6q88n5g4lap51pa-cargo-make-0.37.12/bin:/nix/store/q0v146ks60hyw0lc9z91gg7b991rsvys-cargo-release-0.25.0/bin:/nix/store/gfsnb85cmpavq0qkxl0k62jrxmvrriz8-npm-10.8.1/bin:/nix/store/kkkk3zjfa8nm6ghdd1rpdbavzvlyixcy-semver-7.6.2/bin:/nix/store/gcsghgcyax8ri9z9hif5cgqzqvq2dib0-sd-1.0.0/bin:/nix/store/6g9n96qf1yx139xklnmy3v4xhjvjgsji-nodejs-20.12.2/bin:/nix/store/v3ws3vn47r0lm16rjr6yriq6ywv4xvwv-prettier-3.3.0/bin:/nix/store/3ws1qnjgq1rbxs96q67php6i8aygcxgg-bun-1.1.10/bin:/nix/store/qannz09m66qpcy3ny1f4nkl4ql0g71ks-openssl-3.0.13-bin/bin:/nix/store/zdvrzlvzbn9ymb0z8na50w995j8np16z-pkg-config-wrapper-0.29.2/bin:/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin:/nix/store/y74bzy6hjnc2n5d3hgq309imwfa0fpw4-rustup-1.26.0/bin:/nix/store/yrmlzr241pa9irf54fslilijnwv664c6-musl-1.2.3-dev/bin:/nix/store/bqjb40djywqk5n4dfzs5babpz7mycz04-musl-1.2.3-bin/bin:/nix/store/s25mgibyiwc2ahwpz2pbvdpj4vvngsva-pnpm-9.1.1/bin:/nix/store/k0d34jsjxzr7mg1c02vvbr616rb1m8nq-yarn-1.22.22/bin:/nix/store/hv6lq8yxrk5cb24147qaakp8isqq4pd9-cargo-about-0.6.2/bin:/nix/store/nbad47q0m0m9c5xid7zh05hiknwircbp-patchelf-0.15.0/bin:/nix/store/9bv7dcvmfcjnmg5mnqwqlq2wxfn8d7yi-gcc-wrapper-13.2.0/bin:/nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/bin:/nix/store/c2i631h8i5vcs1sqifwxfsazhwrg6wr5-glibc-2.39-52-bin/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/kln7kinji3b7sz8r50h4gn9yy6k1js9a-binutils-wrapper-2.41/bin:/nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/jjcsr5gs4qanf7ln5c6wgcq4sn75a978-findutils-4.9.0/bin:/nix/store/i34mknsjgrfyy71k2h79gda0bvagzc2j-diffutils-3.10/bin:/nix/store/5zjms21vpxlkbc0qyl5pmj2sidfmzmd7-gnused-4.9/bin:/nix/store/28gpmx3z6ss3znd7fhmrzmvk3x5lnfbk-gnugrep-3.11/bin:/nix/store/8vvkbgmnin1x2jkp7wcb2zg1p0vc4ks9-gawk-5.2.2/bin:/nix/store/rik7p68cq7yzlj5pmfpf4yv6jnrpvlgf-gnutar-1.35/bin:/nix/store/j5chw7v1x3vlmf3wmdpdb5gwh9hl0b80-gzip-1.13/bin:/nix/store/mxcq77rlan82dzpv3cgj0fh6qvv8ncil-bzip2-1.0.8-bin/bin:/nix/store/cdzpn0rdq810aknww3w9fy3wmw9ixr66-gnumake-4.4.1/bin:/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin:/nix/store/0lfxbmchigx9vs9qmrlbahcy6nxwfnj1-patch-2.7.6/bin:/nix/store/6i4xxaa812vsbli9jkq4mksdddrk27lw-xz-5.4.6-bin/bin:/nix/store/xx7x1dwybpssfhq8yikvzz38bh3yrq97-file-5.45/bin:/home/pimeys/code/grafbase/grafbase/.direnv/bin:/home/pimeys/.cargo/bin:/home/pimeys/.local/bin:/home/pimeys/.atuin/bin:/run/wrappers/bin:/home/pimeys/.nix-profile/bin:/nix/profile/bin:/home/pimeys/.local/state/nix/profile/bin:/etc/profiles/per-user/pimeys/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcmngWD1/symbols.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.build_script_build.6d0fdb83d6651bae-cgu.0.rcgu.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.441cvjjahr7ngc3d.rcgu.o" "-Wl,--as-needed" "-L" "/home/pimeys/code/grafbase/grafbase/target/release/deps" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-6a4779412a873200.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-c88c426dd6780435.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-e74540b31113a555.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-8751b61bd13c15cf.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-a8b99dba9f449259.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-f0068d76172a0372.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-c16dfaf47799564e.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-bfa26dd63e299db5.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-75a2330a693e738f.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-76779dce1f7ab63d.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-01c5cc588623cb35.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-07bb9745ec737292.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6eacdcc91004cefb.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-ae2488b58226c836.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-52acaddcaaba04c6.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-326b78eac9ecd050.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-307ebf19f0f13d30.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d9076ee5964191bf.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
  = note: /nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib(std-d5189b81a4fa4d36.std.3c8ba8ebcf555201-cgu.0.rcgu.o): undefined reference to symbol 'gnu_get_libc_version@@GLIBC_2.2.5'
          /nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6: error adding symbols: DSO missing from command line
          collect2: error: ld returned 1 exit status

I'm all in on changing our build setup, but first I'd really like to know what is the recommended setup with musl? What we see with wasmtime is the added build param to include libm really started causing trouble. Without having that include, the compilation works just fine.

Wasmtime GitHub notifications bot (Jul 09 2024 at 13:15):

pimeys edited a comment on issue #8898:

Hey @alexcrichton, is there some documentation how exactly the static musl compilation of a Rust project with C dependencies should look like? In general everybody seems to say you _just_ add a target with rustup and add the target to the cargo cli params. This has worked in the past eight years I've been working with Rust quite well. Now, when changing the CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER to musl-gcc, immediately when getting a crate depending on libc we get:

error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/pimeys/code/grafbase/grafbase/cli/.cargo/bin:/home/pimeys/code/grafbase/grafbase/node_modules/.bin:/nix/store/628y2lfy59bz200rcpbrh9sc2p9v83dm-cargo-insta-1.33.0/bin:/nix/store/vcpa0lb46lyk6afvaj1bqk6z3sayz4ak-cargo-nextest-0.9.72/bin:/nix/store/pxnxx19lr32mqgxkpsz4h7qixzyd84q8-cargo-component-0.13.2/bin:/nix/store/9dna4y1s34nv8glc6d7pkp3a0a9v6ijr-hey-0.1.4/bin:/nix/store/z8lzlvwbygb865qx25wrvxpfa9x8k81i-cargo-bloat-0.12.1/bin:/nix/store/fdg2c6w444j2718mi6q88n5g4lap51pa-cargo-make-0.37.12/bin:/nix/store/q0v146ks60hyw0lc9z91gg7b991rsvys-cargo-release-0.25.0/bin:/nix/store/gfsnb85cmpavq0qkxl0k62jrxmvrriz8-npm-10.8.1/bin:/nix/store/kkkk3zjfa8nm6ghdd1rpdbavzvlyixcy-semver-7.6.2/bin:/nix/store/gcsghgcyax8ri9z9hif5cgqzqvq2dib0-sd-1.0.0/bin:/nix/store/6g9n96qf1yx139xklnmy3v4xhjvjgsji-nodejs-20.12.2/bin:/nix/store/v3ws3vn47r0lm16rjr6yriq6ywv4xvwv-prettier-3.3.0/bin:/nix/store/3ws1qnjgq1rbxs96q67php6i8aygcxgg-bun-1.1.10/bin:/nix/store/qannz09m66qpcy3ny1f4nkl4ql0g71ks-openssl-3.0.13-bin/bin:/nix/store/zdvrzlvzbn9ymb0z8na50w995j8np16z-pkg-config-wrapper-0.29.2/bin:/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin:/nix/store/y74bzy6hjnc2n5d3hgq309imwfa0fpw4-rustup-1.26.0/bin:/nix/store/yrmlzr241pa9irf54fslilijnwv664c6-musl-1.2.3-dev/bin:/nix/store/bqjb40djywqk5n4dfzs5babpz7mycz04-musl-1.2.3-bin/bin:/nix/store/s25mgibyiwc2ahwpz2pbvdpj4vvngsva-pnpm-9.1.1/bin:/nix/store/k0d34jsjxzr7mg1c02vvbr616rb1m8nq-yarn-1.22.22/bin:/nix/store/hv6lq8yxrk5cb24147qaakp8isqq4pd9-cargo-about-0.6.2/bin:/nix/store/nbad47q0m0m9c5xid7zh05hiknwircbp-patchelf-0.15.0/bin:/nix/store/9bv7dcvmfcjnmg5mnqwqlq2wxfn8d7yi-gcc-wrapper-13.2.0/bin:/nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/bin:/nix/store/c2i631h8i5vcs1sqifwxfsazhwrg6wr5-glibc-2.39-52-bin/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/kln7kinji3b7sz8r50h4gn9yy6k1js9a-binutils-wrapper-2.41/bin:/nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/jjcsr5gs4qanf7ln5c6wgcq4sn75a978-findutils-4.9.0/bin:/nix/store/i34mknsjgrfyy71k2h79gda0bvagzc2j-diffutils-3.10/bin:/nix/store/5zjms21vpxlkbc0qyl5pmj2sidfmzmd7-gnused-4.9/bin:/nix/store/28gpmx3z6ss3znd7fhmrzmvk3x5lnfbk-gnugrep-3.11/bin:/nix/store/8vvkbgmnin1x2jkp7wcb2zg1p0vc4ks9-gawk-5.2.2/bin:/nix/store/rik7p68cq7yzlj5pmfpf4yv6jnrpvlgf-gnutar-1.35/bin:/nix/store/j5chw7v1x3vlmf3wmdpdb5gwh9hl0b80-gzip-1.13/bin:/nix/store/mxcq77rlan82dzpv3cgj0fh6qvv8ncil-bzip2-1.0.8-bin/bin:/nix/store/cdzpn0rdq810aknww3w9fy3wmw9ixr66-gnumake-4.4.1/bin:/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin:/nix/store/0lfxbmchigx9vs9qmrlbahcy6nxwfnj1-patch-2.7.6/bin:/nix/store/6i4xxaa812vsbli9jkq4mksdddrk27lw-xz-5.4.6-bin/bin:/nix/store/xx7x1dwybpssfhq8yikvzz38bh3yrq97-file-5.45/bin:/home/pimeys/code/grafbase/grafbase/.direnv/bin:/home/pimeys/.cargo/bin:/home/pimeys/.local/bin:/home/pimeys/.atuin/bin:/run/wrappers/bin:/home/pimeys/.nix-profile/bin:/nix/profile/bin:/home/pimeys/.local/state/nix/profile/bin:/etc/profiles/per-user/pimeys/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcmngWD1/symbols.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.build_script_build.6d0fdb83d6651bae-cgu.0.rcgu.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.441cvjjahr7ngc3d.rcgu.o" "-Wl,--as-needed" "-L" "/home/pimeys/code/grafbase/grafbase/target/release/deps" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-6a4779412a873200.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-c88c426dd6780435.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-e74540b31113a555.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-8751b61bd13c15cf.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-a8b99dba9f449259.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-f0068d76172a0372.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-c16dfaf47799564e.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-bfa26dd63e299db5.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-75a2330a693e738f.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-76779dce1f7ab63d.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-01c5cc588623cb35.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-07bb9745ec737292.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6eacdcc91004cefb.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-ae2488b58226c836.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-52acaddcaaba04c6.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-326b78eac9ecd050.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-307ebf19f0f13d30.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d9076ee5964191bf.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
  = note: /nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib(std-d5189b81a4fa4d36.std.3c8ba8ebcf555201-cgu.0.rcgu.o): undefined reference to symbol 'gnu_get_libc_version@@GLIBC_2.2.5'
          /nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6: error adding symbols: DSO missing from command line
          collect2: error: ld returned 1 exit status

I'm all in on changing our build setup, but first I'd really like to know what is the recommended setup with musl? What we see with wasmtime is the added build param to include libm really started causing trouble. Without having that include, the compilation works just fine.

Wasmtime GitHub notifications bot (Jul 09 2024 at 13:15):

pimeys edited a comment on issue #8898:

Hey @alexcrichton, is there some documentation how exactly the static musl compilation of a Rust project with C dependencies should look like? In general everybody seems to say you _just_ add a target with rustup and add the target to the cargo cli params. This has worked in the past eight years I've been working with Rust quite well. Now, when changing the CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER to musl-gcc, immediately when getting a crate depending on libc we get:

error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/pimeys/code/grafbase/grafbase/cli/.cargo/bin:/home/pimeys/code/grafbase/grafbase/node_modules/.bin:/nix/store/628y2lfy59bz200rcpbrh9sc2p9v83dm-cargo-insta-1.33.0/bin:/nix/store/vcpa0lb46lyk6afvaj1bqk6z3sayz4ak-cargo-nextest-0.9.72/bin:/nix/store/pxnxx19lr32mqgxkpsz4h7qixzyd84q8-cargo-component-0.13.2/bin:/nix/store/9dna4y1s34nv8glc6d7pkp3a0a9v6ijr-hey-0.1.4/bin:/nix/store/z8lzlvwbygb865qx25wrvxpfa9x8k81i-cargo-bloat-0.12.1/bin:/nix/store/fdg2c6w444j2718mi6q88n5g4lap51pa-cargo-make-0.37.12/bin:/nix/store/q0v146ks60hyw0lc9z91gg7b991rsvys-cargo-release-0.25.0/bin:/nix/store/gfsnb85cmpavq0qkxl0k62jrxmvrriz8-npm-10.8.1/bin:/nix/store/kkkk3zjfa8nm6ghdd1rpdbavzvlyixcy-semver-7.6.2/bin:/nix/store/gcsghgcyax8ri9z9hif5cgqzqvq2dib0-sd-1.0.0/bin:/nix/store/6g9n96qf1yx139xklnmy3v4xhjvjgsji-nodejs-20.12.2/bin:/nix/store/v3ws3vn47r0lm16rjr6yriq6ywv4xvwv-prettier-3.3.0/bin:/nix/store/3ws1qnjgq1rbxs96q67php6i8aygcxgg-bun-1.1.10/bin:/nix/store/qannz09m66qpcy3ny1f4nkl4ql0g71ks-openssl-3.0.13-bin/bin:/nix/store/zdvrzlvzbn9ymb0z8na50w995j8np16z-pkg-config-wrapper-0.29.2/bin:/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin:/nix/store/y74bzy6hjnc2n5d3hgq309imwfa0fpw4-rustup-1.26.0/bin:/nix/store/yrmlzr241pa9irf54fslilijnwv664c6-musl-1.2.3-dev/bin:/nix/store/bqjb40djywqk5n4dfzs5babpz7mycz04-musl-1.2.3-bin/bin:/nix/store/s25mgibyiwc2ahwpz2pbvdpj4vvngsva-pnpm-9.1.1/bin:/nix/store/k0d34jsjxzr7mg1c02vvbr616rb1m8nq-yarn-1.22.22/bin:/nix/store/hv6lq8yxrk5cb24147qaakp8isqq4pd9-cargo-about-0.6.2/bin:/nix/store/nbad47q0m0m9c5xid7zh05hiknwircbp-patchelf-0.15.0/bin:/nix/store/9bv7dcvmfcjnmg5mnqwqlq2wxfn8d7yi-gcc-wrapper-13.2.0/bin:/nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/bin:/nix/store/c2i631h8i5vcs1sqifwxfsazhwrg6wr5-glibc-2.39-52-bin/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/kln7kinji3b7sz8r50h4gn9yy6k1js9a-binutils-wrapper-2.41/bin:/nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/jjcsr5gs4qanf7ln5c6wgcq4sn75a978-findutils-4.9.0/bin:/nix/store/i34mknsjgrfyy71k2h79gda0bvagzc2j-diffutils-3.10/bin:/nix/store/5zjms21vpxlkbc0qyl5pmj2sidfmzmd7-gnused-4.9/bin:/nix/store/28gpmx3z6ss3znd7fhmrzmvk3x5lnfbk-gnugrep-3.11/bin:/nix/store/8vvkbgmnin1x2jkp7wcb2zg1p0vc4ks9-gawk-5.2.2/bin:/nix/store/rik7p68cq7yzlj5pmfpf4yv6jnrpvlgf-gnutar-1.35/bin:/nix/store/j5chw7v1x3vlmf3wmdpdb5gwh9hl0b80-gzip-1.13/bin:/nix/store/mxcq77rlan82dzpv3cgj0fh6qvv8ncil-bzip2-1.0.8-bin/bin:/nix/store/cdzpn0rdq810aknww3w9fy3wmw9ixr66-gnumake-4.4.1/bin:/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin:/nix/store/0lfxbmchigx9vs9qmrlbahcy6nxwfnj1-patch-2.7.6/bin:/nix/store/6i4xxaa812vsbli9jkq4mksdddrk27lw-xz-5.4.6-bin/bin:/nix/store/xx7x1dwybpssfhq8yikvzz38bh3yrq97-file-5.45/bin:/home/pimeys/code/grafbase/grafbase/.direnv/bin:/home/pimeys/.cargo/bin:/home/pimeys/.local/bin:/home/pimeys/.atuin/bin:/run/wrappers/bin:/home/pimeys/.nix-profile/bin:/nix/profile/bin:/home/pimeys/.local/state/nix/profile/bin:/etc/profiles/per-user/pimeys/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcmngWD1/symbols.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.build_script_build.6d0fdb83d6651bae-cgu.0.rcgu.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.441cvjjahr7ngc3d.rcgu.o" "-Wl,--as-needed" "-L" "/home/pimeys/code/grafbase/grafbase/target/release/deps" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-6a4779412a873200.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-c88c426dd6780435.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-e74540b31113a555.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-8751b61bd13c15cf.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-a8b99dba9f449259.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-f0068d76172a0372.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-c16dfaf47799564e.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-bfa26dd63e299db5.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-75a2330a693e738f.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-76779dce1f7ab63d.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-01c5cc588623cb35.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-07bb9745ec737292.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6eacdcc91004cefb.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-ae2488b58226c836.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-52acaddcaaba04c6.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-326b78eac9ecd050.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-307ebf19f0f13d30.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d9076ee5964191bf.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
  = note: /nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib(std-d5189b81a4fa4d36.std.3c8ba8ebcf555201-cgu.0.rcgu.o): undefined reference to symbol 'gnu_get_libc_version@@GLIBC_2.2.5'
          /nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6: error adding symbols: DSO missing from command line
          collect2: error: ld returned 1 exit status

I'm all in on changing our build setup (probably even writing a blog post about that), but first I'd really like to know what is the recommended setup with musl? What we see with wasmtime is the added build param to include libm really started causing trouble. Without having that include, the compilation works just fine.

Wasmtime GitHub notifications bot (Jul 09 2024 at 14:19):

alexcrichton commented on issue #8898:

Unfortunately the linker configuration didn't seem to apply, in the error message you can see it's still executing cc. The solution is to get that plumbed correctly so rustc invokes musl-gcc as a linker instead of cc.

What we see with wasmtime is the added build param to include libm really started causing trouble

Cross compliation is not always easy or obvious. If your setup was executing cc before then it was incorrect. If it happened to work without -lm then that's basically just by luck. Cross compilation requires understanding error messages that are totally unrelated to the actual problem at hand unfortunately, such as this one.

Wasmtime GitHub notifications bot (Jul 09 2024 at 16:34):

pimeys commented on issue #8898:

Ok, fair. I think the plan is to just utilize Alpine and not bother with cross compilation. Did a full build using docker for the project and it spat out a fully functional binary without any issues.

Last updated: Oct 23 2024 at 20:03 UTC