Jeff Parsons (Apr 16 2024 at 04:08): Jeff Parsons (Apr 16 2024 at 04:08):I originally mentioned this over on the aws-sdk-rust issue tracker and @Landon James suggested moving the conversation over here.
When I add aws-smithy-wasm as a dependency I end up with this output in cargo-component build:
~/Projects/wasm-aws-sdk/aws-sdk-ec2> cargo-component build
Finished dev [unoptimized + debuginfo] target(s) in 0.03s
error: module requires an import interface named `env`
Stack backtrace:
0: anyhow::error::<impl anyhow::Error>::msg
1: anyhow::__private::format_err.6620
2: wit_component::encoding::world::ComponentWorld::new
3: wit_component::encoding::ComponentEncoder::encode
4: cargo_component::run_cargo_command::{{closure}}
5: cargo_component::main::{{closure}}
6: cargo_component::main
7: std::sys_common::backtrace::__rust_begin_short_backtrace
8: main
9: __libc_start_call_main
at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
10: __libc_start_main_impl
at ./csu/../csu/libc-start.c:392:3
11: _start
I have to admit I don't entirely understand how feature selection works, but the output of cargo-tree looks slightly suspicious to me:
├── aws-smithy-wasm v0.1.2
│ ├── aws-smithy-runtime-api feature "default" (*)
│ ├── aws-smithy-runtime-api feature "http-1x" (*)
│ ├── aws-smithy-types feature "default" (*)
│ ├── bytes feature "default" (*)
│ ├── http feature "default" (*)
│ ├── tracing feature "default" (*)
│ ├── aws-smithy-http feature "default" (*)
│ └── wasi feature "default"
│ ├── wasi v0.12.1+wasi-0.2.0
│ │ └── wit-bindgen feature "realloc"
│ │ └── wit-bindgen v0.19.2
│ │ └── bitflags feature "default"
│ │ └── bitflags v2.5.0
│ └── wasi feature "std"
│ └── wasi v0.12.1+wasi-0.2.0 (*)
And aws-smithy-wasm's Cargo.toml looks like it's pulling in various other crates with their default features enabled... which might not be intended?
Here are my versions of things:
~/Projects/wasm-aws-sdk/aws-sdk-ec2> cargo component --version
cargo-component-component 0.11.0 (40320a9 2024-04-15 wasi:040ec92)
~/Projects/wasm-aws-sdk/aws-sdk-ec2> rustc --version
rustc 1.76.0 (07dca489a 2024-02-04)
Am I just holding it wrong?
Jeff Parsons (Apr 16 2024 at 04:11):Full repro:
/tmp> cargo component new bar
Created binary (application) `bar` package
Updated manifest of package `bar`
Generated source file `src/main.rs`
/tmp> cd bar/
/tmp/bar> cargo add --no-default-features aws-smithy-wasm
Updating crates.io index
Adding aws-smithy-wasm v0.1.2 to dependencies.
Updating crates.io index
/tmp/bar took 3s> cargo component build
Generating bindings for bar (src/bindings.rs)
Compiling proc-macro2 v1.0.80
Compiling pin-project-lite v0.2.14
Compiling unicode-ident v1.0.12
Compiling bytes v1.6.0
Compiling autocfg v1.2.0
Compiling itoa v1.0.11
Compiling fnv v1.0.7
Compiling powerfmt v0.2.0
Compiling pin-utils v0.1.0
Compiling futures-core v0.3.30
Compiling futures-task v0.3.30
Compiling either v1.11.0
Compiling outref v0.5.1
Compiling once_cell v1.19.0
Compiling num-conv v0.1.0
Compiling time-core v0.1.2
Compiling vsimd v0.8.0
Compiling tokio v1.37.0
Compiling ryu v1.0.17
Compiling wit-bindgen v0.19.2
Compiling futures-util v0.3.30
Compiling bitflags v2.5.0
Compiling wit-bindgen-rt v0.24.0
Compiling percent-encoding v2.3.1
Compiling deranged v0.3.11
Compiling tracing-core v0.1.32
Compiling num-traits v0.2.18
Compiling wasi v0.12.1+wasi-0.2.0
Compiling base64-simd v0.8.0
Compiling http v0.2.12
Compiling bytes-utils v0.1.4
Compiling http v1.1.0
Compiling quote v1.0.36
Compiling syn v2.0.59
Compiling time v0.3.36
Compiling http-body v0.4.6
Compiling num-integer v0.1.46
Compiling aws-smithy-async v1.2.1
Compiling aws-smithy-types v1.1.8
Compiling tracing-attributes v0.1.27
Compiling tracing v0.1.40
Compiling aws-smithy-runtime-api v1.4.0
Compiling aws-smithy-http v0.60.7
Compiling aws-smithy-wasm v0.1.2
Compiling bar v0.1.0 (/tmp/bar)
Finished dev [unoptimized + debuginfo] target(s) in 4.97s
Creating component target/wasm32-wasi/debug/bar.wasm
error: module requires an import interface named `env`
Stack backtrace:
0: anyhow::error::<impl anyhow::Error>::msg
1: anyhow::__private::format_err.6620
2: wit_component::encoding::world::ComponentWorld::new
3: wit_component::encoding::ComponentEncoder::encode
4: cargo_component::run_cargo_command::{{closure}}
5: cargo_component::main::{{closure}}
6: cargo_component::main
7: std::sys_common::backtrace::__rust_begin_short_backtrace
8: main
9: __libc_start_call_main
at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
10: __libc_start_main_impl
at ./csu/../csu/libc-start.c:392:3
11: _start
This chunk:
Finished dev [unoptimized + debuginfo] target(s) in 4.97s
Creating component target/wasm32-wasi/debug/bar.wasm
error: module requires an import interface named `env`
Makes it seem like the failure is in embedding the component metadata rather than compiling the project, so I don't think that the issue has to do with un-needed features in the dependencies of aws-smithy-wasm.
I'm wondering what version of cargo-component you are using and if that might be older? Or potentially the smithy crate's dependency on the wasi crate is out of date? I'll try to do some more digging into this tomorrow.
Landon James (Apr 16 2024 at 04:20):In the meantime, there is a working example of this buried deep in the Rust SDK's canary package. https://github.com/smithy-lang/smithy-rs/tree/main/tools/ci-cdk/canary-wasm
Landon James (Apr 16 2024 at 04:40):Weird, I am able to repro your issue, but I can also build
one of my projects using it just fine and I’m not sure what the difference is. The SDK does rely on env to work in WASI (mostly to load credentials from environment variables), but I don’t think you should have to do anything to link it in
Alex Crichton (Apr 16 2024 at 14:16):Ah the env module is the default module emitted by LLD for unresolved symbols, so what you're hitting is something that should be a link-time error but ends up getting papered over for tooling to fail later down the road. I'd recomend wasm-tools print-ing the file and looking at what the symbol is under the env module as that'll help provide a clue as to what library is pulling in a symbol that needs adjusting.
Orthogonally and independently it'd probably be good to improve the error message here to say more than just env as well!
Landon James (Apr 16 2024 at 15:58):Printing the cargo-component init (with the aws-smithy-wasm import) file shows the below:
(import "env" "cabi_realloc_wit_bindgen_0_19_2" (func $cabi_realloc_wit_bindgen_0_19_2 (;4;) (type 7)))
So it seems like it should be getting env from wit-bindgen? Full wat file included if it helps.
Peter Huene (Apr 16 2024 at 16:02):Do you have a dep on wit-bindgen somewhere? cargo-component projects currently should only have a dep on wit-bindgen-rt
Landon James (Apr 16 2024 at 16:04):aws-smithy-wasm depends on the wasi crate which depends directly on wit-bindgen, so it is in there transitively, but not directly
Peter Huene (Apr 16 2024 at 16:05):that should be fine then, not sure why it isn't linking in realloc though
Lann Martin (Apr 16 2024 at 16:06):Possibly-unrelated, what's up with cabi_realloc_wit_bindgen_0_19_2? I saw this the other day and didn't recognize the suffix.
Landon James (Apr 16 2024 at 16:08):I think that suffix is due to another issue I encountered a few weeks ago:
issue: https://github.com/bytecodealliance/wit-bindgen/issues/849
fix: https://github.com/bytecodealliance/wit-bindgen/pull/851
Peter Huene (Apr 16 2024 at 16:08):i think it is an attempt to allow linkage against multiple wit-bindgen deps of differing versions; alex or dan have the context there iirc
Dan Gohman (Apr 16 2024 at 17:16):Yeah. When you have multiple expansions of the wit-bindgen bindings linked into the same program, it's a way to make their cabi_reallocs not collide.
Alex Crichton (Apr 16 2024 at 17:44):hm the fact that this symbol is imported indicates a bug in something in wit-bindgen/wasi/etc for sure
Alex Crichton (Apr 16 2024 at 17:45):Given the repro here I'll try to dig into this this week and see if I can't figure out what's going wrong
Landon James (Apr 16 2024 at 17:55):I just blew out my Cargo.lock and rebuilt my previously working project and hit the same error, so this is likely something fairly new. I think I first built that project about 2 weeks ago, so probably something in that timeframe?
Alex Crichton (Apr 16 2024 at 22:26):I've reduced this to:
[package]
name = "bar"
version = "0.1.0"
edition = "2021"
[dependencies]
wit-bindgen-rt1 = { version = "0.24.0", package = 'wit-bindgen-rt' }
wit-bindgen-rt2 = { version = "0.23.0", package = 'wit-bindgen-rt' }
and
fn main() {
wit_bindgen_rt1::maybe_link_cabi_realloc();
}
My current suspicion is that both versions of wit-bindgen use libwit_bindgen_cabi_realloc.a as a library name. Still digging though
Alex Crichton (Apr 16 2024 at 23:00):@Jeff Parsons do you have a larger reproduction of this issue?
I ask because your current issue as-stated above can be fixed with:
echo 'extern crate aws_smithy_wasm;' >> src/main.rs
I opened https://github.com/bytecodealliance/wit-bindgen/pull/928 to fix that specific issue, but linking is often weird and surprising. If you have a larger reproduction I'd like to test that the fix in that PR fixes the issue as well in case there's something else I'm missing
Alex Crichton (Apr 16 2024 at 23:00):b/c the fix I have relies on a crate not actually being used, which seems unlikely in a larger reproduction situation
Jeff Parsons (Apr 16 2024 at 23:09):Wow, that is wild. I don't have one at hand, but when I get a chance I will try to make a "realistic use" repro.
BTW, the context for all this was roughly:
aws-sdk-ec2 builds are killing me (and my machine, and my colleagues).I'm unlikely to be near a computer today, but when I am I will push that "realistic use" repro somewhere and link it here.
Thanks, y'all! :sparkling_heart:
Jeff Parsons (Apr 16 2024 at 23:13):(Also, this feels like a great excuse to sneak Wasm in the door at work. :wink:)
Alex Crichton (Apr 16 2024 at 23:22):no worries! This may not actually reproduce in a larger context which in that case I'll publish a fix once it's merged
Landon James (Apr 16 2024 at 23:29):Sweet, once you do get around to having it work on your machine feel free to reach out if you have anymore questions about the AWS Rust SDK with wasm bits! As far as I know I'm the only one using it so far so would love to get some extra eyes on it and find any lingering bugs I might have missed.
Jeff Parsons (Apr 17 2024 at 01:39):I can confirm that
let wasi_http_client = aws_smithy_wasm::wasi::WasiHttpClientBuilder::new().build(); then this error no longer occurs.extern crate aws_smithy_wasm;also makes it go away.I guess I hit this one when nobody else did because... I work in a strange way sometimes. :)
Working-ish code is here: https://github.com/jeffparsons/aws-wasm-components. It can talk to an AWS server! :tada: (Haven't gotten auth working yet, but that's off-topic.)
Alex Crichton (Apr 17 2024 at 01:56):Ok sounds good, and thanks regardless for the report!
Last updated: Dec 23 2024 at 12:05 UTC