Component composition confusion · cargo-component

I'm trying to create components using cargo-component, compose them using wasm-compose (wasm-tools compose), and then run them using the wasmtime crate.

I have two components produced using cargo-component, both of which use some WASI interfaces.

The job-server component runs inside wasmtime just fine, so I've at least managed to point it to the WASI .wit files I lifted from the wasmtime properly...

For now I'm just trying to compose a job-worker-composed component that contains the job-server internally. Eventually I intend to be able to swap in middleware to, e.g., connect to a remote job server, and then one to decide between local or remote job server at runtime.

I'm currently stuck trying to understand this error message from wasm-compose:

error: cannot import instance with name `wasi:io/streams` for an instantiation argument of component `jeffparsons:job-server/job-server` because it conflicts with an imported instantiation argument of component `$input`

I've had a look at the code that produces this error message but I'm still failing to understand what I've done wrong.

wasm-tools compose --config components/config.yml --output local/components/job-worker-composed.wasm components/target/wasm32-wasi/debug/job_worker.wasm --verbose

I've attached copies of (what I think are) the relevant files. Can anybody point me in the right direction? I suspect I might have misunderstood several different things along the way.

Forgot some that might be relevant. Let's see if Zulip will accept uploading wasm blobs...:

Peter Huene (Oct 12 2023 at 16:43):

to confirm, the wasi_snapshot_preview1.wasm adapter you're using matches the WASI definitions from wit/deps from both projects (and all those wit files are the same)?

Peter Huene (Oct 12 2023 at 16:45):

if so, wasm-compose might not be handling the resource-based WASI definitions well yet (it was written long before resources were a thing)

Peter Huene (Oct 12 2023 at 16:48):

Peter Huene (Oct 12 2023 at 16:53):

so i think the problem stems from types_compatible, where i believe it'll always return false for resources coming from two different instance types, despite the intention of merging the two instance types together

Peter Huene (Oct 12 2023 at 16:55):

Peter Huene (Oct 12 2023 at 16:56):

and other than implementing its own compatibility check that doesn't rely on is_subtype_of from wasmparser, i don't know of an easy fix for wasm-compose

Peter Huene (Oct 12 2023 at 17:05):

@Alex Crichton currently we use a symmetric is_subtype_of check for wasm-compose's types_compatible logic, which returns false when we compare identical instance types (that have a resource type) from two different components as obviously the resource ids will not match; but for the purpose of this check, we don't care that they don't match as we're going to be merging the two definitions together into a single instance type. thoughts on what we should do here?

Peter Huene (Oct 12 2023 at 17:18):

Peter Huene (Oct 12 2023 at 17:21):

Peter Huene (Oct 12 2023 at 17:28):

assuming we do get past the type check, encoding of resources isn't implemented in wasm-compose anyway

Peter Huene (Oct 12 2023 at 17:41):

it feels like the subtype check of ComponentEntityType::Type should succeed if both the lhs and rhs were (type (sub resource)), at least

Peter Huene (Oct 12 2023 at 17:55):

unfortunately that won't solve the problem as own and borrow checks will also return false

Peter Huene (Oct 12 2023 at 18:05):

@Jeff Parsons all of the above is to say it's not a problem on your end; there's work to do in wasm-compose to support resources, which we need to implement soon as the resources-based WASI will land in Wasmtime 14

Peter Huene (Oct 12 2023 at 18:13):

my current fix idea is to maintain a "resource equivalence map" (i.e. any exported resources with the same name from instance types of the same name) in wasm-compose which could be passed into the subtype check and consulted in addition to strait-up resource id equivalence

Jeff Parsons (Oct 13 2023 at 00:43):

I have to confess I'd forgotten that I was "living in the future" by working off the 14 release branch, so I probably shouldn't have expected other tools to work seamlessly. This all makes perfect sense to me, now. Thanks so much for the detailed reply!

Next time I will try minimal reproductions of my issue based on actually released versions before bothering y'all. :man_bowing:

Alex Crichton (Oct 13 2023 at 05:41):

I would agree with the conclusion here is that resources effectively aren't supported in wasm compose and that's what needs to be fixed. Fixing this will require changes in wasmparsers API most likely to expose the necessary functions with maps for resources as you're thinking.

Deep in wasmparser there's an open_instance_type function which is more or less what's needed here. The resource bits a very tricky so it's one where we will definitely want to reuse what's already in wasmparser (probably with refactored API entry points). One other idea is that a context is built and maintained throughout type checking which internally handles maps of resources and such. This is an underbaked idea though and probably requires more thought

Richard Backhouse (Oct 23 2023 at 10:59):

FYI With the release of wasmtime 14.0.0 I'm now hitting this same issue with trying to build a component. Any ideas on a timeframe for a fix for this ?

Joel Dice (Oct 23 2023 at 13:46):

I'm working on a subset (passing handles across a component boundary when both sides import the same resource) of the problem now, and hope to have something usable this week.

Richard Backhouse (Oct 23 2023 at 14:29):

Joel Dice (Oct 24 2023 at 19:14):

first draft of resource support for `wasm-compose` by dicej · Pull Request #1261 · bytecodealliance/wasm-tools

This aims to support the specific case of passing handles to imported resource types across component boundaries. It does not include any support for guest-exported resources (as would be required...

Richard Backhouse (Nov 17 2023 at 14:45):

I just tested out the jus released 0.4.14 version of the wasm-compose crate and everything works great. Thanks Joel

Stream: cargo-component

Topic: Component composition confusion

Jeff Parsons (Oct 12 2023 at 08:53):