Stream: warg

Topic: publishing libary packages


view this post on Zulip Paul Scheduikat (Feb 07 2024 at 12:39):

Do I understand correctly that is expected for each component implementing a world from a wit file to be it's own package in warg, and the wit file also being published to a separate package encoded in it's own wasm file?
If that's the case, is there a standard way in warg to describe the relationship between the implementation components and the wit component?

Or is there a way to merge the implementation .wasm components with the wasm encoded wit?

Or can a package consist of multiple wasm files?

view this post on Zulip Paul Scheduikat (Feb 07 2024 at 13:05):

Has been helpful writing this down.

Library users only ever import the interface definitions from the wit component.
Implementations are only specified when composing components.

Correct?

view this post on Zulip Paul Scheduikat (Feb 07 2024 at 13:27):

What happens when specifying conflicting implementations i.e. two components each specify an implementation for a subset of the interfaces, but these subsets intersect?

view this post on Zulip Calvin Prewitt (Feb 07 2024 at 18:12):

Great questions! Some of these answers are still being figured out. Here are some partial answers.

A single Wasm binary is what you publish with Warg. However, that binary could be composed of other components that either are bundled in the binary or just referenced with an import statement that needs to be resolved by the registry. So a single Wasm binary could be a composition of multiple Wasm binaries.

A Wasm component binary could be just the binary representation of a WIT text file. Or it could have an implementation of those interfaces. Both are publishable components.

You might want to publish interfaces by themselves if you want them to be reused and targeted by multiple component implementations.

view this post on Zulip Lann Martin (Feb 08 2024 at 13:59):

If a wit file is intended to have multiple implementations (like the WASI specs) then you would publish it as a standalone binary-encoded wit package. If you just want a representation of a particular component's imports/exports then you don't need a separate package; components are self-describing in that way.

view this post on Zulip Ralph (Feb 08 2024 at 14:16):

I think I missed this, but do we have a way to create a "binary-encoded wit package" right now? with no implementations, just the module with the world(s)?

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 14:19):

@Lann Martin
How would I import a "implementation" without a separate wit component defining it's interfaces?

package example:add;

interface add {
    add: func(a: u32, b: u32) -> u32;
}

world adder {
    export add;
}

When I create a component with cargo component which confines to the adder world, I then get this fairly unhelpful wit back.

$ wasm-tools component wit target/wasm32-wasi/release/adder.wasm
package root:component;

world root {
  export example:add/add;
}

view this post on Zulip Calvin Prewitt (Feb 08 2024 at 14:20):

Ralph said:

I think I missed this, but do we have a way to create a "binary-encoded wit package" right now? with no implementations, just the module with the world(s)?

https://crates.io/crates/wit

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 14:21):

@Ralph

wasm-tools component wit wit/ -w -o out.wasm

view this post on Zulip Calvin Prewitt (Feb 08 2024 at 14:26):

@Paul Scheduikat

There's some rough edges importing implementations right now. I know @Peter Huene and @Daniel Macovei are working through the issues.

view this post on Zulip Ralph (Feb 08 2024 at 14:27):

no, this is great! I had missed that you could create an interface-only module

view this post on Zulip Ralph (Feb 08 2024 at 14:27):

That's wonderful!!!!

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 14:29):

@Calvin Prewitt The interface types are probably already contained in the binary is there a way to convert them to wit?

view this post on Zulip Ralph (Feb 08 2024 at 14:29):

I do know you can extract them easily enough, but it's .wit-to-module that I'd missed.....

view this post on Zulip Ralph (Feb 08 2024 at 14:32):

wasm-tools component wit HelloWorld.wasm
package root:root;

world root {
}

view this post on Zulip Ralph (Feb 08 2024 at 14:33):

but THAT has an implementation.....

view this post on Zulip Lann Martin (Feb 08 2024 at 14:35):

@Paul Scheduikat I believe this prints a component's interface as wit: wasm-tools component wit component.wasm

view this post on Zulip Calvin Prewitt (Feb 08 2024 at 14:36):

@Paul Scheduikat The tooling will get more integrated and easier for workflows. But wasm-tools component wit filepath/filename.wasm will print to to stdoout.

view this post on Zulip Calvin Prewitt (Feb 08 2024 at 14:37):

I just ran it on wasi:http encoded as binary and outputted to .wit text file. Works as expected.

view this post on Zulip Calvin Prewitt (Feb 08 2024 at 14:40):

Doc comments will survive the roundtrip to binary and back to .wit text file. But you will end up with a single .wit file and lose the original directory structure of separate .wit files.

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 14:40):

Only for interface components for implementations it doesn't:

$ wasm-tools component wit adder.wasm
package root:component;

world root {
  export example:calculate/add;
}

It is contained in the binary tho:

$ wasm-tools component wit adder.wasm -t
(component
  (type (;0;)
    (component
      (type (;0;)
        (component
          (type (;0;)
            (instance
              (type (;0;) (func (param "a" u32) (param "b" u32) (result u32)))
              (export (;0;) "add" (func (type 0)))
            )
          )
          (export (;0;) "example:calculate/add" (instance (type 0)))
        )
      )
      (export (;0;) "root:component/root" (component (type 0)))
    )
  )
  (export (;1;) "root" (type 0))
  (@producers
    (processed-by "wit-component" "0.20.1")
  )
)

view this post on Zulip Ralph (Feb 08 2024 at 14:41):

AH, yes, is there an issue for that?

view this post on Zulip Daniel Macovei (Feb 08 2024 at 14:44):

Sorry in your example what does your original component's wit look like?

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 14:46):

@Daniel Macovei

package example:calculate;

interface add {
    add: func(a: u32, b: u32) -> u32;
}

world adder {
    export add;
}

view this post on Zulip Daniel Macovei (Feb 08 2024 at 14:51):

It's not that it "only works for interface components". It's that the binary you printed has a different wit than the wit you used to create the component.

view this post on Zulip Ralph (Feb 08 2024 at 14:52):

Paul, are you testing out the component book example?

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 14:52):

Yes rust component example

view this post on Zulip Daniel Macovei (Feb 08 2024 at 14:52):

Sorry I know that's confusing. But if you look at the wat you shared, that is a component that exports a component type that is described by the wit you shared, which is a different component than the one described by the wit

view this post on Zulip Lann Martin (Feb 08 2024 at 14:56):

The tool could recurse into the nested interfaces. You wouldn't always want to do that unconditionally but it is functionality that could be added.

view this post on Zulip Daniel Macovei (Feb 08 2024 at 14:58):

Are you using cargo-component? It seems like your wat was handwritten

view this post on Zulip Ralph (Feb 08 2024 at 14:58):

Lann Martin said:

The tool could recurse into the nested interfaces. You wouldn't always want to do that unconditionally but it is functionality that could be added.

it would be nice to be able to pass a --tree option to recurse, though in many cases those interfaces are not exposed to the outside caller.

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 14:58):

@Daniel Macovei
No cargo component.

view this post on Zulip Ralph (Feb 08 2024 at 14:59):

Daniel Macovei said:

Are you using cargo-component? It seems like your wat was handwritten

he's doing the book's wat example: https://component-model.bytecodealliance.org/tutorial.html

view this post on Zulip Ralph (Feb 08 2024 at 14:59):

Correct, @Paul Scheduikat ?

view this post on Zulip Daniel Macovei (Feb 08 2024 at 14:59):

Was surprised there aren't any wasi interfaces in your output

view this post on Zulip Ralph (Feb 08 2024 at 15:00):

Basically, Paul is user testing the tutorial for us :-) we need to take notes and file prs and issues :-)

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:01):

@Daniel Macovei You only get those if your code is directly runnable.

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:04):

@Ralph The tutorial works, there is just just no example of composing multiple self written packages together.

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:04):

^^

view this post on Zulip Ralph (Feb 08 2024 at 15:04):

right, this is interesting!

view this post on Zulip Ralph (Feb 08 2024 at 15:04):

great!

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:06):

It's a little tricky because composition is the intersection of using several tools together here. You're asking about warg, cargo-component, wasm-tools component wit, and then there are also a variety of ways to perform the composition.

view this post on Zulip Ralph (Feb 08 2024 at 15:06):

I'll see about doing that later today and submit a pr to the doc repo

view this post on Zulip Ralph (Feb 08 2024 at 15:07):

@Daniel Macovei is right, but that shouldn't bother us too much at this point methinhks. once language tooling kicks in, that will all change anyway....

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:07):

Yes this use case definitely is something we care about making more intuitive, it's just going to take some time I think. Folks are aware it's not intuitive

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:09):

But specifically I think one of the take aways is that while wasm-tools could support recursion when it prints wit, probably a common point of confusion is that the top level wit for a binary generated by cargo-component exports the wit that you use to create the binary, meaning it is not going to have the same wit you used generate the component

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:09):

One needs to understand that to start playing around with composition

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:10):

And that probably isn't clear from the component book

view this post on Zulip Ralph (Feb 08 2024 at 15:12):

squillace@idiopath:~/work/hello-wasi-http/target/wasm32-wasi/debug$ wasm-tools component wit hello_wasi_http.wasm
package root:component;

world root {
import wasi:io/error@0.2.0;
import wasi:io/streams@0.2.0;
import wasi:cli/stdout@0.2.0;
import wasi:cli/stderr@0.2.0;
import wasi:cli/stdin@0.2.0;
import wasi:http/types@0.2.0;

export wasi:http/incoming-handler@0.2.0;
}

view this post on Zulip Ralph (Feb 08 2024 at 15:13):

if it were a composed component, you'd be right, only the top level things would be visible; I like the tree idea, though....

view this post on Zulip Ralph (Feb 08 2024 at 15:13):

that's Dan's rust wasi:http hello world

view this post on Zulip Ralph (Feb 08 2024 at 15:14):

lemme go find the wit:
image.png

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:16):

hmm if you have the code available it looks like that one might actually have several wit files

view this post on Zulip Ralph (Feb 08 2024 at 15:17):

Dan had this one running throughout the RC flow, so early on you had to have the wit to compile, and it hasn't been rearranged: https://github.com/sunfishcode/hello-wasi-http

Contribute to sunfishcode/hello-wasi-http development by creating an account on GitHub.

view this post on Zulip Ralph (Feb 08 2024 at 15:18):

which was why I was asking whether we even could now build only wit files!

view this post on Zulip Ralph (Feb 08 2024 at 15:18):

I was "used to" having to have the physical .wit to build

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:20):

@Daniel Macovei
I still don't really understand how I would do this. The following is loosely contrived from the tutorial, but simplified and I now want to split it up in two packages.

package docs:add;

interface add {
    add: func(a: u32, b: u32) -> u32;
}

world adder {
    export add;
}

The adder world is then built to a component with cargo component.

This results in the adder.wasm file implementing the adder world.

I then create a new project (component) with cargo component new and only copy over the adder.wasm file.

package docs:calculator;

world calculator {
    import docs:add:add;
}

What are the steps to implement the calculator world?

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:22):

So that's the part where you write rust code in your cargo component project

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:24):

The rust code isn't included in the docs, but you would import the bindings in your lib.rs and reference the interfaces you're implementing from bindings. This is the part where the cargo-component docs are your friend https://github.com/bytecodealliance/cargo-component?tab=readme-ov-file#getting-started

A Cargo subcommand for creating WebAssembly components based on the component model proposal. - GitHub - bytecodealliance/cargo-component: A Cargo subcommand for creating WebAssembly components bas...

view this post on Zulip Lann Martin (Feb 08 2024 at 15:24):

As a meta-point, this thread might get more relevant eyeballs in #wit-bindgen

view this post on Zulip Ralph (Feb 08 2024 at 15:24):

agreed; can you move it? I'm not a zulip master, I confess.

view this post on Zulip Ralph (Feb 08 2024 at 15:25):

#bleg

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:25):

@Daniel Macovei But I need the type information for docs:add:add;to be able to use it in rust. I now only have adder.wasm which although it contains the type information isn't easily usable.

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:26):

cargo-component leverages wit-bindgen which creates the rust types you need from your wit and exposes them in the bindings module it generates for you

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:27):

in the link i shared there's this snippet

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:27):

I do get that I can copy over the wit interface declaration from the other project, but as I have learned now I should be able to import it with just the adder.wasm file.

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:27):

mod bindings;

use bindings::Guest;

struct Component;

impl Guest for Component {
    /// Say hello!
    fn hello_world() -> String {
        "Hello, World!".to_string()
    }
}

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:31):

@Daniel Macovei I do understand how to implement components in rust.

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:31):

Sorry it's a little confusing what you're trying to import where. A little later on I can try and push an example that recreates the example in the book. Without having looked too closely at it though you need to create two different cargo-component projects, one for the calculator, and one for adder, and then compose the output binaries

view this post on Zulip Daniel Macovei (Feb 08 2024 at 15:31):

Paul Scheduikat said:

Daniel Macovei I do understand how to implement components in rust.

Sure thing, sorry for mistaking that as the point of confusion

view this post on Zulip Ralph (Feb 08 2024 at 15:32):

I think the point is the book doesn't discuss the "composing" or "binding together" of the components, am I correct, Paul?

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:36):

I know how I would compose these two packages together.

If a wit file is intended to have multiple implementations (like the WASI specs) then you would publish it as a standalone binary-encoded wit package. If you just want a representation of a particular component's imports/exports then you don't need a separate package; components are self-describing in that way.

I don't understand how I would go about it when downloading just a implementation wasm component from a registry, without a separate wit package defining the interfaces.

view this post on Zulip Ralph (Feb 08 2024 at 15:37):

EXACTLY

view this post on Zulip Ralph (Feb 08 2024 at 15:38):

up to now, I've been copying around the wit files for implementations; the question is how to reference those types by pointing at an interface-only component module.

view this post on Zulip Ralph (Feb 08 2024 at 15:39):

Let's ignore the registry part for a moment, because that is just a location. First we should be able to replicate this locally and document that in the book.

view this post on Zulip Paul Scheduikat (Feb 08 2024 at 15:40):

Agree registry is irrelevant here.


Last updated: Dec 23 2024 at 13:07 UTC