wasmtime / issue #4308 Support records, variants, enums, ... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / issue #4308 Support records, variants, enums, ...

Wasmtime GitHub notifications bot (Jun 23 2022 at 21:36):

alexcrichton opened issue #4308:

I'm splitting this issue out of https://github.com/bytecodealliance/wasmtime/issues/4185 to write up some thoughts on how this can be done. Specifically today the current Wasmtime support for the component model has mappings for many component model types to Rust native types but not all of them. For example integers, strings, lists, tuples, etc, are all mapped directly to Rust types. Basically if the component model types equivalent in Rust is in the Rust standard library that's already implemented. What that leaves to implement, however, is Rust-defined mappings for component model types that are "structural" like records.

This issue is intended to document the current thinking of how we're going to expose this. The general idea is that we'll create a proc-macro crate, probably named something like wasmtime-component-macro, which is an internal dependency of the wasmtime crate. The various macros would then get reexported at the wasmtime::component::* namespace.

Currently the bindings for host types are navigated through three traits: ComponentValue, Lift, and Lower. We'll want a custom derive for all three of these traits. Deriving Lift and Lower require a ComponentValue derive as well, but users should be able to pick one of Lift and Lower without the other one.

record

Records in the component model correspond to structs in Rust. The rough shape of this will be:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(record)]
struct Foo {
    #[component(name = "foo-bar-baz")]
    a: i32,
    b: u32,
}
To typecheck correctly the record type must list fields in the same order as the fields listed in the Rust code for now. Field reordering may be implemented at a later date but for now we'll do strict matching. Fields must have both matching names and matching types.

The #[component(record)] here may seem redundant but it's somewhat required below for variants/enums.

The #[component(name = "...")] is intended to rename the field from the component model's perspective. The type-checking will test against the name specified.

Using this derive on a tuple or empty struct will result in a compile-time error.

variant

Variants roughly correspond to Rust enums:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(variant)]
enum Foo {
    #[component(name = "foo-bar-baz")]
    A(u32),
    B,
}
Typechecking, like records, will check cases in-order and all cases must match in both name and payload. A missing payload in Rust is automatically interpreted as the unit payload in the component model.

Variants with named fields (B { bar: u32 }) will be disallowed. Variants with multiple payloads (B(u32, u32)) will also be disallowed.

Note that #[component(variant)] here distinguishes it from...

enum
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(enum)]
enum Foo {
    #[component(name = "foo-bar-baz")]
    A,
    B,
}
Typechecking is similar to variants where the number/names of cases must all match.

Variants with any payload are disallowed in this derive mode.

union

This will, perhaps surprisingly, still map to an enum in Rust since this is still a tagged union, not a literal C union:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(union)]
enum Foo {
    A(u32),
    B(f32),
}
The number of cases and the types of each case must match a union definition to correctly typecheck. Union cases don't have names so renaming here isn't needed.

A payload on each enum case in Rust is required, and like with variant it's required to be a tuple-variant with only one element. All other forms of payloads are disallowed. Note that the names in Rust are just informative in Rust, it doesn't affect the ABI or type-checking

flags

These will be a bit "funkier" than the above since there's not something obvious to attach a #[derive] to:
wasmtime::component::flags! {
    #[derive(Lift, Lower)]
    flags Foo {
        #[component(name = "...")]
        const A;
        const B;
        const C;
    }
}
The general idea here is to roughly take inspiration from the bitflags crate in terms of what the generated code does. Ideally this should have a convenient Debug implementation along with various constants to OR-together and such in Rust. The exact syntax here is up for debate, this is just a strawman.

Implementation Details

One caveat is that the ComponentValue/Lift/Lower traits mention internal types in the wasmtime crate which aren't intended to be part of the public API. To solve this the macro will reference items in a path such as:
wasmtime::component::__internal::the_name
The __internal module will be #[doc(hidden)] and will only exist to reexport dependencies needed by the proc-macro. This crate may end up having a bland pub use wasmtime_environ or individual items, whatever works best.

The actual generated trait impls will probably look very similar to the implementations that exist for tuples, and Result<T, E> already present in typed.rs

Alternatives

One alternative to the above is to have #[derive(ComponentRecord)] instead of #[derive(ComponentValue)] #[component(record)] or something like that. While historically some discussions have leaned in this direction with the introduction of Lift and Lower traits I personally feel that the balance is now slightly in the other direction where it would be nice if we can keep derive targeted at the specific traits and then configuration for the derive happens afterwards.

Wasmtime GitHub notifications bot (Jun 23 2022 at 21:36):

alexcrichton labeled issue #4308:

I'm splitting this issue out of https://github.com/bytecodealliance/wasmtime/issues/4185 to write up some thoughts on how this can be done. Specifically today the current Wasmtime support for the component model has mappings for many component model types to Rust native types but not all of them. For example integers, strings, lists, tuples, etc, are all mapped directly to Rust types. Basically if the component model types equivalent in Rust is in the Rust standard library that's already implemented. What that leaves to implement, however, is Rust-defined mappings for component model types that are "structural" like records.

This issue is intended to document the current thinking of how we're going to expose this. The general idea is that we'll create a proc-macro crate, probably named something like wasmtime-component-macro, which is an internal dependency of the wasmtime crate. The various macros would then get reexported at the wasmtime::component::* namespace.

Currently the bindings for host types are navigated through three traits: ComponentValue, Lift, and Lower. We'll want a custom derive for all three of these traits. Deriving Lift and Lower require a ComponentValue derive as well, but users should be able to pick one of Lift and Lower without the other one.

record

Records in the component model correspond to structs in Rust. The rough shape of this will be:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(record)]
struct Foo {
    #[component(name = "foo-bar-baz")]
    a: i32,
    b: u32,
}
To typecheck correctly the record type must list fields in the same order as the fields listed in the Rust code for now. Field reordering may be implemented at a later date but for now we'll do strict matching. Fields must have both matching names and matching types.

The #[component(record)] here may seem redundant but it's somewhat required below for variants/enums.

The #[component(name = "...")] is intended to rename the field from the component model's perspective. The type-checking will test against the name specified.

Using this derive on a tuple or empty struct will result in a compile-time error.

variant

Variants roughly correspond to Rust enums:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(variant)]
enum Foo {
    #[component(name = "foo-bar-baz")]
    A(u32),
    B,
}
Typechecking, like records, will check cases in-order and all cases must match in both name and payload. A missing payload in Rust is automatically interpreted as the unit payload in the component model.

Variants with named fields (B { bar: u32 }) will be disallowed. Variants with multiple payloads (B(u32, u32)) will also be disallowed.

Note that #[component(variant)] here distinguishes it from...

enum
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(enum)]
enum Foo {
    #[component(name = "foo-bar-baz")]
    A,
    B,
}
Typechecking is similar to variants where the number/names of cases must all match.

Variants with any payload are disallowed in this derive mode.

union

This will, perhaps surprisingly, still map to an enum in Rust since this is still a tagged union, not a literal C union:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(union)]
enum Foo {
    A(u32),
    B(f32),
}
The number of cases and the types of each case must match a union definition to correctly typecheck. Union cases don't have names so renaming here isn't needed.

A payload on each enum case in Rust is required, and like with variant it's required to be a tuple-variant with only one element. All other forms of payloads are disallowed. Note that the names in Rust are just informative in Rust, it doesn't affect the ABI or type-checking

flags

These will be a bit "funkier" than the above since there's not something obvious to attach a #[derive] to:
wasmtime::component::flags! {
    #[derive(Lift, Lower)]
    flags Foo {
        #[component(name = "...")]
        const A;
        const B;
        const C;
    }
}
The general idea here is to roughly take inspiration from the bitflags crate in terms of what the generated code does. Ideally this should have a convenient Debug implementation along with various constants to OR-together and such in Rust. The exact syntax here is up for debate, this is just a strawman.

Implementation Details

One caveat is that the ComponentValue/Lift/Lower traits mention internal types in the wasmtime crate which aren't intended to be part of the public API. To solve this the macro will reference items in a path such as:
wasmtime::component::__internal::the_name
The __internal module will be #[doc(hidden)] and will only exist to reexport dependencies needed by the proc-macro. This crate may end up having a bland pub use wasmtime_environ or individual items, whatever works best.

The actual generated trait impls will probably look very similar to the implementations that exist for tuples, and Result<T, E> already present in typed.rs

Alternatives

One alternative to the above is to have #[derive(ComponentRecord)] instead of #[derive(ComponentValue)] #[component(record)] or something like that. While historically some discussions have leaned in this direction with the introduction of Lift and Lower traits I personally feel that the balance is now slightly in the other direction where it would be nice if we can keep derive targeted at the specific traits and then configuration for the derive happens afterwards.

Wasmtime GitHub notifications bot (Jun 23 2022 at 21:37):

alexcrichton commented on issue #4308:

cc @dicej as I believe you were possibly interested in taking a stab at this

Wasmtime GitHub notifications bot (Jun 23 2022 at 22:01):

dicej commented on issue #4308:

Thanks, @alexcrichton -- this is extremely helpful.

BTW, you refer to ComponentValue here and in some of the comments in typed.rs, but the trait is actually named ComponentType, correct?

Anyway, yes, I'll take a stab at this and post questions here as they arise.

Wasmtime GitHub notifications bot (Jun 23 2022 at 22:02):

alexcrichton commented on issue #4308:

Whoops sorry yes it's ComponentType (can't keep track of my own historical list of changes)

Wasmtime GitHub notifications bot (Jun 23 2022 at 22:29):

jameysharp commented on issue #4308:

I had started work on this, but I fully support somebody else taking this on as I've had to switch gears to something else.

My first contribution here was in PR #4217; there's a little rationale discussion there. And my very work-in-progress branch might help in getting started: https://github.com/jameysharp/wasmtime/tree/component-derive

I'm happy to do code review or answer questions regarding this feature. I'm going to defer to Alex in case of any disagreements but I think I have a pretty good sense of where he wants this to go.

Wasmtime GitHub notifications bot (Jun 29 2022 at 14:42):

dicej commented on issue #4308:

Support for record types has been merged. I'm working on variants now and will move on to the others after that.

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:15):

dicej commented on issue #4308:

@alexcrichton How would you suggest I handle cases like these?
#[derive(ComponentValue, Lift, Lower)]
#[component(union)]
enum Foo {
    A(u32),
    B(f32),
    C(f32),
}

#[derive(ComponentValue, Lift, Lower)]
#[component(union)]
enum Bar<A, B, C> {
    A(A),
    B(B),
    C(C),
}

type Baz = Bar<u32, f32, f32>
Should we throw an error in the first case or silently de-duplicate the types? And should we simply disallow #[component(union)] on generic enums?

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:16):

dicej edited a comment on issue #4308:

@alexcrichton How would you suggest I handle cases like these?
#[derive(ComponentValue, Lift, Lower)]
#[component(union)]
enum Foo {
    A(u32),
    B(f32),
    C(f32),
}

#[derive(ComponentValue, Lift, Lower)]
#[component(union)]
enum Bar<A, B, C> {
    A(A),
    B(B),
    C(C),
}

type Baz = Bar<u32, f32, f32>;
Should we throw an error in the first case or silently de-duplicate the types? And should we simply disallow #[component(union)] on generic enums?

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:18):

alexcrichton commented on issue #4308:

Hm I'm not sure I understand, those both look pretty reasonable to me and like they should work. Could you clarify which part you're thinking probably needs an error?

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:27):

dicej commented on issue #4308:

I guess I assumed duplicate types (e.g. two f32s) in a union type with no way to distinguish between them would be a problem. If not, then I guess (union float32 float32) is equivalent to (union float32) and both are equally valid?

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:29):

alexcrichton commented on issue #4308:

Ah ok, my impression is that (union float32 float32) is valid in the component model and because it's a tagged discriminant you can distinguish between the two cases, albeit it does seem a bit silly.

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:32):

dicej commented on issue #4308:

Okay -- I guess I got thrown off by the fact that the cases aren't named. Good point about them having separate discriminants, i.e. you can distinguish them based on the order in which they appear.

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:38):

dicej commented on issue #4308:

Would it make sense to have a custom syntax for unions, like you suggested for flags? With flags, only the names need to be specified. With unions, only the types need to be specified. Seems kind of awkward to make the programmer choose names that will be ignored anyway.

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:43):

alexcrichton commented on issue #4308:

In general I'm happy to leave it to your discretion. My assumption was that a union in the component model is represented in Rust with an enum one way or another so I figured "may as well let the input have the names" but I don't really mind one way or another.

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:43):

dicej commented on issue #4308:

Arguing with myself: the programmer will still use those names in Rust code, so they're not useless. Nevermind my suggestion above.

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:44):

jameysharp commented on issue #4308:

I found it helpful to think of union cases as still being "named". There's a bit of text in the Type Definitions section of the component model explainer that describes union and enum as syntactic sugar for variants:
                (enum <name>+) ↦ (variant (case <name> unit)+)
            (union <valtype>+) ↦ (variant (case "𝒊" <valtype>)+) for 𝒊=0,1,...
So the case names for unions are "0", "1", ...

I've been tempted to suggest that we should expect a union type for any Rust enum where the cases are named _0, _1, ... But that might be too much magic.

Wasmtime GitHub notifications bot (Jun 30 2022 at 15:50):

dicej commented on issue #4308:

Thanks for the feedback, @alexcrichton and @jameysharp . Let's stick with the original plan; being able to specify meaningful names can be nice for use in Rust code, even if they don't get carried over into the component interface.

Wasmtime GitHub notifications bot (Jun 30 2022 at 19:53):

dicej commented on issue #4308:

https://github.com/bytecodealliance/wasmtime/pull/4359 adds support for variant, enum, and union types. I'll follow up with a separate PR for flags, since that will be a different kind of macro.

Wasmtime GitHub notifications bot (Jun 30 2022 at 22:43):

jameysharp commented on issue #4308:

This is excellent work, @dicej! :+1:

Wasmtime GitHub notifications bot (Jul 02 2022 at 23:21):

dicej commented on issue #4308:

How should we handle enums with explicit discriminants? For example:
#[derive(ComponentValue, Lift, Lower)]
#[component(enum)]
enum Foo {
    #[component(name = "foo-bar-baz")]
    A = 42,
    B,
}
Should we ignore them? Throw an error?

Similarly, should we pay any attention to e.g. #[repr(u16)] annotations on the target enum?

Wasmtime GitHub notifications bot (Jul 03 2022 at 14:43):

bjorn3 commented on issue #4308:

I think #[repr(u16)] can be ignored. As for discriminants I think those can be ignored as well as only the name matters for the wasm component model. They might be used for interacting with native code through a C abi for example.

Wasmtime GitHub notifications bot (Jul 03 2022 at 14:50):

dicej commented on issue #4308:

I think #[repr(u16)] can be ignored. As for discriminants I think those can be ignored as well as only the name matters for the wasm component model. They might be used for interacting with native code through a C abi for example.

Agreed. I think the guiding principle here is: just because a type is annotated with #[derive(ComponentValue)] doesn't mean that the _only_ purpose of the type is for interop with Wasm components. It could be used for C interop, among other things.

Wasmtime GitHub notifications bot (Jul 05 2022 at 05:22):

jameysharp commented on issue #4308:

My assumption had been that setting an explicit discriminant would be an error. But, I guess we already keep the Canonical ABI used on the wasm side entirely independent of whatever ABI happens to be used on the Rust side. So yes, you've convinced me that neither explicit discriminants nor #[repr] matter for this purpose.

Wasmtime GitHub notifications bot (Jul 07 2022 at 09:58):

tsoutsman commented on issue #4308:

How does wit-bindgen fit into this? I'm currently trying to create a wasmtime::component::Component from a wasm component with the following export interface file:
record pci-device {
    vendor-id: u16,
    device-id: u16,
}

init: func(dev: pci-device)
but have ran into this unimplemented!() statement.

Is the idea that I would have a corresponding struct definition in my Rust i.e.:
#[derive(ComponentValue, Lift, Lower)]
#[component(record)]
struct PciDevice {
    #[component(name = "vendor-id")]
    vendor_id: u16,
    #[component(name = "device-id")]
    device_id: u16,
}
That I would use when interacting with the module (e.g. TypedFunc type parameters)?

Also, as a side note, if someone could give me some pointers on implementing "component type export" to remove the unimplemented!() statement, I'd be happy to give it a shot.

Wasmtime GitHub notifications bot (Jul 07 2022 at 10:00):

tsoutsman edited a comment on issue #4308:

How does wit-bindgen fit into this? I'm currently trying to create a wasmtime::component::Component from a wasm component with the following export interface file:
record pci-device {
    vendor-id: u16,
    device-id: u16,
}

init: func(dev: pci-device)
but have ran into this unimplemented!() statement.

The wasm component is a separate Rust crate that uses wit-bindgen and wit-component to implement the init function.

Is the idea that I would have a corresponding struct definition in my Rust i.e.:
#[derive(ComponentValue, Lift, Lower)]
#[component(record)]
struct PciDevice {
    #[component(name = "vendor-id")]
    vendor_id: u16,
    #[component(name = "device-id")]
    device_id: u16,
}
That I would use when interacting with the component (e.g. in TypedFunc type parameters)?

Also, as a side note, if someone could give me some pointers on implementing "component type export" to remove the unimplemented!() statement, I'd be happy to give it a shot.

Wasmtime GitHub notifications bot (Jul 07 2022 at 23:04):

dicej commented on issue #4308:

#4414 should be the last PR for this issue :crossed_fingers:

Wasmtime GitHub notifications bot (Jul 13 2022 at 16:18):

dicej commented on issue #4308:

I believe this can be closed now that #4414 has been merged.

Wasmtime GitHub notifications bot (Jul 13 2022 at 16:27):

alexcrichton commented on issue #4308:

Indeed, thanks so much @dicej!

Wasmtime GitHub notifications bot (Jul 13 2022 at 16:27):

alexcrichton closed issue #4308:

I'm splitting this issue out of https://github.com/bytecodealliance/wasmtime/issues/4185 to write up some thoughts on how this can be done. Specifically today the current Wasmtime support for the component model has mappings for many component model types to Rust native types but not all of them. For example integers, strings, lists, tuples, etc, are all mapped directly to Rust types. Basically if the component model types equivalent in Rust is in the Rust standard library that's already implemented. What that leaves to implement, however, is Rust-defined mappings for component model types that are "structural" like records.

This issue is intended to document the current thinking of how we're going to expose this. The general idea is that we'll create a proc-macro crate, probably named something like wasmtime-component-macro, which is an internal dependency of the wasmtime crate. The various macros would then get reexported at the wasmtime::component::* namespace.

Currently the bindings for host types are navigated through three traits: ComponentValue, Lift, and Lower. We'll want a custom derive for all three of these traits. Deriving Lift and Lower require a ComponentValue derive as well, but users should be able to pick one of Lift and Lower without the other one.

record

Records in the component model correspond to structs in Rust. The rough shape of this will be:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(record)]
struct Foo {
    #[component(name = "foo-bar-baz")]
    a: i32,
    b: u32,
}
To typecheck correctly the record type must list fields in the same order as the fields listed in the Rust code for now. Field reordering may be implemented at a later date but for now we'll do strict matching. Fields must have both matching names and matching types.

The #[component(record)] here may seem redundant but it's somewhat required below for variants/enums.

The #[component(name = "...")] is intended to rename the field from the component model's perspective. The type-checking will test against the name specified.

Using this derive on a tuple or empty struct will result in a compile-time error.

variant

Variants roughly correspond to Rust enums:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(variant)]
enum Foo {
    #[component(name = "foo-bar-baz")]
    A(u32),
    B,
}
Typechecking, like records, will check cases in-order and all cases must match in both name and payload. A missing payload in Rust is automatically interpreted as the unit payload in the component model.

Variants with named fields (B { bar: u32 }) will be disallowed. Variants with multiple payloads (B(u32, u32)) will also be disallowed.

Note that #[component(variant)] here distinguishes it from...

enum
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(enum)]
enum Foo {
    #[component(name = "foo-bar-baz")]
    A,
    B,
}
Typechecking is similar to variants where the number/names of cases must all match.

Variants with any payload are disallowed in this derive mode.

union

This will, perhaps surprisingly, still map to an enum in Rust since this is still a tagged union, not a literal C union:
use wasmtime::component::{ComponentValue, Lift, Lower};

#[derive(ComponentValue, Lift, Lower)]
#[component(union)]
enum Foo {
    A(u32),
    B(f32),
}
The number of cases and the types of each case must match a union definition to correctly typecheck. Union cases don't have names so renaming here isn't needed.

A payload on each enum case in Rust is required, and like with variant it's required to be a tuple-variant with only one element. All other forms of payloads are disallowed. Note that the names in Rust are just informative in Rust, it doesn't affect the ABI or type-checking

flags

These will be a bit "funkier" than the above since there's not something obvious to attach a #[derive] to:
wasmtime::component::flags! {
    #[derive(Lift, Lower)]
    flags Foo {
        #[component(name = "...")]
        const A;
        const B;
        const C;
    }
}
The general idea here is to roughly take inspiration from the bitflags crate in terms of what the generated code does. Ideally this should have a convenient Debug implementation along with various constants to OR-together and such in Rust. The exact syntax here is up for debate, this is just a strawman.

Implementation Details

One caveat is that the ComponentValue/Lift/Lower traits mention internal types in the wasmtime crate which aren't intended to be part of the public API. To solve this the macro will reference items in a path such as:
wasmtime::component::__internal::the_name
The __internal module will be #[doc(hidden)] and will only exist to reexport dependencies needed by the proc-macro. This crate may end up having a bland pub use wasmtime_environ or individual items, whatever works best.

The actual generated trait impls will probably look very similar to the implementations that exist for tuples, and Result<T, E> already present in typed.rs

Alternatives

One alternative to the above is to have #[derive(ComponentRecord)] instead of #[derive(ComponentValue)] #[component(record)] or something like that. While historically some discussions have leaned in this direction with the introduction of Lift and Lower traits I personally feel that the balance is now slightly in the other direction where it would be nice if we can keep derive targeted at the specific traits and then configuration for the derive happens afterwards.

Last updated: Nov 22 2024 at 17:03 UTC