alexcrichton opened issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [ ] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [ ] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [ ] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [ ] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [ ] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [ ] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton labeled issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [ ] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [ ] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [ ] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [ ] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [ ] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [ ] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [ ] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [ ] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [ ] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [ ] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [ ] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [ ] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [ ] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [ ] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [ ] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [ ] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [ ] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [x] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [ ] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [ ] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [x] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [x] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [ ] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [x] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [x] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [x] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [ ] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton commented on issue #6583:
As a bit of an update on this the main bulk of the work for wasm-tools is posted at https://github.com/bytecodealliance/wasm-tools/pull/1084 and the main missing piece in wasm-tools is
wasm-compose
. I've since been working on an implementation in Wasmtime and continue to be hard at work on that. Progress is apace but currently no ETA. I've got some stuff working but still need to figure out things like destructors,borrow
, fused adapters, and the specifics of the embedder API.
alexcrichton commented on issue #6583:
The initial wasmtime work should now be complete at https://github.com/bytecodealliance/wasmtime/pull/6691 (pending review of course). I plan on closing this issue when that's merged and deferring further work to the linked issues from that PR.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [x] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [x] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [x] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [x] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [x] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [x] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [x] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [ ] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [x] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [x] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton edited issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [x] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [x] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [x] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [x] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [x] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [x] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
alexcrichton closed issue #6583:
This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.
Settle on a spec
Currently as-is in the component-model repository two resource handles of
own
andborrow
are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.Implement resources in
wasm-tools
Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:
- [x] Integrate resources into
wit-parser
. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.- [x] Settle on the "ABI" pseudo-language used in
wit-parser
to use for resources. This is theabi
module ofwit-parser
which is used by most WIT code generators, and notably theInstruction
enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.- [x] Implement Wasm<->WIT translation for resources. This is part of the
wit-component
crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.- [x] Integrate resources into
wit-smith
to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.- [x] Integrate resources into
wasm-tools component new
. Like the previous point this lives in thewit-component
crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up tocanon ...
definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".- [ ] Integrate resources into
wasm-tools compose
. This is thewasm-compose
crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery inwasmparser
but that's just a vague idea I have at this time.Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.
Implement resources in Wasmtime
Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for
wasm-tools
, but the rough idea I have at this time is:
- [x] Update translation to record and process resources. Right now there are
todo!()
orunimplemented!()
statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.- [x] Determine how to best implement the new resource-related
canon
functions. More-or-less this means expanding theCoreDef
enum and propagating changes outwards from there. I'm thinking there'd be new variants likeResourceNew
and such.- [x] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a
Linker
? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the
bindgen!
macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.
I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.
Last updated: Dec 23 2024 at 12:05 UTC