Stream: git-wasmtime

Topic: wasmtime / issue #6583 Implement resources for the compon...

Wasmtime GitHub notifications bot (Jun 14 2023 at 16:58):

alexcrichton opened issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [ ] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [ ] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [ ] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [ ] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [ ] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [ ] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jun 14 2023 at 16:58):

alexcrichton labeled issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [ ] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [ ] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [ ] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [ ] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [ ] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [ ] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jun 21 2023 at 15:13):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [ ] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [ ] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [ ] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [ ] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [ ] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jun 21 2023 at 15:13):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [ ] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [ ] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [ ] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [ ] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jun 21 2023 at 15:13):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [ ] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [ ] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [x] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [ ] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jun 21 2023 at 15:13):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [ ] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [x] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [x] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [ ] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jun 21 2023 at 15:13):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [x] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [x] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [x] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [ ] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jun 30 2023 at 18:42):

alexcrichton commented on issue #6583:

As a bit of an update on this the main bulk of the work for wasm-tools is posted at https://github.com/bytecodealliance/wasm-tools/pull/1084 and the main missing piece in wasm-tools is wasm-compose. I've since been working on an implementation in Wasmtime and continue to be hard at work on that. Progress is apace but currently no ETA. I've got some stuff working but still need to figure out things like destructors, borrow, fused adapters, and the specifics of the embedder API.

Wasmtime GitHub notifications bot (Jul 14 2023 at 17:03):

alexcrichton commented on issue #6583:

The initial wasmtime work should now be complete at https://github.com/bytecodealliance/wasmtime/pull/6691 (pending review of course). I plan on closing this issue when that's merged and deferring further work to the linked issues from that PR.

Wasmtime GitHub notifications bot (Jul 17 2023 at 14:29):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [x] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [x] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [x] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [x] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [ ] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jul 17 2023 at 14:29):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [x] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [x] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [x] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [ ] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [x] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [x] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jul 17 2023 at 14:29):

alexcrichton edited issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [x] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [x] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [x] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [x] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [x] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [x] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Wasmtime GitHub notifications bot (Jul 22 2023 at 00:06):

alexcrichton closed issue #6583:

This is intended to be a bit of a tracking issue and/or brain-dump about implementing resources in the component model. The work here spans a number of repositories and isn't exclusively limited to just the Wasmtime repository, but I'm choosing here as probably one of the higher-visibility locations to write all this down.

Settle on a spec

Currently as-is in the component-model repository two resource handles of own and borrow are specified but this may change with recent thinking. Settling on this will be necessary for at least completing this work, but isn't necessarily required before any other work goes forward. This will impact the runtime implementation in Wasmtime but likely won't radically change the overall shape, mostly details instead of what each piece is precisely doing.

Implement resources in wasm-tools

Resources have had their initial implementation landed in https://github.com/bytecodealliance/wasm-tools/pull/966 for a number of crates, but not all of them. This implementation will need to be updated depending on how the above spec discussion settles, and the remaining work here is:

• [x] Integrate resources into wit-parser. This involves parsing syntax, creating AST structures, tracking types-of-handles, etc.
• [x] Settle on the "ABI" pseudo-language used in wit-parser to use for resources. This is the abi module of wit-parser which is used by most WIT code generators, and notably the Instruction enum will need to grow variants of what to do for resources. This will be heavily influenced by the spec above.
• [x] Implement Wasm<->WIT translation for resources. This is part of the wit-component crate and dictates how WIT documents are encoded into WASM. This will require defining resource types appropriately and then referring to them via handles appropriately. This is, hopefully, not a large extension beyond what's already implemented.
• [x] Integrate resources into wit-smith to fuzz the previous implementation of Wasm<->WIT translation. This should help stress this piece further to ensure all the various corner cases are covered.
• [x] Integrate resources into wasm-tools component new. Like the previous point this lives in the wit-component crate and this is a bigger lift but is the location to define how core wasm actually interacts with resources. This will need to grow support for intrinsics such as new/drop for each resource type and hook those up to canon ... definitions which will be declared in the component. This is effectively defining the standard of "what actually does a guest generator do to integrate with resources".
• [ ] Integrate resources into wasm-tools compose. This is the wasm-compose crate and I left panicking stubs largely during the initial implementation. It's unclear how difficult this will be because this requires a form of subtype checking which is not trivial to do with resources. This probably needs to reuse more of the validation machinery in wasmparser but that's just a vague idea I have at this time.

Much of this work on wasm-tools is sequentialized and can't be parallelized all that well as each step likely relies on the prior.

Implement resources in Wasmtime

Naturally a big part of resources is the actual runtime impelmentation in Wasmtime! The precise steps involved here are more fuzzy to me than the ones above for wasm-tools, but the rough idea I have at this time is:

• [x] Update translation to record and process resources. Right now there are todo!() or unimplemented!() statements for when resources are encountered during parsing, and effectively those need to be filled out. This will require tracking resources per-component and generally figuring out things such as how many resource tables are required by a component. Precisely how this all works depends on the spec details above but also just figuring out how to track and implement this all in Wasmtime, which I'm less certain about.
• [x] Determine how to best implement the new resource-related canon functions. More-or-less this means expanding the CoreDef enum and propagating changes outwards from there. I'm thinking there'd be new variants like ResourceNew and such.
• [x] Determine how the embedding layer will represent and track resources. For example when a component imports a resource what's actually inserted into a Linker? Additionally sketch out the runtime representation of a resource so the embedder can create resources and pass them to a component. Additionally what it might look like to receive a resource defined in a component. For example the host must be able to create a resource and pass it to a component. Conversely for all resources the host must be able to destroy the resource and possibly run its destructor.

Unfortunately I feel like at this point I've sort of gone from "draw a circle" to "finish the owl" and I'm not precisely sure what other intermediate steps there are to implement resources in Wasmtime, despite there obviously being quite a few more steps. One thing I can say though is that the Wasmtime work can, in theory, progress in parallel to the remaining wasm-tools work. Towards the "end" when the bindgen! macro gets involved they'll need to sync back up but it should be possible to implement everything related to resources in Wasmtime independent of WIT and the integration there.

---

I plan on starting on the wasm-tools based work in the near future and moving outwards from there, but I wanted to be sure to write down my thoughts on all this in the meantime. I can also try to keep this updated over time with more thinking.

Last updated: Oct 23 2024 at 20:03 UTC