I have a use case where I could really use the ability to be able to extract wit from a wam binary as it's being uploaded to storage as a stream. While wasmparser seems to be pretty good at working on streams, from what I can tell it seems that decoding wit parses an uses wasmparser to parse the entire binary before exposing wit information. Might be missing something, but if not, is there openness to stream support, or would it require some heavy lifting that I'm not taking into account?
In my experience, the component imports are generally at the beginning of the file, and the exports are at the end, so I'm not sure how you would avoid looking at the entire binary.
This is technically feasible to do but is not written as such today. For example the WIT decoding process only requires a wasmparser::types::Types
information which can be built in a streaming fashion via wasmparser
APIs. Currently however wit-component
does not expose an API which takes this wasmparser::types::Types
structure, so changes would be required to wit-component
. That being said I'd prefer to ideally avoid adding such an API because it makes wasmparser
a public dependency which is something I'd rather avoid due to the rapid version bumps that wasmparser
requires.
I think adding something to wit-component
though that looks like you incrementally feed in a binary and get out WIT would work well, but that'd be new support to add to wit-component
oh and I'm assuming that you're basically asking if you can change this parameter to something like impl Read
. If you're asking for something else then I'm answering the wrong question
Oh, sounds like you don't mind waiting until the whole binary has arrived -- you just want to avoid buffering the whole thing, which is reasonable.
oh that was just my interpretation, which may also be wrong
yeah i don't mind waiting for the whole thing, just avoiding buffering all of it
wasmparser
can already do incremental parsing, can't it? Could you use that to grab the parts you need, skip past the parts you don't, and then somehow construct an "abbreviated" version of the component to give to wit-component
? I'm handwaving a lot, so maybe that's not feasible.
no yeah that's what I meant by it's definitely feasible, it's just not exposed from wit-component
so I think it'd be best to add a struct to wit-component where you feed it bytes and tell it when you're done and it pops out a Resolve
/Package
/etc
could probably just impl Read
this struct? https://github.com/bytecodealliance/wasm-tools/blob/main/crates/wit-component/src/decoding.rs#L15
something along those lines would work yeah
Last updated: Dec 23 2024 at 13:07 UTC