Daniel Macovei (Oct 20 2023 at 18:42): Daniel Macovei (Oct 20 2023 at 18:42):I have a use case where I could really use the ability to be able to extract wit from a wam binary as it's being uploaded to storage as a stream. While wasmparser seems to be pretty good at working on streams, from what I can tell it seems that decoding wit parses an uses wasmparser to parse the entire binary before exposing wit information. Might be missing something, but if not, is there openness to stream support, or would it require some heavy lifting that I'm not taking into account?
Joel Dice (Oct 20 2023 at 18:45):In my experience, the component imports are generally at the beginning of the file, and the exports are at the end, so I'm not sure how you would avoid looking at the entire binary.
Alex Crichton (Oct 20 2023 at 18:46):This is technically feasible to do but is not written as such today. For example the WIT decoding process only requires a wasmparser::types::Types information which can be built in a streaming fashion via wasmparser APIs. Currently however wit-component does not expose an API which takes this wasmparser::types::Types structure, so changes would be required to wit-component. That being said I'd prefer to ideally avoid adding such an API because it makes wasmparser a public dependency which is something I'd rather avoid due to the rapid version bumps that wasmparser requires.
I think adding something to wit-component though that looks like you incrementally feed in a binary and get out WIT would work well, but that'd be new support to add to wit-component
Alex Crichton (Oct 20 2023 at 18:47):oh and I'm assuming that you're basically asking if you can change this parameter to something like impl Read. If you're asking for something else then I'm answering the wrong question
Joel Dice (Oct 20 2023 at 18:50):Oh, sounds like you don't mind waiting until the whole binary has arrived -- you just want to avoid buffering the whole thing, which is reasonable.
Alex Crichton (Oct 20 2023 at 18:54):oh that was just my interpretation, which may also be wrong
Daniel Macovei (Oct 20 2023 at 18:57):yeah i don't mind waiting for the whole thing, just avoiding buffering all of it
Joel Dice (Oct 20 2023 at 19:02):wasmparser can already do incremental parsing, can't it? Could you use that to grab the parts you need, skip past the parts you don't, and then somehow construct an "abbreviated" version of the component to give to wit-component? I'm handwaving a lot, so maybe that's not feasible.
Alex Crichton (Oct 20 2023 at 19:10):no yeah that's what I meant by it's definitely feasible, it's just not exposed from wit-component
Alex Crichton (Oct 20 2023 at 19:10):so I think it'd be best to add a struct to wit-component where you feed it bytes and tell it when you're done and it pops out a Resolve/Package/etc
Daniel Macovei (Oct 20 2023 at 19:18):could probably just impl Read this struct? https://github.com/bytecodealliance/wasm-tools/blob/main/crates/wit-component/src/decoding.rs#L15
Alex Crichton (Oct 20 2023 at 19:19):something along those lines would work yeah
Last updated: Oct 23 2024 at 20:03 UTC