Joel Dice (Aug 16 2023 at 22:06): Joel Dice (Aug 16 2023 at 22:06):As mentioned in the component model meeting earlier today, @Kate G and @Ivan Towlson have been working on documenting Component Model tooling for various languages, and they'd like to have something useful ready for the Componentize the World event co-located with WasmCon at the beginning of September. One thing that would make their job easier -- as well as help make that event go smoothly -- is if we can agree on a set of versions of tools to recommend that are known to be compatible with each other.
Specifically, I'm thinking of wasmtime, wasm-tools, wit-bindgen, cargo-component, jco, and componentize-py. The component binary format has been stable for a while now, so that should be easy, but wasi-cli and the WIT syntax have seen a lot of changes lately, so it would be great to get all those tools aligned.
I recall that @Guy Bedford had been keeping track of this kind of thing on GitHub, but I can't seem to find it now, and I don't know how up-to-date it is. Guy, would you mind reminding me where it is?
Guy Bedford (Aug 16 2023 at 22:08):Thanks for picking this up. I was maintaining https://github.com/bytecodealliance/wasm-tools/wiki/Component-Model-Tooling-Compatibility but haven't done so for the past couple of releases, especially since we haven't had particularly clear release cadences recently.
Guy Bedford (Aug 16 2023 at 22:08):the last couple of releases should really be added though
Guy Bedford (Aug 16 2023 at 22:08):at least to get this next one before the conf would be great
Joel Dice (Aug 16 2023 at 22:12):@Pat Hickey Ideally we'd start with a known good version of the Wasmtime WASI Preview 2 implementation and have everything else standardize on that. Do we think we're close to having something like that, or alternatively, could we call Wasmtime 11's implementation "good enough" and just document known issues?
Pat Hickey (Aug 16 2023 at 22:17):I don't think the wasmtime 11 wasi is a great implementation, it didnt have the streams/pollable rewrite
Pat Hickey (Aug 16 2023 at 22:18):that landed in the release branch for 12, which will go out in 4 more days, but that implementation also has some problems
Pat Hickey (Aug 16 2023 at 22:18):i expect we'll need to tell folks to use whatever is in main
Joel Dice (Aug 16 2023 at 22:18):I see that cargo-component is using a pretty bleeding-edge version of the Preview 1 adapter (using a Wasmtime tag from a few days ago), so we probably don't want to go backwards from there.
Pat Hickey (Aug 16 2023 at 22:18):right, thats the other wrinkle
Pat Hickey (Aug 16 2023 at 22:19):we need to all be on the same versions of the wits
Pat Hickey (Aug 16 2023 at 22:19):https://github.com/bytecodealliance/wasmtime/pull/6846 is a breaking change that will hopefully land today, getting rid of the stream error dummy type
Pat Hickey (Aug 16 2023 at 22:20):and the fixes to the backpressure / flushing situation (which is why wasmtime-wasi 12 will be fairly hosed) are still not in. i would be pretty bummed if we didnt finish that in time for componentize the world, i hope it will be in next week
Joel Dice (Aug 16 2023 at 22:20):I just don't want to get to September 8th and have all the tools using slightly different, incompatible versions (plus already-obsolete docs), so I'm trying to figure if we can pick a commit and rally around it.
Pat Hickey (Aug 16 2023 at 22:20):i guess we will have the wasmtime 13 release branch cut before componentize the world, maybe thats the best target.
Pat Hickey (Aug 16 2023 at 22:20):that will cut on the 4th.
Pat Hickey (Aug 16 2023 at 22:21):if we need more lead time, then i guess lets pick a date and try to hit it.
Pat Hickey (Aug 16 2023 at 22:21):im not thrilled that its late august and im still fixing streams, i started on these in june
Joel Dice (Aug 16 2023 at 22:21):yeah, turns out it was a tough problem
Pat Hickey (Aug 16 2023 at 22:23):lets say this: ill do everything i can to have backpressure done and landed by friday, aug 25th.
Pat Hickey (Aug 16 2023 at 22:24):we're also (thanks to eduardo!) close to landing a significant update to the http impl, syncing it up with the other wits
Pat Hickey (Aug 16 2023 at 22:24):trevor is planning to hammer on that and get it fully integrated with streams and pollable asap
Pat Hickey (Aug 16 2023 at 22:25):btw our conclusion there was that you and eduardo converged on writing nearly the precise same code: differences were just streams and pollables, and some minor style nits. that seems like a good sign that the spec has an "obvious" implementation...
Joel Dice (Aug 16 2023 at 22:27):August 25th sounds great. @Peter Huene @Guy Bedford @Alex Crichton any anyone else working on the tools I listed above: can we agree to pick a Wasmtime commit sometime after August 25th (or whenever Pat feels it's ready) and align all those tools?
Guy Bedford (Aug 16 2023 at 22:28):that would be great, having a good week to get all the upgrade paths in will help
Pat Hickey (Aug 16 2023 at 22:29):the other big q is whether we can land timmys resources bindgen by then?
Pat Hickey (Aug 16 2023 at 22:29):we wont be switched to resources in wasi, but theyll at least be available in new wits?
Guy Bedford (Aug 16 2023 at 22:29):as long as they're not in wasi yet, as I definitely won't have time to update wasi virt to resources
Guy Bedford (Aug 16 2023 at 22:30):but JS resources implementation should just be ready
Pat Hickey (Aug 16 2023 at 22:30):yeah we definitely wont have them in wasi by then
Joel Dice (Aug 16 2023 at 22:30):I've been assuming resources won't be ready (or at least fully baked), and I'm not sure I'll get to adding them to componentize-py by WasmCon, but we could create a little demo that works for some combination of tools, perhaps.
Guy Bedford (Aug 16 2023 at 22:31):I'm shooting to at least demonstrate them in jco and componentize-js by then, but will see how it goes
Pat Hickey (Aug 16 2023 at 22:31):it may be best to wait to introduce them to new people until they are fully baked
Peter Huene (Aug 16 2023 at 22:31):i don't anticipate any significant changes to cargo-component until the end of august; can sync on the WASI adapters to use
Pat Hickey (Aug 16 2023 at 22:32):but idk i have sorta just been leaving the resources roll out to yall, because ive been mired in other stuff
Peter Huene (Aug 16 2023 at 22:32):as timmy's example shows, we should at a minimum have an example working with cargo-component
Joel Dice (Aug 16 2023 at 22:33):I'd say we call resources "nice to have" for WasmCon, but not _require_ them to make stuff work.
Peter Huene (Aug 16 2023 at 22:33):the registry subgroup is also hoping to have a "for the event" registry set up for people to push packages to
Peter Huene (Aug 16 2023 at 22:33):which it'd be nice to prepopulate with corresponding WASI packages
Pat Hickey (Aug 16 2023 at 22:35):i guess that means we gotta publish the wasi snapshot as of the 25th as a prerelease to that registry
Pat Hickey (Aug 16 2023 at 22:35):so that stuff can depend on it
Peter Huene (Aug 16 2023 at 22:35):yeah, i can help out with that
Peter Huene (Aug 16 2023 at 22:36):the registry will be writable only for the weekend probably, with the intention of it going read-only after the event and then decomissioned in the future
Peter Huene (Aug 16 2023 at 22:36):(writable by anyone; writable by us pre-event)
Pat Hickey (Aug 16 2023 at 22:42):cool
Mariano Guerra (Aug 17 2023 at 11:58):wouldn't it be useful for the event but also moving forward to have a "caniuse" table for components?
pinging projects weekly or similar should be enough to get started.
I can help set it up :)
Joel Dice (Aug 17 2023 at 15:31):As a reminder, we discussed monthly tagging using a major.minor.patch-rc-YYYY-MM-DD.N format in a recent WASI meeting. I'm not sure if that's been happening, but if not perhaps the snapshot on the 25th(ish) would be a good time to start. @Mariano Guerra 's "caniuse" table could label features based on that tag.
Joel Dice (Aug 17 2023 at 15:35):Also, we could offer more user-friendly messages if the versions were in the WIT files (and thus the preview1->preview2 adapters) themselves (e.g. "This version of Wasmtime supports wasi-cli@0.2.0-rc-2023-08-25.0 but the component you're trying to run targets wasi-cli@0.2.0-rc-2023-09-25.0 -- please consider using a newer version of Wasmtime").
Pat Hickey (Aug 17 2023 at 16:18):it hasnt been happening because (at least, in my view) things havent been stable enough to warrant it. the big round of syncing wasmtime to standards repos was with a goal of stabilizing things
Pat Hickey (Aug 17 2023 at 16:18):but yes, i would like to use the version tag on the 25thish.
Pat Hickey (Aug 17 2023 at 16:19):one wrinkle i found: the wit publish tool expects the version to be present in wit.toml and not in the package/use declarations themselves, but if someone copies the files out of the repo theyll expect the opposite
Pat Hickey (Aug 17 2023 at 16:20):but thats based on me just messing around trying things and maybe @Peter Huene has some better solution
Peter Huene (Aug 17 2023 at 16:27):i can certainly change it so that wit requires the package directive to have a version on it to publish instead and we remove the version field in wit.toml; i'm holding out hope that we can kill the package directive as i don't like having versions in "source code", but until then we can improve the wit tool DX
Peter Huene (Aug 17 2023 at 16:28):same goes for specifying versions on foreign interface references in worlds, they don't belong in source; they belong externally specified from the tool that resolved the versions from a registry
Pat Hickey (Aug 17 2023 at 16:45):yeah i agree with you that keeping versions in a single spot rather than scattered all over the source code is a better dx.
Kate Goldenring (Aug 17 2023 at 21:04):Thanks for starting this thread @Joel Dice and everyone for putting together this great tooling! For context on docs for Componentize the World, I've drafted up documentation that takes a simple example world shows how to implement a component of that world using the Rust (cargo component), JS (jco), and Python (componentize py) toolchains. It also has a host application that can run all of them, regardless of the toolchain that created the component, so the tooling as is is already a powerful example. The work to align all the tools on WASI versioning and WIT will probably prevent some issues folks may run into if they go beyond the document and start playing with more complex worlds.
Alex Crichton (Aug 18 2023 at 14:38):can we agree to pick a Wasmtime commit sometime after August 25th (or whenever Pat feels it's ready) and align all those tools?
I personally liked the idea of the Wasmtime 13 branch point being used for this, was that happening too late in the game though? We could perhaps update the release process to manually perform the Wasmtime 13 branch on Aug 25 instead of Sep 5 if necessary, but having a concrete branch (and eventual release artifact) to point to I think would be quite useful as opposed to a commit. (e.g. I'd prefer to avoid recommending rev = '...' in Cargo.toml
Joel Dice (Aug 18 2023 at 14:57):Yeah, I guess we could treat whatever we have on the 25th as a "release candidate", but then point everything to the Wasmtime 13 branch on the 5th. I just want to make sure everyone has time to test and tweak before WasmCon, but if all we have left to do on the 5th is update some Cargo.toml files, that should be fine.
Alex Crichton (Aug 18 2023 at 15:01):I'm also ok yeah with having a loose understanding of "no more major changes after the 25th"
Alex Crichton (Aug 18 2023 at 15:02):and the 25th-to-the-5th is "let's get everything sync'd" e.g. publish the wasm-tools crates and integrate them into Wasmtime to ensure everything's at the same version
Pat Hickey (Aug 18 2023 at 16:46):yes, we can at least declare no more wit changes landing between the 25th and 5th
Christof Petig (Aug 20 2023 at 10:14):Guy Bedford said:
I'm shooting to at least demonstrate them in jco and componentize-js by then, but will see how it goes
I found that host support for resources in jco already works nicely - guest needs your ongoing work in componentize-js.
See https://github.com/cpetig/resource-demo/ for a matrix (a three guest languages, two host environments working demo)
Disclaimer: I still need to improve my C++ code generation to include the fixes I made by hand in this repo.
Guy Bedford (Aug 21 2023 at 16:34):@Christof Petig nice to see it's working for you, yes got it shipped out last week for jco, guest side may take a little longer but will see
Pat Hickey (Aug 24 2023 at 06:41):the streams work will not be ready by the 25th.
Pat Hickey (Aug 24 2023 at 06:42):hopefully next week. plan to do demos without it.
Pat Hickey (Aug 24 2023 at 06:44):the short story on why is: porting sockets to the new scheme (which landed since we started working on it) will be some work, and also we have to mess with poll oneoff again because now we could be polling on both the ready-for-writing future and the flush-completed future from the same output-stream.
Joel Dice (Aug 24 2023 at 13:37):Thanks for the update, @Pat Hickey, and thanks for all the work you've been putting into this.
I think we should still aim to align the tools with whatever we have on September 5th, even if streams aren't ready by then. The main point is that all the tools produce and consume compatible files.
Alex Crichton (Aug 24 2023 at 22:32):As a reminder the 25th is tomorrow, so a heads up to try to hold off on breaking changes after tomorrow and/or get them in tomorrow. The next week or so should then focus on getting everything up-to-date with each other and such.
Pat Hickey (Aug 25 2023 at 03:40):the streams fixes wont be in tomorrow. they might be possible to get in on monday?
Pat Hickey (Aug 25 2023 at 03:41):my concern right now is that if we freeze before we get them in, they wont ship as part of wasmtime 13, and will need to wait until 14. (unless we change the cut and release dates for 13)
Pat Hickey (Aug 25 2023 at 19:01):is it going to ruin anyone's week if streams land on monday?
Joel Dice (Aug 25 2023 at 19:08):Works for me.
Alex Crichton (Aug 25 2023 at 19:10):The "final" wasm-tools publish just went out so I'm going to try to consider that frozen until after wasmcon except for bugfixes
Pat Hickey (Aug 28 2023 at 22:41):alright folks, trevor and i have both been working on the streams revisions all day and theyre not ready yet
Pat Hickey (Aug 28 2023 at 22:42):i am going to keep plugging away at these for as long as it takes to get them done. its up to yall whether we land them as soon as theyre ready, or whether we extend the wasmtime 13 cut date to after wasmcon and land them then.
Pat Hickey (Aug 28 2023 at 22:44):so please lmk here what your preferences are. @Joel Dice @Bailey Hayes @Alex Crichton @Guy Bedford @Kate Goldenring @Peter Huene
Peter Huene (Aug 28 2023 at 22:45):i have no preference and it's fairly easy to update cargo-component any at point to updated adapters
Joel Dice (Aug 28 2023 at 22:48):Pretty much all I need for WasmCon regarding wasi-cli I/O is for stdio to work at least as well as it currently does in main, so if there's danger of regressions there I'd prefer to wait until after. Otherwise, either is fine with me.
Pat Hickey (Aug 28 2023 at 22:50):stdout and stderr in main run the risk of being swallowed if the table is dropped before the AsyncWriteStream worker gets a chance to run. my patch will definitely fix that problem, but it will do so at the cost of a breaking wit change for streams.wit
Guy Bedford (Aug 28 2023 at 22:52):no major preference from me, although I would prefer to not be fiddling with demos last minute, so will likely lock down at the current state myself, only upgrading if it's easy to do / I have the time.
Bailey Hayes (Aug 28 2023 at 23:14):One of my demos is locked in now. I was considering creating another demo that's a what's next and would have to run off of branches anyways to highlight resources.
Since this is a breaking change to streams.wit my preference is that we do not change any of the in place processes. That means missing the wasmtime 13 cut date and giving streams time to salient in dev. That will also give us time to align upstream WebAssembly WASI repos with an updated streams.wit
Pat Hickey (Aug 28 2023 at 23:17):the time between tagging the release branch and making the release is to give time for cranelift & related fuzzers to find a bug. there is no testing of anything wasi related that goes on during that time
Pat Hickey (Aug 28 2023 at 23:18):wasmtime 12 streams are fairly broken for users, and i dont want to make them broken until wasmtime 14
Alex Crichton (Aug 28 2023 at 23:19):How about we get it ready to land a make a decision at that time? That'll give us a more concrete idea in terms of the time frame
Alex Crichton (Aug 28 2023 at 23:20):I'm worried that things feel broken now with output being swallowed by default almost, so id err on the side of landing personally
Alex Crichton (Aug 28 2023 at 23:20):But that's assuming that updating most demos is a matter of downloading a new adapter and not much else
Bailey Hayes (Aug 28 2023 at 23:22):I imagine most demos are locked in either way which I guess is a pro for taking a risk now to unbreak
Joel Dice (Aug 28 2023 at 23:22):I think the main thing is what Guy said: we need to have enough time to align all the tools and test that they work together. If we can have it ready to land in the next couple of days, great. Otherwise, we need to pick a revision prior to landing the stream changes and align on that.
Joel Dice (Aug 28 2023 at 23:30):I'm less concerned about whatever demos people are doing for WasmCon presentations -- presenters can hand-select whatever combination of tools they like (and probably already have). I'm thinking of the Componentize the World event, where ideally we can give people a list of tools that are known to work together, without leaving them to figure it out themselves.
Alex Crichton (Aug 28 2023 at 23:38):as of right now wasm-tools/wit-bindgen I think are considered frozen til wasmcon, at their latest publications on crates.io
Alex Crichton (Aug 28 2023 at 23:38):for wasmtime I think we should nominally consider main as-is, the upcoming release-13.0.0 branch, as what we should use
Alex Crichton (Aug 28 2023 at 23:38):so no more breaking changes at all on main except possibly streams
Alex Crichton (Aug 28 2023 at 23:39):I suppose I should say I'm less worried about an exact revision everyone's sharing as opposed to making sure no more breaking changes happen until release-13.0.0 is branched
Alex Crichton (Aug 28 2023 at 23:39):e.g. if we fix a bug I think it'd be good to have that picked up
Alex Crichton (Aug 30 2023 at 16:57):To copy over and write down what we discussed today:
So everyone should be able to continue as-is with current tooling and versions to ensure everything all works in concert
Peter Huene (Aug 30 2023 at 17:38):What version of adapters should we settle on, then?
Alex Crichton (Aug 30 2023 at 17:41):The dev tag on wasmtime right now should work, and the WITs they use should be frozen until after wasmcon
Alex Crichton (Aug 30 2023 at 17:42):this makes me thing we should have a prerelease tag created as part of the wasmtime release process for prerelease artifacts, but perhaps another time!
Last updated: Nov 22 2024 at 16:03 UTC