As mentioned in the component model meeting earlier today, @Kate G and @Ivan Towlson have been working on documenting Component Model tooling for various languages, and they'd like to have something useful ready for the Componentize the World event co-located with WasmCon at the beginning of September. One thing that would make their job easier -- as well as help make that event go smoothly -- is if we can agree on a set of versions of tools to recommend that are known to be compatible with each other.
Specifically, I'm thinking of wasmtime
, wasm-tools
, wit-bindgen
, cargo-component
, jco
, and componentize-py
. The component binary format has been stable for a while now, so that should be easy, but wasi-cli
and the WIT syntax have seen a lot of changes lately, so it would be great to get all those tools aligned.
I recall that @Guy Bedford had been keeping track of this kind of thing on GitHub, but I can't seem to find it now, and I don't know how up-to-date it is. Guy, would you mind reminding me where it is?
Thanks for picking this up. I was maintaining https://github.com/bytecodealliance/wasm-tools/wiki/Component-Model-Tooling-Compatibility but haven't done so for the past couple of releases, especially since we haven't had particularly clear release cadences recently.
the last couple of releases should really be added though
at least to get this next one before the conf would be great
@Pat Hickey Ideally we'd start with a known good version of the Wasmtime WASI Preview 2 implementation and have everything else standardize on that. Do we think we're close to having something like that, or alternatively, could we call Wasmtime 11's implementation "good enough" and just document known issues?
I don't think the wasmtime 11 wasi is a great implementation, it didnt have the streams/pollable rewrite
that landed in the release branch for 12, which will go out in 4 more days, but that implementation also has some problems
i expect we'll need to tell folks to use whatever is in main
I see that cargo-component
is using a pretty bleeding-edge version of the Preview 1 adapter (using a Wasmtime tag from a few days ago), so we probably don't want to go backwards from there.
right, thats the other wrinkle
we need to all be on the same versions of the wits
https://github.com/bytecodealliance/wasmtime/pull/6846 is a breaking change that will hopefully land today, getting rid of the stream error dummy type
and the fixes to the backpressure / flushing situation (which is why wasmtime-wasi 12 will be fairly hosed) are still not in. i would be pretty bummed if we didnt finish that in time for componentize the world, i hope it will be in next week
I just don't want to get to September 8th and have all the tools using slightly different, incompatible versions (plus already-obsolete docs), so I'm trying to figure if we can pick a commit and rally around it.
i guess we will have the wasmtime 13 release branch cut before componentize the world, maybe thats the best target.
that will cut on the 4th.
if we need more lead time, then i guess lets pick a date and try to hit it.
im not thrilled that its late august and im still fixing streams, i started on these in june
yeah, turns out it was a tough problem
lets say this: ill do everything i can to have backpressure done and landed by friday, aug 25th.
we're also (thanks to eduardo!) close to landing a significant update to the http impl, syncing it up with the other wits
trevor is planning to hammer on that and get it fully integrated with streams and pollable asap
btw our conclusion there was that you and eduardo converged on writing nearly the precise same code: differences were just streams and pollables, and some minor style nits. that seems like a good sign that the spec has an "obvious" implementation...
August 25th sounds great. @Peter Huene @Guy Bedford @Alex Crichton any anyone else working on the tools I listed above: can we agree to pick a Wasmtime commit sometime after August 25th (or whenever Pat feels it's ready) and align all those tools?
that would be great, having a good week to get all the upgrade paths in will help
the other big q is whether we can land timmys resources bindgen by then?
we wont be switched to resources in wasi, but theyll at least be available in new wits?
as long as they're not in wasi yet, as I definitely won't have time to update wasi virt to resources
but JS resources implementation should just be ready
yeah we definitely wont have them in wasi by then
I've been assuming resources won't be ready (or at least fully baked), and I'm not sure I'll get to adding them to componentize-py by WasmCon, but we could create a little demo that works for some combination of tools, perhaps.
I'm shooting to at least demonstrate them in jco and componentize-js by then, but will see how it goes
it may be best to wait to introduce them to new people until they are fully baked
i don't anticipate any significant changes to cargo-component
until the end of august; can sync on the WASI adapters to use
but idk i have sorta just been leaving the resources roll out to yall, because ive been mired in other stuff
as timmy's example shows, we should at a minimum have an example working with cargo-component
I'd say we call resources "nice to have" for WasmCon, but not _require_ them to make stuff work.
the registry subgroup is also hoping to have a "for the event" registry set up for people to push packages to
which it'd be nice to prepopulate with corresponding WASI packages
i guess that means we gotta publish the wasi snapshot as of the 25th as a prerelease to that registry
so that stuff can depend on it
yeah, i can help out with that
the registry will be writable only for the weekend probably, with the intention of it going read-only after the event and then decomissioned in the future
(writable by anyone; writable by us pre-event)
cool
wouldn't it be useful for the event but also moving forward to have a "caniuse" table for components?
pinging projects weekly or similar should be enough to get started.
I can help set it up :)
As a reminder, we discussed monthly tagging using a major.minor.patch-rc-YYYY-MM-DD.N
format in a recent WASI meeting. I'm not sure if that's been happening, but if not perhaps the snapshot on the 25th(ish) would be a good time to start. @Mariano Guerra 's "caniuse" table could label features based on that tag.
Also, we could offer more user-friendly messages if the versions were in the WIT files (and thus the preview1->preview2 adapters) themselves (e.g. "This version of Wasmtime supports wasi-cli@0.2.0-rc-2023-08-25.0
but the component you're trying to run targets wasi-cli@0.2.0-rc-2023-09-25.0
-- please consider using a newer version of Wasmtime").
it hasnt been happening because (at least, in my view) things havent been stable enough to warrant it. the big round of syncing wasmtime to standards repos was with a goal of stabilizing things
but yes, i would like to use the version tag on the 25thish.
one wrinkle i found: the wit publish
tool expects the version to be present in wit.toml and not in the package/use declarations themselves, but if someone copies the files out of the repo theyll expect the opposite
but thats based on me just messing around trying things and maybe @Peter Huene has some better solution
i can certainly change it so that wit
requires the package
directive to have a version on it to publish instead and we remove the version
field in wit.toml
; i'm holding out hope that we can kill the package
directive as i don't like having versions in "source code", but until then we can improve the wit
tool DX
same goes for specifying versions on foreign interface references in worlds, they don't belong in source; they belong externally specified from the tool that resolved the versions from a registry
yeah i agree with you that keeping versions in a single spot rather than scattered all over the source code is a better dx.
Thanks for starting this thread @Joel Dice and everyone for putting together this great tooling! For context on docs for Componentize the World, I've drafted up documentation that takes a simple example world shows how to implement a component of that world using the Rust (cargo component
), JS (jco
), and Python (componentize py
) toolchains. It also has a host application that can run all of them, regardless of the toolchain that created the component, so the tooling as is is already a powerful example. The work to align all the tools on WASI versioning and WIT will probably prevent some issues folks may run into if they go beyond the document and start playing with more complex worlds.
can we agree to pick a Wasmtime commit sometime after August 25th (or whenever Pat feels it's ready) and align all those tools?
I personally liked the idea of the Wasmtime 13 branch point being used for this, was that happening too late in the game though? We could perhaps update the release process to manually perform the Wasmtime 13 branch on Aug 25 instead of Sep 5 if necessary, but having a concrete branch (and eventual release artifact) to point to I think would be quite useful as opposed to a commit. (e.g. I'd prefer to avoid recommending rev = '...'
in Cargo.toml
Yeah, I guess we could treat whatever we have on the 25th as a "release candidate", but then point everything to the Wasmtime 13 branch on the 5th. I just want to make sure everyone has time to test and tweak before WasmCon, but if all we have left to do on the 5th is update some Cargo.toml files, that should be fine.
I'm also ok yeah with having a loose understanding of "no more major changes after the 25th"
and the 25th-to-the-5th is "let's get everything sync'd" e.g. publish the wasm-tools
crates and integrate them into Wasmtime to ensure everything's at the same version
yes, we can at least declare no more wit changes landing between the 25th and 5th
Guy Bedford said:
I'm shooting to at least demonstrate them in jco and componentize-js by then, but will see how it goes
I found that host support for resources in jco already works nicely - guest needs your ongoing work in componentize-js.
See https://github.com/cpetig/resource-demo/ for a matrix (a three guest languages, two host environments working demo)
Disclaimer: I still need to improve my C++ code generation to include the fixes I made by hand in this repo.
@Christof Petig nice to see it's working for you, yes got it shipped out last week for jco, guest side may take a little longer but will see
the streams work will not be ready by the 25th.
hopefully next week. plan to do demos without it.
the short story on why is: porting sockets to the new scheme (which landed since we started working on it) will be some work, and also we have to mess with poll oneoff again because now we could be polling on both the ready-for-writing future and the flush-completed future from the same output-stream.
Thanks for the update, @Pat Hickey, and thanks for all the work you've been putting into this.
I think we should still aim to align the tools with whatever we have on September 5th, even if streams aren't ready by then. The main point is that all the tools produce and consume compatible files.
As a reminder the 25th is tomorrow, so a heads up to try to hold off on breaking changes after tomorrow and/or get them in tomorrow. The next week or so should then focus on getting everything up-to-date with each other and such.
the streams fixes wont be in tomorrow. they might be possible to get in on monday?
my concern right now is that if we freeze before we get them in, they wont ship as part of wasmtime 13, and will need to wait until 14. (unless we change the cut and release dates for 13)
is it going to ruin anyone's week if streams land on monday?
Works for me.
The "final" wasm-tools publish just went out so I'm going to try to consider that frozen until after wasmcon except for bugfixes
alright folks, trevor and i have both been working on the streams revisions all day and theyre not ready yet
i am going to keep plugging away at these for as long as it takes to get them done. its up to yall whether we land them as soon as theyre ready, or whether we extend the wasmtime 13 cut date to after wasmcon and land them then.
so please lmk here what your preferences are. @Joel Dice @Bailey Hayes @Alex Crichton @Guy Bedford @Kate Goldenring @Peter Huene
i have no preference and it's fairly easy to update cargo-component
any at point to updated adapters
Pretty much all I need for WasmCon regarding wasi-cli I/O is for stdio to work at least as well as it currently does in main
, so if there's danger of regressions there I'd prefer to wait until after. Otherwise, either is fine with me.
stdout and stderr in main
run the risk of being swallowed if the table is dropped before the AsyncWriteStream worker gets a chance to run. my patch will definitely fix that problem, but it will do so at the cost of a breaking wit change for streams.wit
no major preference from me, although I would prefer to not be fiddling with demos last minute, so will likely lock down at the current state myself, only upgrading if it's easy to do / I have the time.
One of my demos is locked in now. I was considering creating another demo that's a what's next and would have to run off of branches anyways to highlight resources.
Since this is a breaking change to streams.wit
my preference is that we do not change any of the in place processes. That means missing the wasmtime 13 cut date and giving streams time to salient in dev. That will also give us time to align upstream WebAssembly WASI repos with an updated streams.wit
the time between tagging the release branch and making the release is to give time for cranelift & related fuzzers to find a bug. there is no testing of anything wasi related that goes on during that time
wasmtime 12 streams are fairly broken for users, and i dont want to make them broken until wasmtime 14
How about we get it ready to land a make a decision at that time? That'll give us a more concrete idea in terms of the time frame
I'm worried that things feel broken now with output being swallowed by default almost, so id err on the side of landing personally
But that's assuming that updating most demos is a matter of downloading a new adapter and not much else
I imagine most demos are locked in either way which I guess is a pro for taking a risk now to unbreak
I think the main thing is what Guy said: we need to have enough time to align all the tools and test that they work together. If we can have it ready to land in the next couple of days, great. Otherwise, we need to pick a revision prior to landing the stream changes and align on that.
I'm less concerned about whatever demos people are doing for WasmCon presentations -- presenters can hand-select whatever combination of tools they like (and probably already have). I'm thinking of the Componentize the World event, where ideally we can give people a list of tools that are known to work together, without leaving them to figure it out themselves.
as of right now wasm-tools/wit-bindgen I think are considered frozen til wasmcon, at their latest publications on crates.io
for wasmtime I think we should nominally consider main as-is, the upcoming release-13.0.0 branch, as what we should use
so no more breaking changes at all on main except possibly streams
I suppose I should say I'm less worried about an exact revision everyone's sharing as opposed to making sure no more breaking changes happen until release-13.0.0 is branched
e.g. if we fix a bug I think it'd be good to have that picked up
To copy over and write down what we discussed today:
So everyone should be able to continue as-is with current tooling and versions to ensure everything all works in concert
What version of adapters should we settle on, then?
The dev
tag on wasmtime right now should work, and the WITs they use should be frozen until after wasmcon
this makes me thing we should have a prerelease tag created as part of the wasmtime release process for prerelease artifacts, but perhaps another time!
Last updated: Dec 23 2024 at 12:05 UTC