Ryan Hunt (Nov 13 2020 at 22:19): Ryan Hunt (Nov 13 2020 at 22:19):Hi all,
The spidermonkey wasm team is currently trying to release cranelift to our nightly aarch64 user base. As part of this we're reviewing our stability/release policy for this code. I understand that there were some other discussions about a future release policy for wasmtime/cranelift, so I thought I should open up a thread.
We currently vendor specific commits of the crates we directly depend on. All the recursive dependencies of each crate are then vendored using normal semver and lockfile semantics.
We need a few things to manage releasing cranelift effectively.
This is complicated by the fact that the crates required for our compiler are spread across multiple repositories.
There's a balance here between a mozilla-specific process and one that benefits other community members. We need something like this in the near term, so if possible we would like to get 'something' working for this and potentially expand it in the future.
So in summary:
If there's a better approach here, suggestions are appreciated!
cc @Till Schneidereit @Chris Fallin @Julian Seward @Lars Hansen
Chris Fallin (Nov 13 2020 at 23:34):@Ryan Hunt thanks for writing up these thoughts! I'm very much in favor of a stable-branch policy/process of some sort -- I think it would be useful for a bunch of folks.
I wanted to weigh in on one point in particular: regalloc.rs vs wasmtime repositories. On balance I think I'd be receptive to this; while there is definitely an advantage to presenting regalloc.rs as an independent, reusable component (and I hope it remains so!), it seems that some of the reasoning that led to consolidation of wasmtime and Cranelift into one repo (while still releasing as separate crates) might apply again. As we developed various features that crossed the regalloc/CL boundary, we had to do a bunch of release/version-bumps of regalloc.rs just to pick up minor changes. (We're at v0.0.31 now and most of those 31 releases were necessitated by cross-cutting work.) If we choose to do that, I'm happy to help out with the git magic to actually do the merge with history.
I don't have as much of an opinion on wasmparser; it seems to be much more widely used outside of CL than regalloc is, and is not as closely coupled, so maybe the above reasoning doesn't apply as strongly, though I'm not sure.
Chris Fallin (Nov 16 2020 at 18:18):A followup thought here, after talking with @Benjamin Bouvier a bit about regalloc.rs: perhaps we should instead pull in regalloc (and wasmparser?) as git submodules in the wasmtime repo. This would let us version/branch things as we please, since the root project pins particular commits of its deps that we control, without relying on crates.io releases. The alternative if we don't Monorepo All The Things is that if a stable branch of a dependency (regalloc or wasmparser, e.g.) changes to incorporate a security fix, we'd need to release a crate version and refer to that in the stable branch of wasmtime; or else pin git commits in wasmtime's Cargo.toml, but that's most of the way to submodules and submodules let one hack locally without editing dep paths. Thoughts?
Alex Crichton (Nov 16 2020 at 18:36):I would personally prefer to avoid submodules if we can, exploring other avenues first. They're generally just a pain to deal with and having a forcing function for us to release crates on crates.io I think is also valuable (at least I like it for wat/wast/wasmparser). I would hope that a security fix for something like wasmparser wouldn't be too bad (and presumably other projects would also want to be able to pull in the fix so we'd want to publish anyway), but it sounds like we'd definitely need documentation of how to release the various components
Chris Fallin (Nov 16 2020 at 18:40):So one particular unanswered question, in my head at least, is how a crate-release-based dependency reference would work with a bugfix on a stable branch. Imagine we have regalloc.rs main at v0.0.40, but its stable corresponds to v0.0.39, and wasmtime stable points to v0.0.39. Say we need to backport a security fix to regalloc stable; do we release v0.0.41 (nonlinear, doesn't supersede v0.0.40) or do we pin the stable wasmtime to a git commit in regalloc.rs? In other words it seems to me that at least for stable branches, the nonlinearity forces us off of a pure crate-release-based process, but possibly I'm missing some workaround :-)
Alex Crichton (Nov 16 2020 at 18:48):I think the fix for that would be to switch regalloc to 0.x.0 releases
Alex Crichton (Nov 16 2020 at 18:48):so that way if we wanted to patch something it would be 0.x.1
Alex Crichton (Nov 16 2020 at 18:48):and 0.x.y is still "this is unstable" in semver-speak
Chris Fallin (Nov 16 2020 at 18:49):Ah, of course
Ryan Hunt (Nov 17 2020 at 23:28):It seems like the easiest immediate path forward is for us to manage our own fork(s) and branches. So I'm going to be looking into that now.
As a related but separate note, It seems like there is support for merging the regalloc and wasmtime repositories (while keeping them separate crates). @Chris Fallin would you be interested in helping out with this? Or giving me pointers on how to best accomplish this.
Chris Fallin (Nov 17 2020 at 23:31):@Ryan Hunt re: regalloc.rs, the latest thoughts are I think that we are leaning toward keeping it as a separate repository; or at least, that's what we were converging toward above (@Benjamin Bouvier, @Alex Crichton or others correct me if wrong!)
Ryan Hunt (Nov 18 2020 at 00:00):Ah apologies, I misread the previous comments then.
So I think I understand the concerns with using submodules, but what is the concern with merging regalloc.rs into the wasmtime repository, using path dependencies and still publishing regalloc as an independent crate?
Chris Fallin (Nov 18 2020 at 00:05):The main concerns raised by @Benjamin Bouvier when I spoke to him (please correct me if I misrepresent this!) were CI time (PRs to regalloc.rs shouldn't necessarily need to wait for wasmtime CI) and the way in which regalloc.rs is presented as, and maintains the ability to be, an independently-usable building block
Ryan Hunt (Nov 18 2020 at 00:17):Ah, yeah, running a full wasmtime test run for regalloc changes would be unfortunate. And if there's desire for it to be presented as an independent component then I'm fine with it remaining so. Wasn't aware of Benjamin's concerns, and don't want to shake things up unless it's worth it.
Benjamin Bouvier (Nov 18 2020 at 09:21):Hi! Thanks @Chris Fallin, this is a correct explanation of my concerns. Unrelated CI failures in addition to long CI runs have been cumbersome in the past, and since regalloc has a very low CI latency, as well as very focused code / tests / issues / PRs, it is quite nice that we can keep it separate from the rest.
Till Schneidereit (Nov 18 2020 at 11:24):Hey Ryan,
I think this is all sorted out for the near-term concerns at least, but I wanted to chime in and say that I agree the proposed course of action makes sense! The one thing perhaps being that it doesn't seem necessary to me to maintain the stable branches in a Mozilla-specific fork. Do you see complications with doing so in the main repository? I'd have a slight preference for that as it'd more clearly put us on a path towards shared stable branches.
As Alex says, it seems good to continue doing releases to crates.io, and use patch version bumps for changes to stable branches. To some degree that's more long- or at least medium-term though: ideally we'd have crates.io releases automated to a degree where it can just happen in automation. Plus, those only make sense for shared stable branches, not Mozilla-specific ones. For those, I guess you'll want to use git refspecs as the dependency? I.e., the Mozilla stable branch(es) of the wasmtime repo would refer to the associated Mozilla stable branch(es) of the regalloc.rs and wasm-tools repositories. That'd give you full control and flexibility in how to handle patches and what to pull into your monorepo
Last updated: Dec 23 2024 at 13:07 UTC