Stream: git-wasmtime

Topic: wasmtime / issue #6900 Update Rust in CI to 1.72.0

Wasmtime GitHub notifications bot (Aug 24 2023 at 15:50):

cfallin commented on issue #6900:

I'll offer a point in favor of option 2, with uncertain weight but maybe nonzero: not everyone uses rustup and can easily update their Rust compiler right away. Probably most folks do, and it's certainly the recommended way to use Rust, but the language is also packaged in distros, installed system-wide on shared machines, etc. Also platforms that are Rust tier 3 (hence have no rustup binaries) but have OS-packaged rustc and happen to work (at one point this included various BSDs, especially non-x86). Finally, some projects (e.g. Firefox back when it had an option to use Cranelift) pin on a certain Rust version, and so we're excluding ourselves from use in large software systems that may have more involved Rust upgrade procedures. (Any large company with a monorepo and carefully-packaged toolchains, for example.)

Many of those probably carry lower weight than e.g. using a sorely-needed feature in a new release; but if the habit is instead to update whenever we have time, with no other particular benefits, maybe we could consider the above?

Wasmtime GitHub notifications bot (Aug 24 2023 at 15:59):

cfallin commented on issue #6900:

For a little more data: the 2021 Rust survey report shows 7585 of 10214 responses use latest stable or nightly Rust in CI (74.2%); 8888 of 10868 responses use Rustup (81.8%). So a majority, but I'm definitely not comfortable excluding folks who don't fit into that "happy path", if we can help it, especially if the cases that don't are some of the more interesting ones (big software systems, etc).

Wasmtime GitHub notifications bot (Aug 24 2023 at 17:01):

alexcrichton commented on issue #6900:

I think personally I just want something clear and easy to follow. I've spent way too much time in my Rust career watching debates about the Rust version either end in "ok you can only use a version of Rust that's 4 years old" to the benefit of almost no one or a not-well-defined policy which ends up reigniting discussion every time it comes up.

I changed CI to pin Rust versions awhile back to avoid breakage from Rust updates (e.g. new warnings or such), and my hope is that future updates would not be blocked on "settle a MSRV policy". While developing such a policy is probably inevitable and useful, the lack of community consensus on what to do here makes figuring this out quite slog in my opinion. If you'd like I'm happy to back out the change to rust-version in the manifest and only update CI.

Wasmtime GitHub notifications bot (Aug 24 2023 at 17:42):

elliottt commented on issue #6900:

I'm generally in favor of staying up-to-date. What about adopting policy 1, and re-examining it if we get a report that it's breaking someone else's workflow?

Wasmtime GitHub notifications bot (Aug 24 2023 at 17:44):

cfallin commented on issue #6900:

Ah, sorry, I didn't mean to reignite any old discussions or scars. AFAIK this is the first time I've been involved in a "MSRV discussion"; hopefully I don't have as bumpy a path with them!

I agree it'd be great to set down an explicit policy for Wasmtime, to avoid any ambiguity. Maybe we should discuss and decide on something in the next Wasmtime meeting? I'll put it on the agenda.

Wasmtime GitHub notifications bot (Aug 24 2023 at 17:47):

cfallin commented on issue #6900:

(My comment raced with @elliottt's; I'm fine with seeing this go through as-is, then we can discuss more at the meeting...)

Wasmtime GitHub notifications bot (Aug 24 2023 at 20:05):

alexcrichton commented on issue #6900:

Ok sounds good! I'm happy to sit on this until the next meeting. If someone wants to use something from 1.72 I'd be in favor of merging and enabling that, but in the meantime I think it's ok to update locally and otherwise keep CI at 1.71

Wasmtime GitHub notifications bot (Aug 31 2023 at 17:12):

cfallin commented on issue #6900:

We talked about this in the Wasmtime biweekly just now, and reached a good amount of consensus; to summarize: we agreed that it makes sense to support a sliding window of latest stable Rust versions (stable down to stable - N; N is a parameter we still need to choose). This allows flexibility to embedders of Wasmtime by not forcing an upgrade to latest stable immediately after release -- others can design release processes that work within the N-version window. It also gives some flexibility for embedders if they ned to hold a version upgrade temporarily for whatever reason (bug or miscompile elsewhere, etc).

The main points to be decided are:

• What is N? (Some common choices seem to be 1, 2, 4; ideally not too much more than that?)
• How do we test this in CI?

On the latter point, discussion revolved around (at least) two main constraints: CI resources are limited, so that we can't simply duplicate the test runs for all Rust versions in the window; and we also would prefer "CI determinism", i.e., a PR that previously passed CI should not stop passing if a new Rust version happens to be released between runs. (This is important for the contributor experience but also for e.g. ability to respond quickly without roadblocks to emergency patches and that sort of thing.)

Given the determinism constraint, we'd likely define the window by pinning two versions of rust -- "newest supported" and "oldest supported" -- defining our window manually and bumping it manually. We should do these bumps regularly when new releases of Rust occur, and not only when we actually need a feature: otherwise, compatibility pressure builds up, dependencies acrete, and we have more problems when we do bump.

Given the CI resources constraint, there seem to be two main options: run all tests at newest supported and add one additional job on Linux/x86-64 that runs at oldest supported, or vice-versa. The former catches platform-specific issues due to new Rust features sooner, but does not cover the "tail" of the sliding window, so N-1/N-2 Rust could break in platform-specific ways. The latter ensures the tail of the window stays working everywhere, but may catch platform-specific issues later.

Does this capture everything @alexcrichton, @fitzgen, @tschneidereit, others?

---

My own thoughts:

• I think N = 2 strikes a reasonable balance; it gives a 12-week window for upgrades if someone is tracking latest Wasmtime.
• It seems to me that emprically, it may be better to pin most platform CI jobs to newest supported Rust, and cover the tail of the window with one Linux/x86-64 job: we've had more issues from new Rust versions (miscompiles mainly) than issues from changes to Wasmtime interacting poorly with older Rust. (Or said another way, the Rust compiler makes large changes more frequently than Wasmtime, so let's test new Rust everywhere, and new Wasmtime features/PRs only on one older Rust.) But I could see us going either way here.

Wasmtime GitHub notifications bot (Aug 31 2023 at 17:20):

alexcrichton commented on issue #6900:

Thanks for writing that up @cfallin and that all looks accurate to me! I'd also agree with your conclusions of N=2 and "test latest, one job msrv" personally.

Wasmtime GitHub notifications bot (Aug 31 2023 at 18:16):

alexcrichton commented on issue #6900:

Ok I've updated this with the various variables involved, but this is to showcase what I think enforcing the proposed MSRV policy would look like. Folks should still weigh in on the specifics if they have thoughts!

• The rust-version field in Cargo.toml is now the "source of truth" about our MSRV
• CI reads the rust-version field and installs rustc "msrv+2" by default
• A new msrv CI builder is added which runs the full test suite on x86_64 Linux on the MSRV
• Documentation is updated about our Rust policy.

Wasmtime GitHub notifications bot (Aug 31 2023 at 18:45):

fitzgen commented on issue #6900:

Thanks for writing that up @cfallin and that all looks accurate to me! I'd also agree with your conclusions of N=2 and "test latest, one job msrv" personally.

+1 on both these things from me as well.

Wasmtime GitHub notifications bot (Aug 31 2023 at 19:15):

tschneidereit commented on issue #6900:

Thanks for writing that up @cfallin and that all looks accurate to me! I'd also agree with your conclusions of N=2 and "test latest, one job msrv" personally.

+1 on both these things from me as well.

And from me, thank you!

Last updated: Oct 23 2024 at 20:03 UTC