Wasmtime GitHub notifications bot (Oct 29 2026 at 23:45): Wasmtime GitHub notifications bot (Oct 29 2026 at 23:45):cfallin opened PR #15083 from cfallin:isle-veri-cache-s3 to bytecodealliance:main:
This PR addresses some problems we've had with the current "cached SMT query solver" implementation we have on top of GitHub CI's builtin cache action. In particular, the current cache action seems to refresh its cache once per night; so on a change that invalidates most of the cache (which may plausibly happen with any crosscutting change to the verifier itself or some base-level lowering rules or helpers), all CI jobs on that day will take multiple hours to run.
This PR instead implements a cache on top of an S3 bucket that we control separately. The basic data structure is a bit like an LSM tree: there is a packfile with sorted keys/values, and there are delta files that can be uploaded with new keys/values. Every so often, the current base packfile and deltas are consolidated into a new base packfile. This design is meant to minimize the number of PUT requests to S3 (which are relatively expensive) and the number of GET roundtrips (which can have nontrivial latency): ideally, we (i) have one or just a few big binary downloads to get the whole cache, and (ii) do no uploads if nothing changed, or one upload with everything that changed otherwise.
The read/write scoping is somewhat subtle too: previously we relied on GitHub CI's built-in key scoping hierarchy, with
mainCI runs implicitly visible to all PR branches, and each PR having its own private key-space. With this PR, we now have a simpler design: CI jobs onmainwill write the S3 cache (using a secret visible only to CI), and jobs on branches will read the cache. The cache bucket is publicly readable, so any job can query it without a secret. This keeps the AWS secret away from CI runs on branches, which we might not have fully vetted yet.With this design, any change that merges to
mainwill have its new cache entries immediately available to all PRs based on it, once it merges, so a cross-cutting change that invalidates a lot of the cache should only cause one long CI run in the merge queue.<!--
Please make sure you include the following information:
If this work has been discussed elsewhere, please include a link to that
conversation. If it was discussed in an issue, just mention "issue #...".Explain why this change is needed. If the details are in an issue already,
this can be brief.Our development process is documented in the Wasmtime book:
https://docs.wasmtime.dev/contributing-development-process.htmlPlease review the Bytecode Alliance's AI tool usage policy at
https://github.com/bytecodealliance/governance/blob/main/AI_TOOL_POLICY.mdPlease ensure all communication follows the code of conduct:
https://github.com/bytecodealliance/wasmtime/blob/main/CODE_OF_CONDUCT.md
-->
Wasmtime GitHub notifications bot (Oct 29 2026 at 23:45):cfallin requested fitzgen for a review on PR #15083.
Wasmtime GitHub notifications bot (Oct 29 2026 at 23:45):cfallin removed the review request for wasmtime-compiler-reviewers from PR #15083.
Wasmtime GitHub notifications bot (Oct 29 2026 at 23:45):cfallin requested wasmtime-compiler-reviewers for a review on PR #15083.
Wasmtime GitHub notifications bot (Oct 29 2026 at 23:45):cfallin requested wasmtime-default-reviewers for a review on PR #15083.
Wasmtime GitHub notifications bot (Oct 29 2026 at 23:46):cfallin commented on PR #15083:
(I've created the appropriate secrets in this repo pointing to the S3 bucket that I created for testing -- let's see if this works!)
Wasmtime GitHub notifications bot (Oct 30 2026 at 02:44):github-actions[bot] added the label cranelift on PR #15083.
Wasmtime GitHub notifications bot (Oct 30 2026 at 02:44):github-actions[bot] added the label isle on PR #15083.
Wasmtime GitHub notifications bot (Oct 30 2026 at 03:45):github-actions[bot] commented on PR #15083:
Subscribe to Label Action
cc @avanhatt, @cfallin, @fitzgen, @mmcloughlin
<details>
This issue or pull request has been labeled: "cranelift", "isle"Thus the following users have been cc'd because of the following labels:
- avanhatt: isle
- cfallin: isle
- fitzgen: isle
- mmcloughlin: isle
To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.
Learn more.
</details>
Wasmtime GitHub notifications bot (Oct 30 2026 at 15:14)::memo: alexcrichton submitted PR review.
Wasmtime GitHub notifications bot (Oct 30 2026 at 15:14)::speech_balloon: alexcrichton created PR review comment:
Could this unconditionally set the
environmentkey? My assumption is that PRs already don't get access to secrets, even if this is set
Wasmtime GitHub notifications bot (Oct 30 2026 at 15:14)::speech_balloon: alexcrichton created PR review comment:
Could
CACHE_MODEbe inferred based on the presence/absence of the access key and secret access key?
Wasmtime GitHub notifications bot (Oct 30 2026 at 15:14)::speech_balloon: alexcrichton created PR review comment:
This file in general is pretty intense -- could this perhaps shell out to the AWS CLI which I believe is installed on github runners instead?
Wasmtime GitHub notifications bot (Oct 30 2026 at 15:15):alexcrichton commented on PR #15083:
I'll also note that ideally we'd leverage S3-based caching (or just something external) for much more than just ISLE, for example integrating
sccache, caching docker images, etc. IMO we should probably replace basically all usage ofactions/cachewith something that uses S3 instead
Wasmtime GitHub notifications bot (Oct 30 2026 at 20:13)::memo: bjorn3 submitted PR review.
Wasmtime GitHub notifications bot (Oct 30 2026 at 20:13)::speech_balloon: bjorn3 created PR review comment:
- You can restrict which workflows and branches can access an environment. Setting the environment here unconditionally means that wouldn't be possible as defense in depth.
- Each time a workflow runs, this causes an entry in https://github.com/bytecodealliance/wasmtime/deployments and in addition, an event is added to the PR that caused the environment saying that it deployed to the environment. So each time you add a new commit to a PR you did get a rather noisy event that is way bigger than a commit event and persists after force pushes.
<img width="643" height="250" alt="noise caused by setting an environment on PR CI" src="https://github.com/user-attachments/assets/be401532-2fda-4f7b-86a3-d0e3edf836da" />
Wasmtime GitHub notifications bot (Oct 30 2026 at 20:15)::speech_balloon: bjorn3 edited PR review comment.
Wasmtime GitHub notifications bot (Oct 30 2026 at 23:04)::memo: cfallin submitted PR review.
Wasmtime GitHub notifications bot (Oct 30 2026 at 23:04)::speech_balloon: cfallin created PR review comment:
I don't understand much about GitHub environments beyond what I briefly read yesterday, but I did believe that this rule was necessary to ensure that secrets would not leak to PRs; and even if technically not with current scoping rules, I'd prefer to have defense-in-depth.
Reading the page in settings, I see: "They are not passed to workflows that are triggered by a pull request from a fork." That seems to imply (by omission) that secrets are present for PRs from branches within the main repo. I don't want to grant access to the secrets to anyone who has collaborator access on their own unreviewed PRs, so I do think we need an explicit condition here (or at least, that's my reading!).
What do you think?
Wasmtime GitHub notifications bot (Oct 30 2026 at 23:07)::memo: cfallin submitted PR review.
Wasmtime GitHub notifications bot (Oct 30 2026 at 23:07)::speech_balloon: cfallin created PR review comment:
Sure, I can take a look at this next week! My intent was to make this usable for developers running client-side too, for what it's worth, where I would not necessarily want to require them to install the AWS SDK. With the public-readonly ACL on the bucket, the idea is that folks could let the tool download the cache when the run locally and it should Just Work.
The other options seemed to be either a little 400-line handwritten (err, robot-written) S3 client with just the methods we need, or the full AWS Rust SDK (and vet it + deps).
What do you think?
Wasmtime GitHub notifications bot (Oct 30 2026 at 23:07)::speech_balloon: cfallin edited PR review comment.
Wasmtime GitHub notifications bot (Oct 30 2026 at 23:09)::memo: bjorn3 submitted PR review.
Wasmtime GitHub notifications bot (Oct 30 2026 at 23:09)::speech_balloon: bjorn3 created PR review comment:
That seems to imply (by omission) that secrets are present for PRs from branches within the main repo.
I believe so yeah.
so I do think we need an explicit condition here (or at least, that's my reading!).
Yes and you will have to restrict the environment to the main branch (+ branch protection to prevent direct pushes) to prevent a contributor from just modifying the workflow to use the environment on their branch.
Last updated: Oct 31 2026 at 23:03 UTC