Stream: cranelift

Topic: Cache busting & ISLE verification

Alex Crichton (Oct 29 2026 at 21:28):

We've got a bit of a weird situation right now which has happened a few times now but I wanted to note it. Just recently https://github.com/bytecodealliance/wasmtime/pull/14964 landed which I believe invalidated all prior ISLE caches I assume due to structural changes since it's ISLE-related. This means that PR, expectedly, took ~4h in the merge queue to run full verification.

The problem though is that every PR right now is going to take 4h to do a full verification as well. We've got a 13-deep merge queue at the moment where everything active is doing ISLE verification. I believe the reason for this is that github caches for the merge queue restore from main, but the main cache is only populated once a night when a manual run is done on main.

I'm not really sure of a great fix for this myself other than maybe more aggressively pulling from the dev release as opposed to github caches. I've long not really liked github caches and how they're affine to branches since it's kind of never what we actually want for this very reason...

Chris Fallin (Oct 29 2026 at 21:34):

the main cache is only populated once a night when a manual run is done on main.

ah, shoot, that explains it -- privately we were puzzling over why the cache wasn't hitting...

Chris Fallin (Oct 29 2026 at 21:35):

This is a tricky one. This kind of nondeterminism/brittleness is what I was worried about when proposing to check in the cache into git instead; of course there are other good reasons we didn't do that (trust root, non-invasiveness, merge conflicts). I worry that a best-effort "slurp up the cache from [all the sources] to try to get latest runs" may still be brittle in some cases. But maybe that's the best we can do...

Chris Fallin (Oct 29 2026 at 21:36):

I'll see if I can put together a PR to do that. In the meantime if we want to unblock, I'm happy to comment out the isle-veri job and jump the queue with that if folks agree

Alex Crichton (Oct 29 2026 at 21:37):

nah it's not the end of the world really, it just means that everything will build up a big queue and land all at once after we all go to sleep

Alex Crichton (Oct 29 2026 at 21:37):

ideally what I want, for other reasons too, is to get off of github actions' cache onto something manual like azure or aws

Alex Crichton (Oct 29 2026 at 21:37):

With the hashing we have we basically have no need for branch-affine caches

Alex Crichton (Oct 29 2026 at 21:38):

I want any cache-write from any merge queue entry to get visible to all the others immediately

Chris Fallin (Oct 29 2026 at 21:38):

right, does our GitHub/Microsoft relationship give us any Azure bucket credits or anything? it wouldn't be that hard to do a content-addressed thing

Alex Crichton (Oct 29 2026 at 21:38):

Not that I know of

Alex Crichton (Oct 29 2026 at 21:38):

I want to turn this on for sccache too as well

Chris Fallin (Oct 29 2026 at 21:39):

this seems like a good question for the BA TSC/board -- if BA has budget for some cloud storage, we could do Good Things (cc @Oscar Spencer @Till Schneidereit)

Alex Crichton (Oct 29 2026 at 21:39):

In the meantime though I mostly just wanted to flag the issue here, the ISLE bits aren't changing that much so this doesn't happen like every day

Till Schneidereit (Oct 29 2026 at 21:40):

we already discussed this, and do indeed have the budget. Setting it up turned out to be the more difficult thing

Chris Fallin (Oct 29 2026 at 21:40):

ah, interesting

Chris Fallin (Oct 29 2026 at 21:40):

to the extent that it's not really feasible, or could someone (say, one of us) spend some time to look into it more?

Till Schneidereit (Oct 29 2026 at 21:42):

the latter. I started looking into it, and then was eaten by a grue ran into SSO auth issues when trying to connect our azure marketplace account to something that supports billing, or somesuch thing

Chris Fallin (Oct 29 2026 at 21:46):

that sounds like a fairly... nightmarish... nightmare, but if there's a way I can help (with what access I have anyway), let me know

Till Schneidereit (Oct 29 2026 at 21:51):

I wonder if the easiest thing to do might be to try it out with some account and get things hooked up with a BA credit card it if works. I'd be happy to pay for a month of it myself, but I also don't have the bandwidth to do the setting up :confused:

Chris Fallin (Oct 29 2026 at 21:54):

I mean, I have AWS already with autobilling (all my personal DNS+web+...), and S3 is extremely cheap, so I'm happy to play with it on a personal access token scoped to one bucket, as long as the endgame is "BA provides an IAM access token for production S3"

I don't know anything about secrets management in GitHub CI but I can figure that part out...

Till Schneidereit (Oct 29 2026 at 21:55):

if you're willing to do that, that I'll gladly commit to fighting the grue once it's time to hook up billing

Till Schneidereit (Oct 29 2026 at 21:56):

note that with AWS egress costs might be a concern

Alex Crichton (Oct 29 2026 at 21:56):

my rough assumption for the architecture is that we'd have a bucket somewhere which is world-readable and readable without a token (e.g. PRs which don't get secrets can get cache hits), and then if egress costs are a problem we'd put a CDN in front of it

Chris Fallin (Oct 29 2026 at 21:56):

looking at the pricing table (https://aws.amazon.com/s3/pricing/) the other thing that might bite is us per-request costs, unless we batch somehow -- a GET per SMT query could add up at 1k req for a half-cent (with a few thousand requests per run)

Chris Fallin (Oct 29 2026 at 21:56):

so maybe we need to invent the equivalent of git packfiles or something

Alex Crichton (Oct 29 2026 at 21:57):

nono we'd still have that be just "one file"

Alex Crichton (Oct 29 2026 at 21:57):

but I'd want to ideally cache a whole bunch of things here, docker images, crates, ccache, etc

Alex Crichton (Oct 29 2026 at 21:57):

that's in theory where a CDN would help though since those typically have far cheaper egress costs

Chris Fallin (Oct 29 2026 at 21:57):

ah so you're imagining the whole cache tarball (or equivalent) as one file in a bucket -- I was thinking each SMT query hits a different key so we effectively get merge/union semantics for free (versus "which version of the cache bundle do I take")

Alex Crichton (Oct 29 2026 at 21:58):

that's true yeah, I don't really mind so long as it works heh

Alex Crichton (Oct 29 2026 at 21:58):

and doesn't costs someone dollars-per-run

Chris Fallin (Oct 29 2026 at 21:58):

I'll play with this a bit

Till Schneidereit (Oct 29 2026 at 22:05):

I can see if we can sort something out, but wouldn't want to get anyone's hopes up. And maybe @Tyler McMullen could get something working with Fastly?

Chris Fallin (Oct 29 2026 at 22:56):

Well, things escalated with the robots and I have an S3-backed SMT query cache now. Little LSM-tree-like cache (base packfile, delta of new query results, compaction after 10 deltas, read/write on main and readonly on PR branches); works locally on my own S3 bucket. 3MB download from S3 per CI run in 1..10 GETs (.0004/1k GETs, so cheap). No GC, grows forever, we can do a separate GC pass later if we want but 3MB is tiny

Is there a way to add my own role-limited IAM token on my personal AWS account to BA's secrets so I can test it in a PR? I'm willing to subsidize for a bit too if it takes time to fight the grues

Alex Crichton (Oct 29 2026 at 22:57):

IIRC what we did in rust-lang/rust was to expire everything after a month or so of inactivity so s3 sort of takes care of the GC for us

Chris Fallin (Oct 29 2026 at 22:57):

Ah, that's a good idea

Alex Crichton (Oct 29 2026 at 22:57):

or basically I think there's some option per-bucket to do that

Till Schneidereit (Oct 29 2026 at 22:57):

that sounds great! And you should be able to add the secret to the repo with your access rights

Chris Fallin (Oct 29 2026 at 22:57):

cool, I will dig around

Alex Crichton (Oct 29 2026 at 22:58):

although for testing I'd recommend running in your personal fork if you can

Alex Crichton (Oct 29 2026 at 22:58):

not the full matrix of course but subsets

Chris Fallin (Oct 29 2026 at 23:47):

https://github.com/bytecodealliance/wasmtime/pull/15083 -- I went ahead and just put the secrets in the repo and am testing here so I don't burn my own minutes / go through the trouble of testing on my fork

Chris Fallin (Oct 29 2026 at 23:47):

("secrets in the repo" meaning in secrets management under an environment visible from CI)

Chris Fallin (Oct 29 2026 at 23:51):

napkin math is ~$0.0001/month of storage (5MiB steady-state), a fraction of a cent per day in PUT/GET reqs, and maybe $4-5/month in egress for 100 CI jobs/day (note that job is gated on ISLE changes)

Chris Fallin (Oct 29 2026 at 23:55):

OK, that run worked -- merging this and jumping it to head of queue should fix our CI delays today

Chris Fallin (Oct 30 2026 at 00:28):

To be clear with current state: I'm happy to see that merged as-is, if r+'d, and let it use my bucket for now until folks figure out how to create one that's billed to BA

Last updated: Oct 31 2026 at 23:03 UTC