wasmtime / issue #13576 Continuous Sightglass benchmarking · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / issue #13576 Continuous Sightglass benchmarking

Wasmtime GitHub notifications bot (Jun 05 2026 at 22:33):

We've wanted continuous Sightglass benchmarking for Cranelift and Wasmtime for a looong time; we are finally getting to the point where we can prioritize it. But first we need to agree exactly what we measure, how we measure it (i.e. high-level consensus on architecture), and what we do with those measurements (i.e. how we display them, what kind of automatic alerting/issue-filing/etc we do).

Benchmarking is hard. Lots of people have opinions on how it should best be done. Historically, this is a topic that has attracted a lot of drive-by opinions. I ask that people kindly refrain from engaging on this issue unless you are regularly hacking on Wasmtime, Cranelift, or Sightglass.

I think we have roughly two separate use cases for continuous benchmarking:

Viewing long-term trends over time. Basically, drawing arewefastyet.com-style graphs where the x axis is ~nightly Wasmtime builds and the y axis is some metric of performance. Gives us a rough sense of how performance today is compared to last week/month/year/etc.

Catching and reporting performance regressions as quickly as we can. Basically, do a statistical significance test between ~yesterday's main and today's main. If the difference is significant and performance got worse, then open an issue detailing the regression's commit range and effect size. Helps us avoid unwittingly regressing performance. This could also be triggered explicitly by Wasmtime/Cranelift maintainers for a particular PR, to compare its performance against main.

These two use cases have (or can have) different requirements:

(1) must have either a dedicated machine or some other manner of making results recorded on different machines ~deterministic (e.g. measure with callgrind rather than native cycles) or else the trend over time is meaningless.

However, (2) doesn't necessarily need dedicated hardware or even the ability to compare results recorded on different machines: it can build yesterday's and today's Wasmtime, run them both at the same time on the same current machine, and then do a statistical significance test between their results, and it doesn't matter if this process happened on a different machine two days ago because it isn't reusing that data.

But if it did have access to dedicated hardware, it could potentially reuse yesterday's data, instead of re-running yesterday's Wasmtime

(1) requires saving the results (or at least a summary of the results) of historical benchmark runs

(2) does not necessarily need that

I just point this out to make sure we don't conflate the two use cases and over-constrain one because of the requirements of the other.

Which measurements should we record?

Sightglass supports recording a variety of measurements:

Instructions retired

Cycles

Wall time

Various perf counters, notably cache accesses and misses

Simulated (and therefore ~deterministic across machines) microarch details via callgrind:

instructions-retired

data-reads

data-writes

l1-icache-misses

l1-dcache-read-misses

l1-dcache-write-misses

ll-icache-misses

ll-dcache-read-misses

ll-dcache-write-misses

conditional-branches

conditional-branch-misses

indirect-branches

indirect-branch-misses

(We don't currently but we could also compute "virtual cycles" or "virtual wall time" based on this data, which should also be ~deterministic across machines)

Doing everything is going to be too noisy and give us information overload. It will also take more resources. So I think we really do want to narrow things down here, ideally to a single measure.

Which should we use for use case (1), long-term trends over time?

Which should we use for use case (2), significance tests between today's and ~yesterday's Wasmtime to catch performance regressions quickly?

How do we measure it?

What's the high-level architecture? Dedicated hardware? How does it integrate into github and actions?

What do we do with it?

We've recorded some benchmark data, what actions do we take based on its results? Do we plot it and serve it on a web page? Do we file issues? Etc...

My Recommendation: Two separate jobs

Use case (1), long-term trends over time

Measure: virtual cycles/wall-time based on callgrind simulation, so we can avoid dedicated hardware but still measure roughly the thing we really care about (wall time) as opposed to things that are too far-removed (like instructions retired)

How:

make a bytecodealliance/arewefastyet repo (or other name)

github daily cron action to fetch the latest Wasmtime and run Sightglass

save the raw benchmark data as a file (it will be small since we only need ~3 iterations for callgrind because it is very low noise)

run a script to render/summarize all historical data, write it to a canonical file

index.html loads that summary and displays it

very simple; no dedicated hardware involved, no S3, no CDN, etc...

What:

two scatter plots for each benchmark: compilation and execution

x axis: date of wasmtime build

y axis: performance (virtual wall time)

Use case (2), significance tests between today's and ~yesterday's Wasmtime

Measure: cycles

How:

github action/workflow in the Wasmtime repo

check out today's and yesterday's Wasmtimes

build both of their libwasmtime_bench_api.sos

run sightglass, passing -e yesterday.so -e today.so

if there is any statistically significant regression, open an issue detailing the regression's commit range and effect size

What:

No webpage, no graphs/plots

Just the issues opened on regression

Or a comment on a PR, if explicitly triggered for a particular PR (main vs PR branch, instead of today vs yesterday)

Issue/comment has an HTML table with a row for each regression, columns for benchmark, phase, and effect size, something like this:
| benchmark | phase | effect size |
|-----------|-------|-------------|
| bz2.wasm | compilation | 1.23x +/- .03 more cycles |

Issue/comment also has info on how to run locally

I could also be convinced we should reuse the virtual wall time data from use case 1, rather than rebenching, but that wouldn't play nice with the benchmark-this-particular-PR-against-main sub-use case.

cc @alexcrichton @cfallin @ricochet

Wasmtime GitHub notifications bot (Jun 05 2026 at 23:19):

alexcrichton commented on issue #13576:

Thoughts, in no particular order:

I agree about the most interesting metrics here, but I think we should retain and have a "secondary" display for all metrics we gather. For example in the perf-over-time there'd be a little drop down of all metrics and by default it's be cycles or instructions, but you could set it up to be cache-misses if you really wanted. My hope is that this would be a constant factor more of plumbing to parameterize metrics and generate graphs/divs/etc but wouldn't be too too onerous (even on the storage side of things). I think this would be useful in regression reports since if one "weird" metric regresses and everything else stays the same that's probably still quite interesting.

I like starting simple on the rendering side of things, basically generating a "database" in a git repo for now and having the cron job slurp up the database and render html. I suspect this will get fancier over time, but seems like a fine starting spot and relatively easily migratable to another format should we so choose.

Personally I'm dubious about the assertion that we can use github actions to perform a regression test. I don't dispute that recording/using the same machine reduces the noise, but I'd still predict that despite that it'd be quite noisy. Modeling this as github actions, though, gives us an "out" to eventually doing this on custom hardware (or, rather, rented in some data center). In that sense I don't think I'd change much about this.

I'll have a lot of thoughts of how exactly this is all implemented w.r.t. the exact github actions integration, triggers, repo/script setup/etc. I'll probably have very few opinons about the format of data storage or precisely how it's rendered though. I'm pretty confident though that such opinions don't affect anything here and is something that can be handled during review.

For comparisons with PRs, you'll want to use the merge-base of main and the PR rather than main in case that's diverged.

For trends over time, one piece of data that I think should be collected, per-day, is the commit of the automation repo that the day was using. That way we can track down things like what rustc version was used by looking at the actions configuration.

Wasmtime GitHub notifications bot (Jun 08 2026 at 16:38):

fitzgen commented on issue #13576:

@alexcrichton

I agree about the most interesting metrics here, but I think we should retain and have a "secondary" display for all metrics we gather. For example in the perf-over-time there'd be a little drop down of all metrics and by default it's be cycles or instructions, but you could set it up to be cache-misses if you really wanted. My hope is that this would be a constant factor more of plumbing to parameterize metrics and generate graphs/divs/etc but wouldn't be too too onerous (even on the storage side of things). I think this would be useful in regression reports since if one "weird" metric regresses and everything else stays the same that's probably still quite interesting.

The tricky bit here isn't the graphing or storage side of things, I don't think.

We can't record all metrics at the same time (e.g. you really don't want to measure wall time while running the programs under valgrind), so I think this would unfortunately mean running the full suite multiple times with different metrics enabled and making a "full run" take N times longer from an outside perspective. That said, we could record all of the perf counters at once or all of the valgrind counters (and the virtual wall time derived from them) at once and have graphs for each of those things, depending on which we chose.

Also, it isn't clear which use case you are thinking about here, but we also probably wouldn't want to mix machine ~independent metrics with machine dependent ones, at least for the trends-over-time use case.

Personally I'm dubious about the assertion that we can use github actions to perform a regression test. I don't dispute that recording/using the same machine reduces the noise, but I'd still predict that despite that it'd be quite noisy. Modeling this as github actions, though, gives us an "out" to eventually doing this on custom hardware (or, rather, rented in some data center). In that sense I don't think I'd change much about this.

Yeah, that's fair and I'm not 100% sure it will work out either. But I agree it is a good place to start, and if we have to move to dedicated hardware or something, then at least we can evolve what we have while maintaining nice github integration.

For comparisons with PRs, you'll want to use the merge-base of main and the PR rather than main in case that's diverged.

Yep, good call.

For trends over time, one piece of data that I think should be collected, per-day, is the commit of the automation repo that the day was using. That way we can track down things like what rustc version was used by looking at the actions configuration.

Ah great point. We should probably capture both the sightlgass commit and the infrastructure commit. Well I guess the latter could presumably allow us to recover the former, so maybe the latter is good enough.

Wasmtime GitHub notifications bot (Jun 08 2026 at 18:27):

alexcrichton commented on issue #13576:

Ah ok yeah I misunderstood, I assumed a collection contained all metrics and we selectively threw out some. If we only get a subset of metrics from one run that seems like the place to start. For over-time metrics I'd assume everything would go through callgrind/cachegrind and we wouldn't use perf at all (due to machine variance). For head-to-head runs though running twice to get lots of metrics may be plausible, but if callgrind/cachegrind are good enough over time I figure they'd be reasonable to head-to-head as well.

Wasmtime GitHub notifications bot (Jun 09 2026 at 18:56):

cfallin commented on issue #13576:

All of this sounds reasonable to me!

Nick and I talked about quite a bit of this previously; the most interesting question was how to get long-term trends in the face of changing hardware. The callgrind (simulated cache/memory effects) runs get a first-order approximation of that and are (hopefully almost perfectly) deterministic over time.

One other idea I floated was to do a kind of "skiplist differential wallclock run": at any given T, do a differential run against T-1, T-2, T-4, T-8, T-16, T-32, ... (or pick a higher geometric base to go sparser). With the right kind of aggregation, and interpolation, we could combine that with the day-by-day differential run ratios; the "against latest" run disciplines/refines the accumulated error when multiplying day-by-day ratios together over time. (Given a historical recording of all of these logN datapoints per day, we can get from T to any T-k in logN steps and so have asymptotically better error bounds.) That allows us to build wallclock trends over changing underlying hardware.

That's pretty "out there", though, and absolutely we should get the basic thing working first. I'm fully in favor (with only weakly held opinions) of whatever way we can manage to do that.

The only other more strongly-held opinion I have is that I don't think it's worth doing any real wallclock measurements on GitHub runners; they're going to be too noisy. Look at the work that Jamey did back in the day to figure out how to isolate a CPU core for reliable runs: disable interrupts, edit system config to keep all other processes off of a core, pin dynamic core frequency scaling to a fixed frequency, etc. Even then we still had nonzero error bars. For that reason I think it makes sense to focus only on callgrind-produced "synthetic wallclock" plus real instruction counts for now; we're thus portable; and if/when needed we can rent a dedicated machine (not a VM -- same noisy-neighbor problem), isolate a core, and do real measurements that way.

Wasmtime GitHub notifications bot (Jun 10 2026 at 20:11):

fitzgen commented on issue #13576:

Thanks Alex and Chris! It sounds like we have consensus on the following as an initial milestone. Please confirm so that we can start building out the infra without having to throw away work or whatever.

Use case (1), long-term trends over time

Measure: callgrind-based metrics, plus the "virtual cycles" derived from them

How:

make a bytecodealliance/arewefastyet repo (or other name)

github daily cron action to fetch the latest Wasmtime and run Sightglass

save the raw benchmark data as a file in the repo (it will be small since we only need ~3 iterations for callgrind because it is very low noise)

run a script to render/summarize all historical data, write it to a canonical file in the repo

commit new data and updated summary, push

index.html is served via gh-pages, loads that summary and displays it

What:

a scatter plot for each combo of benchmarks x phases x metrics, where:

benchmarks is our set of benchmarks

phases is { compilation, instantiation, execution }

metrics is the set of metrics that we record under callgrind, plus the "virtual cycles" metric derived from them

x axis: date of wasmtime build

y axis: metric

Use case (2), significance tests between today's and ~yesterday's Wasmtime

Measure: virtual cycles derived from callgrind metrics

How:

daily cron action to look at today's and yesterday's data already recorded and committed in repo from the previous use case's cron job

run sightglass-cli effect-size ... to do a significance test and get the effect size of any statistically significant regressions or speedups

What:

No webpage, no graphs/plots

File new issues on Wasmtime repo on statistically significant regressions

Issue has an HTML table with a row for each regression, columns for benchmark, phase, and effect size, something like this:
| benchmark | phase | effect size |
|-----------|-------|-------------|
| bz2.wasm | compilation | 1.23x +/- .4 more virtual cycles |

Issue also has info on how to run locally

Issue also has regressing commit range / git log

Sound good?

Wasmtime GitHub notifications bot (Jun 10 2026 at 20:21):

alexcrichton commented on issue #13576:

:+1:

Wasmtime GitHub notifications bot (Jun 10 2026 at 20:27):

cfallin commented on issue #13576:

:+1:

Last updated: Jul 29 2026 at 05:03 UTC