Stream: git-wasmtime

Topic: wasmtime / issue #14313 Proposal: first-class Instance::r...

Wasmtime GitHub notifications bot (Sep 10 2026 at 20:10):

stevendore opened issue #14313:

Feature

We embed wasmtime in a high-throughput HTTP proxy (the embedding from #14312) and pool WASI 0.2 component instances across requests, resetting each instance to pristine state between checkouts. Today we do this from the embedder side via two small introspection accessors we carry as a local patch. That patch is ~90 additive lines with tests. We are working through our company's open source approval process to be able to offer it upstream, and will link the PR here once we can. Either way it is the minimal primitive, not the complete solution: this issue proposes the runtime-owned reset that the accessors cannot deliver.

The feature: a way to reset a live instance back to its just-instantiated state, for example component::Instance::reset(&self, store) -> Result<()>, and possibly a core Instance::reset as well. Reset rewinds linear memories, tables, mutable globals, and internal flags (data/elem drops, lazy funcref init) so the instance is indistinguishable from a fresh instantiation of the same InstancePre in the same store, without re-running start or component initializers.

Benefit

It makes instance reuse safe for untrusted guests, which is currently a gap. wasmtime serve reuses instances by trusting the guest to keep itself request-ready (#9542), default-on for WASIp3 but deliberately default-off for WASIp2 (max_instance_reuse_count defaults to 1). Embedders whose p2 guests are untrusted or operator-supplied cannot opt into trust-based reuse at all, and per-request instantiation is a host-wide kernel ceiling, not a per-core cost.

The three strategies below all give every request a pristine instance. They differ in who restores the state and whether the kernel is involved:

• fresh instantiate (on-demand): new Store + instantiate per request on the default allocator. Each instance mmaps its linear memory and unmaps it on drop, so every request pays kernel address-space work that serializes on the process mmap_lock.
• fresh instantiate (pooling allocator): same per-request instantiate, but instances come from the pooling allocator's pre-reserved slab (no per-instance mmap). Recycling a slot decommits its dirty pages back to the CoW image with madvise, which costs TLB-shootdown IPIs across the process's CPUs.
• reset-based reuse (embedder-side): instantiate once, then per request: call, then reset in userspace. The reset memcmps each 4 KiB page against a post-instantiate snapshot, copies back only the pages that differ, and rewinds mutable globals via the accessors. No syscalls at all.

Measured on a dual-socket Xeon Gold 6330 (2x28 cores, 112 threads with SMT), dispatch-bound component call, aggregate calls/s:

threads	fresh instantiate (on-demand)	fresh instantiate (pooling allocator)	reset-based reuse (embedder-side)
1	26.0K	36.7K	143.9K
8	26.3K	92.8K	971.3K
96	21.3K	191.1K	4.32M

On-demand instantiation is flat from 1 to 96 threads, with per-call latency inflating from 0.026 ms to 2.88 ms as threads queue on the kernel. The pooling allocator is better but stalls near 191K/s: only 2x more throughput for 12x more threads past 8. The userspace reset scales with cores: 22.6x the best instantiate-per-request strategy at 96 threads. Per-call isolation overhead at 1 thread: 25.2 us (on-demand) and 17.1 us (pooling) vs 5.8 us (reset).

A runtime-owned reset would beat our embedder-side numbers further and close a correctness gap at the same time:

• The runtime owns the memory images (MemoryImageSlot) and the PAGEMAP_SCAN machinery, so discovery can be O(dirty pages) with no embedder snapshot. Our embedder compare pass is O(total memory): about 2.5 us at 256 KiB but 230 us at 16 MiB, where PAGEMAP_SCAN stays at 2 to 8 us for small dirty counts.
• The runtime can reach state no public API can: table mutations (table.set, elem.drop effects), data/element drop flags, additional linear memories, and component-layer runtime state. Our embedder reset compensates with a load-time self-check and by constraining accepted modules. A first-class reset deletes that class of edge entirely.
• It could make p2 reuse in wasmtime serve safe by default rather than trust-gated.

Implementation

Semantics: rewind to the post-instantiate point, as above. The runtime already holds everything needed: the memory image as the pristine reference, initial global values and element segments from the module, and the internal flags.

Two mechanism notes from our production experience that a design should account for:

• Restoring page contents is not restoring page state. A memcpy restore re-writes the page, so it stays an anonymous WRITTEN copy and every subsequent reset re-copies it. Per-reset cost ratchets toward O(cumulative-touched) for both PAGEMAP_SCAN and compare discovery (at 4 MiB: 2.0 us/reset at 8 touched pages, 161.7 us at all 1024). Only decommit (madvise back to the image) re-arms clean kernel state.
• Decommit has a concurrency cost microbenchmarks miss: a madvise-based reset collapsed for us at ~256 concurrent connections on the same 112-thread host (TLB-shootdown IPI storms). A reset API should be able to choose per page between memcpy and decommit, and expose a shootdown-free mode. Periodic full recycle stays structurally necessary regardless (as wasmtime serve already ships via max_instance_reuse_count).

There are also aspects of the runtime we are not familiar enough with to know how they would interact with a reset: fuel and epochs, async and concurrent state, resource handles, resetting while other instances share the store, and whether the pooling allocator could service a reset without unsharing images. We defer to maintainers on all of those. Happy to share the full benchmark methodology and an embedder's-eye API review now, and to contribute the accessors patch and implementation work under maintainer direction once our approval process completes.

Alternatives

• What we run today: embedder-side reset via two introspection accessors (Instance::core_instances, Instance::defined_globals, ~90 additive lines, PR pending our company's open source approval). Snapshot post-instantiate state, then compare-and-copy linear memory and rewind mutable globals on checkout. Proven in production and the source of the numbers above, but this approach has a hard ceiling no accessor extension can lift: discovery is O(total memory) per reset (kernel dirty-page tracking does not help an embedder, whose own memcpy restores keep pages marked written forever, per the ratchet note above), and tables, drop flags, extra memories, and component-layer state stay out of reach. The O(dirty) column and the completeness both require the runtime-owned reset. The accessors are still worth landing on their own: they are the minimal primitive that works today and remain useful plumbing under any future reset.
• Memory-only Memory::reset_to_image(&self, store): a much smaller design surface that exposes the existing PAGEMAP_SCAN plus image-slot machinery. Delivers the O(dirty) column, and globals are already coverable from the embedder side with the accessors, leaving tables and drop flags as the residual gap.
• Fresh instantiate per request: the safe default today, but the table above shows it is a host-wide ceiling (21K to 191K calls/s regardless of core count) that binds at proxy request rates.
• Trust-based reuse (#9542 temporal isolation): works for trusted p3 guests, unusable when guests are untrusted or operator-supplied, which is exactly the p2 population.
  

Wasmtime GitHub notifications bot (Sep 10 2026 at 20:43):

pchickey commented on issue #14313:

This reads to me as LLM-generated english to the point where I can't be confident that you (the human) and I share any understanding via this artifact, so I'd like to direct you to https://github.com/bytecodealliance/governance/blob/main/AI_TOOL_POLICY.md, once you have made any edits required to comply with that policy you can respond and I'll try to re-read.

Wasmtime GitHub notifications bot (Sep 10 2026 at 22:14):

alexcrichton commented on issue #14313:

Benchmarking-wise for something like this it's expected to basically ignore the deafult instance allocator as that's known to have poor performance in this scenario. The pooling allocator, however, is intended to be tailor-made for this sort of scenario, and that's the benchmark we typically use.

Otherwise the numbers you've measured here look a bit suspect to me -- low hundreds-of-thousands is roughly what I'd expect for a setup like this, but low-millions is quite surprising to achieve. That'd be great if we could get close to that number, but I'd want to dig into what you're doing. It sounds like you're more-or-less doing a memset of linear memories to reset them, and I'd expect any non-negligible-sized linear memory to tank performance with such a memset. I realize you're describing a compare-and-only-write-if-different strategy which might help perf a bit, but I wouldn't expect that to be an order of magnitude faster than a memset.

Put another way -- in addition to turning down the LLM output a bit it'd be helpful to be able to poke around what you're comparing. Whether or not this is a true reset of instance state matters for correctness and is something at least I'd want to review.

Orthogonally I might also recommend investigating tuning pooling allocator options:

• linear_memory_keep_resident - if turned up high enough this should basically be equivalent to your approach of resetting memory back to its original contents.
• pagemap_scan in combination with linear_memory_keep_resident can greatly reduce the size of the memory that's reset, but it does require a syscall.
• For highly-concurrent situations I'd recommend using the latest Linux kernel and then with decommit_batch_size you might be able to hit the process_madvisev path which could reduce IPIs and TLB contention.

Wasmtime GitHub notifications bot (Sep 15 2026 at 02:40):

stevendore commented on issue #14313:

Thank you for the tuning suggestions, they helped in all cases. linear_memory_keep_resident and pagemap_scan(Auto) were the most helpful, and those two are what the tuned pooling arm below uses.

Overall I work with pretty small wasm guests (~137 KB .wasm, 256 KiB linear memory, only a few functions each) and found that only resetting the memory space is more efficient for my use case of needing a fresh image on every request. I put the whole comparison in a gist, every permutation we discussed plus the reset with and without the tuning:

https://gist.github.com/stevendore/52c9458e09c5342e93f61bfbe9f7787f

Two files: the accessors patch and a single test file that drops into crates/wasmtime/tests/ of a patched checkout. The test file has a correctness test that runs in the normal suite and the benchmark behind #[ignore]. Run steps are in the file header, with MEM_PAGES / THREADS / SECS / ROUNDS / DIRTY knobs to reproduce everything below. The benchmark guest is a minimal embedded component whose linear memory defaults to 256 KiB, matching my production guests, and MEM_PAGES scales it.

The permutations, all giving each call a pristine instance except the floor:

• fresh (on-demand): new Store + instantiate per call, default allocator
• fresh (pooling): pooling allocator with default knobs
• fresh (pooling tuned): same pooling + linear_memory_keep_resident covering the guest + pagemap_scan(Auto)
• reset-reuse: call + userspace reset (memcmp per 4 KiB page vs a post-instantiate snapshot, copy back diffs, rewind mutable globals) via the two accessors
• reset on tuned pooling: same reset, on the tuned pooling engine
• no-reset floor: call same instance without a reset

The table numbers are from a 2x28-core Xeon (112 threads with SMT), kernel 6.9.6 (pagemap active), DIRTY=2, 3 rounds x 5s per arm.

256 KiB guest (my case), total calls per second across all threads:

THREADS	fresh on-demand	fresh pooling	fresh pooling tuned	reset-reuse	reset on tuned pooling	no-reset floor
1	52.8K	126.6K	201.6K	218.4K	217.0K	1.96M
8	49.4K	399.0K	885.4K	1.76M	1.76M	15.6M
32	41.5K	901.7K	1.92M	5.88M	5.99M	52.6M
96	35.3K	1.04M	2.54M	10.7M	10.8M	89.5M

The reset is a full memcmp of the linear memory per call, so its cost grows with guest size while the pooling arms stay flat. Crossover was around 0.5 MiB in my tests.

4 MiB guest (MEM_PAGES=64), total calls per second across all threads, shows the existing pooling allocator is more performant:

THREADS	fresh on-demand	fresh pooling	fresh pooling tuned	reset-reuse	reset on tuned pooling	no-reset floor
8	50.6K	403.1K	766.1K	129.8K	129.5K	15.7M
32	43.7K	884.8K	2.08M	414.9K	426.0K	52.0M

Tuned pooling wins by 5-6x at this size. The two reset columns are within a few percent of each other everywhere, the reset does not touch the allocator, so the tuning only shows up when instances get created (pool fill, periodic recycle).

Wasmtime GitHub notifications bot (Sep 15 2026 at 17:54):

alexcrichton commented on issue #14313:

One thing I notice in your code is that you're not using InstancePre for repeated instantiation, so I'd definitely recommend updating that.

Otherwise though could you perhaps perform analysis to determine why your strategy is faster than the "fresh pooling tuned" column? For example are syscalls the bottleneck? Memory traffic? etc.

One other possible dimension is that you can forcibly disable PAGEMAP_SCAN and then also set keep-resident values high. That in theory should be a head-to-head comparison of your memory reset logic with this reset logic. It should in theory be possible to tweak the built-in logic to basically be the same as your logic so I would expect that in the limit these should basically perform the same.

Wasmtime GitHub notifications bot (Sep 24 2026 at 16:48):

stevendore commented on issue #14313:

Thanks for the questions. Working through them, I realized guest size explains most of what I was seeing. A fresh instantiate has a fixed cost per call, any memory reset scales with the guest, and the two cross around 0.5 MiB. My production guests are 256 KiB, so reuse plus reset comes out ahead for me.

I updated the gist. Every strategy now instantiates through InstancePre, and there are two new knobs: PAGEMAP=0 forces the scan off for the head to head, and ARM= runs a single strategy per process so profiles are not mixed. It also has the flame graphs from the profiling below, captured with the kernel included:

https://gist.github.com/stevendore/173ef4901f25396f6b86d239c3dfd02c

One thing I notice in your code is that you're not using InstancePre for repeated instantiation, so I'd definitely recommend updating that.

Good find, thanks. It's fixed in the gist above. The effect is small because this guest imports nothing, and no ranking changed. Everything below is on the updated bench.

could you perhaps perform analysis to determine why your strategy is faster than the "fresh pooling tuned" column? For example are syscalls the bottleneck? Memory traffic? etc.

I profiled each strategy in its own process, at 8 threads with a 256 KiB guest. Default pooling makes one madvise per call and tuned pooling makes one PAGEMAP_SCAN ioctl, while the reset makes no syscalls at all. Tuned pooling costs 26.9K cycles per call with 13.5% of that in the kernel, and the reset costs 13.8K with 0.6% in the kernel. For comparison, a bare call with no isolation is 1.7K. The flame graphs in the gist show where the time goes:

• after_pooling_tuned.svg: about 12% is the PAGEMAP_SCAN ioctl (the __x64_sys_ioctl / do_pagemap_scan tower on the right), and the other wide frames are instantiate work like StoreOpaque::allocate_instance, the slot index allocator, and ComponentInstance drop.
• after_reset_reuse.svg: 81% is apply_reset, and 74% is a single memcmp frame running at IPC 3.0. There are no kernel or allocator frames.

Of the 13K cycle gap, about 3.6K is the ioctl and about 9.5K is instantiating, so it's mostly not syscalls. To rule out memory traffic, I ran DIRTY=64 so both sides write every page, and the reset still led by 1.8x to 2.1x. None of the memory tuning touches that instantiate cost, which I think is why your suggestions sped up the fresh strategies without changing the order.

you can forcibly disable PAGEMAP_SCAN and then also set keep-resident values high. That in theory should be a head-to-head comparison of your memory reset logic with [the built-in] reset logic ... I would expect that in the limit these should basically perform the same.

I ran it with pagemap_scan(Enabled::No) and keep_resident covering the whole guest, at 4 MiB and 8 threads, and swept how many pages each call dirties. Total calls/s:

DIRTY (of 1024 pages)	built-in restore (pagemap off)	compare then write reset
2	49.0K	133.0K
16	45.6K	131.0K
512	48.0K	68.9K
1024	46.1K	42.2K

Your _in the limit_ holds. When every page is dirty they cross and the built-in edges ahead. The built-in column stays flat because with pagemap off it has no dirty information, so it rewrites the whole extent every recycle. At low dirty ratios that leaves compare then write 2.7x to 2.9x ahead. One thing that works against me here: this guest has no data segments, so the built-in path is a plain zero fill with no image copy, which favors the built-in side.

Sweeping guest size instead is what tied it together for me (DIRTY=2, pagemap on, 8 threads):

guest size	fresh pooling tuned	reset reuse	reset vs tuned
256 KiB	888.9K	1.78M	2.0x
512 KiB	896.3K	963.4K	1.1x
1 MiB	857.2K	507.5K	0.6x
4 MiB	797.2K	133.1K	0.17x
16 MiB	736.3K	32.9K	0.04x

Tuned pooling only drops about 17% across a 64x size range. Pagemap only reports pages written since the slot was decommitted, and in this bench that's always the same two pages, so the restore stays small and only the scan itself grows. The reset compares all of memory, so it halves every time the guest doubles. You can see it in the flame graphs too. From after_pooling_tuned.svg to after_pooling_tuned_4mib.svg the ioctl grows from 12% to 22% of samples as the scan walks a 16x larger range, while the instantiate frames around it barely move. The reset's memcmp block grows from 74% in after_reset_reuse.svg to 89% in after_reset_reuse_4mib.svg.

I also need to correct the 96 thread numbers from my last comment. On this two socket box, tuned pooling at 32 threads and up doesn't settle on one number. At 96 threads it landed anywhere from 1.86M to 2.65M, and memory placement alone moves it. The same binary under numactl --interleave=all gives the low end every time, and under numactl --membind=0 the high end. The reset stays at 10.7M to 10.9M every run.

For guests under about 0.5 MiB, a supported way to reuse an instance and bring it back to its post-instantiate state is what would help. The two accessors in the patch are the smallest version of that I can build and maintain from the embedder side.

Wasmtime GitHub notifications bot (Sep 25 2026 at 14:58):

alexcrichton commented on issue #14313:

Thanks for the data, and I wasn't super specific in my above comment but by:

It should in theory be possible to tweak the built-in logic to basically be the same as your logic so I would expect that in the limit these should basically perform the same.

I meant that it might be interesting to implement the compare-then-write logic that you've implemented within Wasmtime itself in this function. With pagemap_scan disabled and keep_resident set high enough it should be possible to basically test your handwritten path against reinstantiation using the same reset logic. My hunch is that there won't be much performance difference left after that.

To clarify as well, adding new or orthogonal ways to access state or reset is a big hazard within Wasmtime. Anything that's gotten wrong is a CVE, for example. In the spirit of that I'd like to push as hard as possible on not adding new paths or features to Wasmtime and instead exercise what's already integrated/tested first.

Wasmtime GitHub notifications bot (Sep 29 2026 at 19:35):

stevendore closed issue #14313:

Feature

We embed wasmtime in a high-throughput HTTP proxy (the embedding from #14312) and pool WASI 0.2 component instances across requests, resetting each instance to pristine state between checkouts. Today we do this from the embedder side via two small introspection accessors we carry as a local patch. That patch is ~90 additive lines with tests. We are working through our company's open source approval process to be able to offer it upstream, and will link the PR here once we can. Either way it is the minimal primitive, not the complete solution: this issue proposes the runtime-owned reset that the accessors cannot deliver.

The feature: a way to reset a live instance back to its just-instantiated state, for example component::Instance::reset(&self, store) -> Result<()>, and possibly a core Instance::reset as well. Reset rewinds linear memories, tables, mutable globals, and internal flags (data/elem drops, lazy funcref init) so the instance is indistinguishable from a fresh instantiation of the same InstancePre in the same store, without re-running start or component initializers.

Benefit

It makes instance reuse safe for untrusted guests, which is currently a gap. wasmtime serve reuses instances by trusting the guest to keep itself request-ready (#9542), default-on for WASIp3 but deliberately default-off for WASIp2 (max_instance_reuse_count defaults to 1). Embedders whose p2 guests are untrusted or operator-supplied cannot opt into trust-based reuse at all, and per-request instantiation is a host-wide kernel ceiling, not a per-core cost.

The three strategies below all give every request a pristine instance. They differ in who restores the state and whether the kernel is involved:

• fresh instantiate (on-demand): new Store + instantiate per request on the default allocator. Each instance mmaps its linear memory and unmaps it on drop, so every request pays kernel address-space work that serializes on the process mmap_lock.
• fresh instantiate (pooling allocator): same per-request instantiate, but instances come from the pooling allocator's pre-reserved slab (no per-instance mmap). Recycling a slot decommits its dirty pages back to the CoW image with madvise, which costs TLB-shootdown IPIs across the process's CPUs.
• reset-based reuse (embedder-side): instantiate once, then per request: call, then reset in userspace. The reset memcmps each 4 KiB page against a post-instantiate snapshot, copies back only the pages that differ, and rewinds mutable globals via the accessors. No syscalls at all.

Measured on a dual-socket Xeon Gold 6330 (2x28 cores, 112 threads with SMT), dispatch-bound component call, aggregate calls/s:

threads	fresh instantiate (on-demand)	fresh instantiate (pooling allocator)	reset-based reuse (embedder-side)
1	26.0K	36.7K	143.9K
8	26.3K	92.8K	971.3K
96	21.3K	191.1K	4.32M

On-demand instantiation is flat from 1 to 96 threads, with per-call latency inflating from 0.026 ms to 2.88 ms as threads queue on the kernel. The pooling allocator is better but stalls near 191K/s: only 2x more throughput for 12x more threads past 8. The userspace reset scales with cores: 22.6x the best instantiate-per-request strategy at 96 threads. Per-call isolation overhead at 1 thread: 25.2 us (on-demand) and 17.1 us (pooling) vs 5.8 us (reset).

A runtime-owned reset would beat our embedder-side numbers further and close a correctness gap at the same time:

• The runtime owns the memory images (MemoryImageSlot) and the PAGEMAP_SCAN machinery, so discovery can be O(dirty pages) with no embedder snapshot. Our embedder compare pass is O(total memory): about 2.5 us at 256 KiB but 230 us at 16 MiB, where PAGEMAP_SCAN stays at 2 to 8 us for small dirty counts.
• The runtime can reach state no public API can: table mutations (table.set, elem.drop effects), data/element drop flags, additional linear memories, and component-layer runtime state. Our embedder reset compensates with a load-time self-check and by constraining accepted modules. A first-class reset deletes that class of edge entirely.
• It could make p2 reuse in wasmtime serve safe by default rather than trust-gated.

Implementation

Semantics: rewind to the post-instantiate point, as above. The runtime already holds everything needed: the memory image as the pristine reference, initial global values and element segments from the module, and the internal flags.

Two mechanism notes from our production experience that a design should account for:

• Restoring page contents is not restoring page state. A memcpy restore re-writes the page, so it stays an anonymous WRITTEN copy and every subsequent reset re-copies it. Per-reset cost ratchets toward O(cumulative-touched) for both PAGEMAP_SCAN and compare discovery (at 4 MiB: 2.0 us/reset at 8 touched pages, 161.7 us at all 1024). Only decommit (madvise back to the image) re-arms clean kernel state.
• Decommit has a concurrency cost microbenchmarks miss: a madvise-based reset collapsed for us at ~256 concurrent connections on the same 112-thread host (TLB-shootdown IPI storms). A reset API should be able to choose per page between memcpy and decommit, and expose a shootdown-free mode. Periodic full recycle stays structurally necessary regardless (as wasmtime serve already ships via max_instance_reuse_count).

There are also aspects of the runtime we are not familiar enough with to know how they would interact with a reset: fuel and epochs, async and concurrent state, resource handles, resetting while other instances share the store, and whether the pooling allocator could service a reset without unsharing images. We defer to maintainers on all of those. Happy to share the full benchmark methodology and an embedder's-eye API review now, and to contribute the accessors patch and implementation work under maintainer direction once our approval process completes.

Alternatives

• What we run today: embedder-side reset via two introspection accessors (Instance::core_instances, Instance::defined_globals, ~90 additive lines, PR pending our company's open source approval). Snapshot post-instantiate state, then compare-and-copy linear memory and rewind mutable globals on checkout. Proven in production and the source of the numbers above, but this approach has a hard ceiling no accessor extension can lift: discovery is O(total memory) per reset (kernel dirty-page tracking does not help an embedder, whose own memcpy restores keep pages marked written forever, per the ratchet note above), and tables, drop flags, extra memories, and component-layer state stay out of reach. The O(dirty) column and the completeness both require the runtime-owned reset. The accessors are still worth landing on their own: they are the minimal primitive that works today and remain useful plumbing under any future reset.
• Memory-only Memory::reset_to_image(&self, store): a much smaller design surface that exposes the existing PAGEMAP_SCAN plus image-slot machinery. Delivers the O(dirty) column, and globals are already coverable from the embedder side with the accessors, leaving tables and drop flags as the residual gap.
• Fresh instantiate per request: the safe default today, but the table above shows it is a host-wide ceiling (21K to 191K calls/s regardless of core count) that binds at proxy request rates.
• Trust-based reuse (#9542 temporal isolation): works for trusted p3 guests, unusable when guests are untrusted or operator-supplied, which is exactly the p2 population.
  

Wasmtime GitHub notifications bot (Sep 29 2026 at 19:35):

stevendore commented on issue #14313:

Thanks, that makes sense. I tried it: I put the compare then write logic into manually_reset_region and the no image fill, ran with pagemap_scan off and keep_resident covering the guest, and raced reinstantiation with that restore against reuse with my reset. Same bench as the gist plus that one cow.rs change, DIRTY=2, total calls/s (the 32 and 96 thread rows ran under numactl --membind=0):

guest size	threads	reinstantiate + compare then write	reuse + reset	ratio
256 KiB	8	719K	1.77M	2.5x
256 KiB	32	1.43M to 1.82M	6.52M	3.6x to 4.6x
256 KiB	96	1.96M	10.77M	5.5x
512 KiB	8	555K	962K	1.7x
1 MiB	8	376K	505K	1.3x
4 MiB	8	124K	130K	1.05x
4 MiB	32	455K	480K	1.05x
4 MiB	96	764K	792K	1.04x
16 MiB	8	33K	33K	1.0x

You were right for larger guests. From 4 MiB up there's basically nothing left. For small guests there's still about 6 us per call, and it doesn't change with guest size. The restore itself costs the same on both sides now (about 10K cycles of memcmp in both profiles), so what's left is creating and dropping the instance. It holds with every page dirty (2.0x at 256 KiB) and with a data segment (2.0x), and gets wider with threads (5.5x at 96).The cow.rs compare then write change speeds up the built-in path on its own when pagemap is off: 2.5x at 4 MiB and 4.7x at 16 MiB, no change at 256 KiB, and about 14% slower when every page is dirty. I understand the concern about adding new ways to reach or reset instance state, so I'll close out the request and keep carrying the patch on my side. Thanks for taking the time on this.

Last updated: Oct 11 2026 at 04:10 UTC