wasmtime / PR #14098 fuzz: round pooling GC heap limit do... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / PR #14098 fuzz: round pooling GC heap limit do...

Wasmtime GitHub notifications bot (Aug 10 2026 at 13:08):

4ktLuffy opened PR #14098 from 4ktLuffy:fix-pooling-gc-heap-clamp to bytecodealliance:main:

#13855 clamped gc_heap_initial_size to the pooling allocator's max_memory_size, fixing an oversight from #13841. But it rounds both sides up to the 64K GC page size:
if let Some(amt) = mcfg.gc_heap_initial_size {
    let page_size = 64 * 1024;
    let amt = amt.next_multiple_of(page_size);
    let max = (pcfg.max_memory_size as u64).next_multiple_of(page_size);
    mcfg.gc_heap_initial_size = Some(amt.min(max));
}
Rounding amt up is correct — it matches what Wasmtime does internally. Rounding the limit up is not: max_memory_size is a hard ceiling, so whenever it isn't page-aligned the clamp target itself lands above the limit, and the clamp can't do its job.

With max_memory_size = 933888 (0xe4000, 14.25 pages):

computation result vs limit

old 933888.next_multiple_of(65536) 983040 over

new 933888 / 65536 * 65536 917504 ok

Engine::new then fails in Config::to_store with:
instance allocator cannot support configured GC heap memory

Caused by:
    memory has a minimum byte size of 983040 which exceeds the limit of 0xe4000 bytes
Testing

On 687cbc171, using cargo +nightly fuzz run --no-default-features instantiate:

before: panicked within 90 seconds

after: 60182 executions over 7 minutes complete cleanly, and the recorded reproducer passes 5/5

Checked the clamp across the domain, since the concern with rounding down is whether it regresses already-aligned values. The old form overshoots for every non-page-aligned max_memory_size; the new form never does, and both agree wherever the input is already aligned:
max_memory_size=0         old=0        ok    new=0        ok
max_memory_size=65536     old=65536    ok    new=65536    ok
max_memory_size=100000    old=131072   OVER  new=65536    ok
max_memory_size=933888    old=983040   OVER  new=917504   ok
max_memory_size=983040    old=983040   ok    new=983040   ok
max_memory_size=1048576   old=1048576  ok    new=1048576  ok
cargo fmt --check -p wasmtime-fuzzing passes.

Found while fuzzing locally; no prior issue filed

	computation	result	vs limit
old	`933888.next_multiple_of(65536)`	983040	over
new	`933888 / 65536 * 65536`	917504	ok

Wasmtime GitHub notifications bot (Aug 10 2026 at 13:08):

4ktLuffy requested alexcrichton for a review on PR #14098.

Wasmtime GitHub notifications bot (Aug 10 2026 at 13:08):

4ktLuffy requested wasmtime-fuzz-reviewers for a review on PR #14098.

Wasmtime GitHub notifications bot (Aug 10 2026 at 13:32):

4ktLuffy updated PR #14098.

Wasmtime GitHub notifications bot (Aug 10 2026 at 13:33):

4ktLuffy updated PR #14098.

Wasmtime GitHub notifications bot (Aug 10 2026 at 13:57):

4ktLuffy edited PR #14098:

#13855 clamped gc_heap_initial_size to the pooling allocator's max_memory_size, fixing an oversight from #13841. But it rounds both sides up to the 64K GC page size:
if let Some(amt) = mcfg.gc_heap_initial_size {
    let page_size = 64 * 1024;
    let amt = amt.next_multiple_of(page_size);
    let max = (pcfg.max_memory_size as u64).next_multiple_of(page_size);
    mcfg.gc_heap_initial_size = Some(amt.min(max));
}
Rounding amt up is correct — it matches what Wasmtime does internally. Rounding the limit up is not.

The pool rejects a memory whose minimum size exceeds max_memory_size rounded up to a host page — MemoryPool::new via HostAlignedByteCount::new_rounded_up, checked in MemoryPool::validate_memory. A host page is never larger than the 64K GC page, so rounding the clamp target up to 64K can place it above that limit whenever max_memory_size is not 64K-aligned, and the clamp then cannot do its job.

Observed with a generated max_memory_size of 921102 on a 16K-page host:
limit = round_up_host(921102)          = 933888 (0xe4000)
old   = 921102.next_multiple_of(65536) = 983040   > limit  -> panic
new   = 921102 / 65536 * 65536         = 917504  <= limit  -> ok
Engine::new then fails in Config::to_store with:
instance allocator cannot support configured GC heap memory

Caused by:
    memory has a minimum byte size of 983040 which exceeds the limit of 0xe4000 bytes
Why rounding down is correct generally

The new value is always <= max_memory_size <= limit, for any host page size. Swept against the real limit rather than against max_memory_size:
max_memory_size   host    limit       old              new
         100000   4096   102400   131072  OVER      65536  ok
         921102   4096   921600   983040  OVER     917504  ok
         983040   4096   983040   983040    ok     983040  ok

         100000  16384   114688   131072  OVER      65536  ok
         921102  16384   933888   983040  OVER     917504  ok
         983040  16384   983040   983040    ok     983040  ok

         100000  65536   131072   131072    ok      65536  ok
         921102  65536   983040   983040    ok     917504  ok
         983040  65536   983040   983040    ok     983040  ok
Two things worth noting from that sweep:

The old form overshoots on 4K and 16K hosts but not on a 64K-page host, where round_up_host and round_up_64k coincide. So this does not reproduce everywhere — which may be why it has survived.

The new form is never over the limit on any host page size, and both forms agree wherever max_memory_size is already 64K-aligned.

Testing

On 687cbc171, cargo +nightly fuzz run --no-default-features instantiate:

before: panicked within 90 seconds

after: 60182 executions over 7 minutes complete cleanly; the recorded reproducer passes 5/5

The max_memory_size = 921102 figure above was read from an instrumented build replaying that reproducer, not inferred from the panic message.

cargo fmt --check -p wasmtime-fuzzing and cargo clippy -p wasmtime-fuzzing --no-default-features are clean.

Found while fuzzing locally; no prior issue filed.

Wasmtime GitHub notifications bot (Aug 10 2026 at 14:09):

:thumbs_up: alexcrichton submitted PR review:

Thanks! Could this update the comment above and cut down a bit on the wordiness? It should be ok to basically just say that this is rounding down so the GC's round-up behavior isn't hit.

Wasmtime GitHub notifications bot (Aug 10 2026 at 14:21):

4ktLuffy updated PR #14098.

Wasmtime GitHub notifications bot (Aug 10 2026 at 14:23):

alexcrichton has enabled auto merge for PR #14098.

Wasmtime GitHub notifications bot (Aug 10 2026 at 14:24):

4ktLuffy commented on PR #14098:

Done, thanks!

Wasmtime GitHub notifications bot (Aug 10 2026 at 14:36):

alexcrichton added PR #14098 fuzz: round pooling GC heap limit down, not up to the merge queue.

Wasmtime GitHub notifications bot (Aug 10 2026 at 15:11):

:check: alexcrichton merged PR #14098.

Wasmtime GitHub notifications bot (Aug 10 2026 at 15:11):

alexcrichton removed PR #14098 fuzz: round pooling GC heap limit down, not up from the merge queue.

Last updated: Aug 30 2026 at 09:07 UTC