Stream: git-wasmtime

Topic: wasmtime / issue #14251 wasi: inputs a pre-init snapshot ...

view this post on Zulip Wasmtime GitHub notifications bot (Aug 31 2026 at 23:32):

composia-dave opened issue #14251:

Using wasmtime-wasi 46 as the host for a Wizer-style pre-initialization (componentize-py pre-initializes CPython and snapshots its heap), we wanted two runs to produce byte-identical snapshots. After pinning wasi:random (both randoms and the seed), both clocks and the env with WasiCtxBuilder, three inputs were still reaching the guest from the host:

  1. FileInputStream::read scheduling — the Idle arm spawns a background read and answers an empty chunk regardless of allow_blocking_current_thread, so the guest sees a load-dependent number of empty reads (measured: 42 B → 14.8 MB of snapshot divergence with host load). A PR that honors the flag in read is opened alongside this issue.
  2. Filesystem timestamps — data-access, data-modification and status-change timestamps come from the host files; a tool that writes its inputs fresh per run leaks the current time through them, and status-change (ctime) cannot be set from userspace at all, so the only complete answer on the host side is to clamp what descriptor-stat returns.
  3. The metadata hash — MetadataHashValue::from hashes the host (dev, ino); wasi-libc turns it into st_ino/st_dev, so every stat of a freshly written file carries a per-run value. In our snapshot it survived only as stale heap (the uninitialized padding bits of Python's string objects, previously a stat buffer) — 10 bytes per pair, and the last ones.

We carry 2 and 3 as a small local patch (constants), which is fine for a pre-init host that runs one guest once, but is clearly policy rather than a fix, so this is an issue, not a PR: would a WasiCtxBuilder option that makes filesystem metadata deterministic (constant timestamps and a path-independent metadata hash) be welcome, or is the recommended posture that such hosts wrap descriptor-stat themselves? Happy to write it either way.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 31 2026 at 23:33):

composia-dave commented on issue #14251:

The standalone fix for item 1 (FileInputStream::read under allow_blocking_current_thread) is #14252.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 01 2026 at 00:35):

composia-dave commented on issue #14251:

A fourth input, found when reproducing the same snapshot on a second machine (x86-64 Linux vs arm64 macOS): read-directory order. wasmtime-wasi yields entries in whatever order the host filesystem returns them (APFS and ext4 each iterate in their own hash order), and a guest that lists directories early — CPython's importlib caches every sys.path entry's listing at the first imports — allocates in that order, so the whole heap layout of the snapshot followed the host filesystem (12.6 MB of an 18 MB component differed between the two machines; the first differing words were heap pointers). Our local patch sorts the collected entries by name in read_directory before building the ReaddirIterator; with it the two machines are converging (and the macOS digest moved too, so APFS's native order was always an input). Mentioning it here because a deterministic-filesystem option would want name-ordered listings alongside the constant timestamps and metadata hash.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 01 2026 at 00:46):

composia-dave commented on issue #14251:

Closing the loop on the cross-machine reproduction: with the listing order sorted, a pre-init probe (a guest that records at module import everything it can observe — path, env, flags, loaded modules, every path entry's listing and stat, tty state — and answers it at call time) showed the two machines' guests differing in exactly one thing: the stat of directories. descriptor-stat's link-count and size for a directory are filesystem-implementation artifacts (APFS: entries+2 and 32 B per entry; ext4: 2+subdirs and 4096; btrfs: 1 and 0), and one of them (96 vs 4096) crossed CPython's small-int cache boundary, which moved every later allocation in the snapshot. Our local patch answers directories as btrfs does (1 and 0) and files with link-count 1 and their real size. So the full list of host inputs a pinned pre-init still observed through wasmtime-wasi was: FileInputStream::read scheduling (#14252), timestamps, the metadata hash, read-directory order, and directory link-count/size — all filesystem metadata, which is what a "deterministic filesystem view" option would cover.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 01 2026 at 13:43):

alexcrichton added the wasi:impl label to Issue #14251.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 01 2026 at 13:44):

alexcrichton commented on issue #14251:

Currently yeah there's no mode for wasmtime-wasi's implementation of wasi:filesystem to operate in a deterministic manner or such. It might be reasonable to add, yeah, but I'm not entirely sure how to tame all the sources of nondeterminism either.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 01 2026 at 15:10):

composia-dave deleted a comment on issue #14251:

The standalone fix for item 1 (FileInputStream::read under allow_blocking_current_thread) is #14252.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 01 2026 at 15:10):

composia-dave deleted a comment on issue #14251:

A fourth input, found when reproducing the same snapshot on a second machine (x86-64 Linux vs arm64 macOS): read-directory order. wasmtime-wasi yields entries in whatever order the host filesystem returns them (APFS and ext4 each iterate in their own hash order), and a guest that lists directories early — CPython's importlib caches every sys.path entry's listing at the first imports — allocates in that order, so the whole heap layout of the snapshot followed the host filesystem (12.6 MB of an 18 MB component differed between the two machines; the first differing words were heap pointers). Our local patch sorts the collected entries by name in read_directory before building the ReaddirIterator; with it the two machines are converging (and the macOS digest moved too, so APFS's native order was always an input). Mentioning it here because a deterministic-filesystem option would want name-ordered listings alongside the constant timestamps and metadata hash.

view this post on Zulip Wasmtime GitHub notifications bot (Sep 01 2026 at 15:10):

composia-dave deleted a comment on issue #14251:

Closing the loop on the cross-machine reproduction: with the listing order sorted, a pre-init probe (a guest that records at module import everything it can observe — path, env, flags, loaded modules, every path entry's listing and stat, tty state — and answers it at call time) showed the two machines' guests differing in exactly one thing: the stat of directories. descriptor-stat's link-count and size for a directory are filesystem-implementation artifacts (APFS: entries+2 and 32 B per entry; ext4: 2+subdirs and 4096; btrfs: 1 and 0), and one of them (96 vs 4096) crossed CPython's small-int cache boundary, which moved every later allocation in the snapshot. Our local patch answers directories as btrfs does (1 and 0) and files with link-count 1 and their real size. So the full list of host inputs a pinned pre-init still observed through wasmtime-wasi was: FileInputStream::read scheduling (#14252), timestamps, the metadata hash, read-directory order, and directory link-count/size — all filesystem metadata, which is what a "deterministic filesystem view" option would cover.

Last updated: Sep 20 2026 at 18:08 UTC