Victor Adossi (Aug 03 2026 at 16:00): Victor Adossi (Aug 03 2026 at 16:00):tl;dr - write-via-stream or some new API should probably return a stream of disk write acks, because stream write != disk write.
Recently we ran into a racy test in the wasi-testsuite with regards to use of write-via-stream.
To try to explain the race concisely:
append-via-stream is called with a stream s1s1stat is called, to check if the file reflects the written bytesstat more bytes may be sent to s1 (i.e. finishing a partial write)The issue here is between steps (2) and (3) -- bytes being written to the stream does not mean those bytes hit disk. In this case, the test mistakes those two things, and because wasmtime does host magic to make that the case, the test runs fine in wasmtime. The stream-write is delayed until the disk-write completes, so by the time stat gets called, we really do have a guarantee that the bytes are on disk and thus show up when stat executes.
On the Jco side, however, we don't have this same host magic (though we can add it), and we actually end up blocking. Some context on why:
write happens in a WebWorker (somewhat "far away")stat does notThis manifests as two problem scenarios on the Jco side:
write gets called before stat on the Jco side, the stat may actually happen before the writewrite is (properly) called first, we hang because while the partial write happens and we start waiting on the stream read for more bytes, but we'll never finish (amongst other issues the stat blows up because the writes did not yet land on disk, and thus the additional bytes/write-end closure will never happen)(2) is solvable with changes on the Jco side (essentially, allowing partial writes, an explicit work queue), and we'll make some of those, but this bug makes it clear the API might be a bit lacking.
AFAICT the core of the problem is the API for write-via-stream -- there needs to be some way to signal disk writes, not just stream writes, so that users that want to acknowledge partial writes explicitly may do so.
A new, more fine-grained fn would be perfectly fine IMO.
As far as the test goes the test can/should be rewritten to be less racy, but this kind of code is easy for someone to write and is not super obviously wrong, it would be nice if we could make this behavior explicit, so people are a bit less likely to mistake stream writes for operation completion.
It may be a good general rule of thumb that any API which takes a stream to allow for asynchronous operation should likely return a stream of asynchronous/batched acks unless it is explicitly meant to coalesce -- this is clearly something that would apply to any async operation that works in chunks.
Lann Martin (Aug 03 2026 at 19:25):Isn't this what descriptor.sync is for?
Dave Bakker (badeend) (Aug 03 2026 at 19:27):I'm not familiar with the internals of JCO, but it sounds like JCO reports the stream.write as completed even while it is still being processed on the background thread. If JCO can't finish the write immediately, it should report the write as in-progrss, so that the guest can wait for it to reach the kernel.
Additionally, even when you manage to implement it that way, you can't assume that written data will be immediately reflected in the info returned by stat, unless you use fsync/O_SYNC.
Dave Bakker (badeend) (Aug 03 2026 at 20:13):there needs to be some way to signal disk writes (..) so that users that want to acknowledge partial writes explicitly may do so.
wasi-filesystem is a relatively thin abstraction layer over the POSIX filesystem API. And in POSIX, all writes are _not_ synchronized by default. Only when you use fsync/fdatasync/O_*SYNC/.. do you get a stronger guarantee about the data's consistency.
One could interpret this as: wasi-filesystem already provides the semantics you're asking for. But as you also have already experienced the hard way; there is a difference between
Given that the test passes under wasmtime but not on JCO, I'd focus on the JCO / stream bindgen side of things first.
Side note: you refer to "bytes hitting the disk". AFAIK, there's not a single mainstream OS that can actually reliably guarantee that. What fstat & friends guarantee is a consistent view of the filesystem to across processes. Whether or not that also includes the data being durably written to disk depends on a whole lot of other factors, some of them beyond the kernel's control.
Welcome to the wonderful world of filesystems :octopus:
Alex Crichton (Aug 03 2026 at 20:17):If JCO can't finish the write immediately, it should report the write as in-progrss
I talked with Victor about this in DMs, but one problem with this, while I agree this is the correct semantics, is that there's no way to actually do this in a pure guest-to-guest situation today. We (some set of folks more than just me but I forget who) discussed this in the past and supporting this in guest-to-guest scenarios will require a new intrinsic on the read side at the very least. If JCO is implementing the host as-if it were guest-to-guest while this is the synchronization desired it's not actually possible with streams today
Alex Crichton (Aug 03 2026 at 20:18):but I would agree that this isn't about disk/etc, it's about "I sent bytes to the WASI API, then I read the size, and it should account for the thing I wrote"
Dave Bakker (badeend) (Aug 03 2026 at 20:19):in guest-to-guest interactions, can 0-sized writes be used to check for readiness after a non-0-sized write?
Alex Crichton (Aug 03 2026 at 20:21):I believe so, yeah, you'd write bytes, do a 0-sized write, and then when the other side gets around to reading again you'd get woken up
Alex Crichton (Aug 03 2026 at 20:21):so that could be a wasi-libc-style workaround pehraps
Alex Crichton (Aug 03 2026 at 20:21):(or a fix for the test in wasi-testsuite that was changed)
Victor Adossi (Aug 04 2026 at 03:05):Lann Martin said:
Isn't this what
descriptor.syncis for?
So yes, this is certainly true, and fsync after every stream write would certainly write some bytes to disk, but we don't know which bytes have been accepted/written (e.g. fine grained ACKs out of read-via-stream).
In the current world you could call read-via-stream, all bytes are read off the stream, and no bytes are written at the OS level, and sync (fsync) is called. Test would fail in the same way I think.
Victor Adossi (Aug 04 2026 at 03:09):Dave Bakker (badeend) said:
I'm not familiar with the internals of JCO, but it sounds like JCO reports the
stream.writeas completed even while it is still being processed on the background thread. If JCO can't finish the write immediately, it should report the write as in-progrss, so that the guest can wait for it to reach the kernel.
Additionally, even when you manage to implement it that way, you can't assume that written data will be immediately reflected in the info returned bystat, unless you usefsync/O_SYNC.
Yep, that's it! The stream.write and the fs.write are distinct, and IMO should be separate, if the thing the consumer cares about is the operation happening on the bytes, then we should have a specific channel for that signal.
Agree on the sync (fsync/fdatasync) -- this was Lann's suggestion, and wanted to point out that we have the same problem there -- you could call fsync but you don't know what has been purported to be written to the OS layer.
Victor Adossi (Aug 04 2026 at 03:14):Dave Bakker (badeend) said:
there needs to be some way to signal disk writes (..) so that users that want to acknowledge partial writes explicitly may do so.
wasi-filesystem is a relatively thin abstraction layer over the POSIX filesystem API. And in POSIX, all writes are _not_ synchronized by default. Only when you use
fsync/fdatasync/O_*SYNC/.. do you get a stronger guarantee about the data's consistency.
One could interpret this as: wasi-filesystem already provides the semantics you're asking for. But as you also have already experienced the hard way; there is a difference between
- what POSIX technically allows, and
- what code out of the wild actually depends on. :slight_smile:
Given that the test passes under wasmtime but not on JCO, I'd focus on the JCO / stream bindgen side of things first.
Side note: you refer to "bytes hitting the disk". AFAIK, there's not a single mainstream OS that can actually reliably guarantee that. What
fstat& friends guarantee is a consistent view of the filesystem to across processes. Whether or not that also includes the data being durably written to disk depends on a whole lot of other factors, some of them beyond the kernel's control.Welcome to the wonderful world of filesystems :octopus:
Yes, I know this -- the point is that when you come back from a read, you know you've at least given the read to the OS (i.e. the layer that will make your fsync/fdatasync mean anything). We don't have that signal with the current API -- we only have "bytes were delivered to the operation that will maybe eventually give your writes to the system underneath".
And sorry, I was using "bytes hitting the disk" as a shorthand for the guarantees at the OS level, I should have been more precise! The only way to guarantee is an fsync, and we're not even talking about that behavior, I'm literally only talking about a write & stat.
What
fstat& friends guarantee is a consistent view of the filesystem to across processes.
One thing I did consider on the spectrum of solutions to this was trying to guarantee some sort of same-FD consistency of the read-your-writes flavor, and I thought better of that -- preferring to leave it to the OS (i.e. not guarantee anything the OS isn't).
Even queueing the work isn't necessarily the best solution, I think -- I'd like to have whichever operation complete as fast as possible, but the problem is that the user has no way to wait for a "write" right now, they can only wait for "stream consumption".
Victor Adossi (Aug 04 2026 at 03:27):write-via-stream does not currently provide a way to know that the underlying write operation (whether it's a real OS or virtualized) has been completed/submitted.
For most platforms some fsync-like operation (descriptor.sync in WASI) is the way to make sure the bytes hit the disk (as far as the platform can tell), but this problem is separate from that -- the user cannot tell when the bytes written to the stream actually got submitted to the platform, with the current API. You can only tell when the bytes you submitted to the stream got read.
For the current system (and this example in particular) to work, hosts essentially must do magic to make bytes-operated-on equivalent to stream-writes-delivered, and outside of being non-virtualizable, I think that's a mixing of signals -- the two should be distinct.
The current API certainly works very well for complete writes from a stream (i.e. every write on the stream lands, then write-via-stream delivers it's success in writing all the content with platform-specific meaning), but it does not work for partial writes of what was streamed.
Of course, "just don't do that" is a valid option (i.e. all bytes to the stream will be written, wait for that instead)! But I think this kind of fine grained writing & acking/probing disk presence is a usecase that should be supported.
Lann Martin (Aug 04 2026 at 13:12):With the current design I think you have to treat a write-via-stream as a single write operation, at least for this purpose, so stat on the same descriptor wouldn't necessarily be updated until the call finishes.
Lann Martin (Aug 04 2026 at 13:13):If that isn't already happening I'd consider it a bug.
Lann Martin (Aug 04 2026 at 13:14):Re-reading your original message I'm not sure if "After the stream write completes" is referring to the CM stream write or the append-via-stream call.
Victor Adossi (Aug 04 2026 at 13:15):Ah it definitely is already happening, but the test is trying to do a partial write thing, which is the problem.
write-via-stream is fine for what it does, but the test made the mistake of thinking CM stream write confirmation was file write confirmation
Victor Adossi (Aug 04 2026 at 13:15):Ah sorry, when I say "stream write" I mean CM stream write
Lann Martin (Aug 04 2026 at 13:16):Yeah I think that behavior is reasonable in the scope of wasi:filesystem trying to be posix-ish.
Victor Adossi (Aug 04 2026 at 13:16):write-via-stream is working as intended, but it's kind of easy to misuse if what you're looking for is chunked write ACKs, so yeah I am happy with a new API for this, but wanted to call it out, as it's easy to misuse (if you want fine grained acks)
Lann Martin (Aug 04 2026 at 13:17):A lot of the wasi mappings to posix-ish IO are easy to misuse, partly because there is a bit of an execution model mismatch and partly because posix is just easy to misuse. :sweat_smile:
Victor Adossi (Aug 04 2026 at 13:21):Yeah I think a new API could make this pretty explicit though, and help us get rid of non-virtualizable host magic.
Again, maybe this use case is niche, but I think we kind of suggested/welcomed it when write-via-stream was introduced. If you write via a stream, it makes sense that sometimes you'd want to know when a specific individual chunk was written (and maybe fsync right then and there, then go do some other stuff, etc).
Lann Martin (Aug 04 2026 at 13:30):You could break up a large write into multiple write-via-stream calls. That's what I would reach for if the data is already in memory at the same point that you want the write acknowledgements. It gets hairy if you're trying to do this for something middlewareish that wouldn't otherwise need to buffer the data. I think stream.forward would serve that use case if/when implemented.
Victor Adossi (Aug 04 2026 at 14:33):Yup I had that thought as well -- I thought it might be a bit inefficient having to create and pass in multiple streams and then to keep track of all of it would be pretty annoying to write out in the guest.
Last updated: Aug 30 2026 at 09:07 UTC