itowlson opened PR #9680 from itowlson:closeable-async-write-stream
to bytecodealliance:main
:
Summary
This proposal adds an optional
Receiver<()>
field toAsyncWriteStream
. When signalled, this gracefully closes the underlyingAsyncWrite
.Context
This emerged from trying to implement wasi-blobstore. I was passing the read end of a stream to a back-end provider, and the write end to the guest. I was using
wasmtime_wasi
to store the write end in a resource table as aHostOutputStream
because I didn't want to write a stream host interface myself!The API specifies that the guest calls a "finish" method (on a separate
outgoing-value
resource) to indicate that it has finished writing to the stream. However, because I was usingAsyncWriteStream
andwasmtime_wasi::HostOutputStream
, my host didn't have access to the underlying AsyncWrite'sshutdown
function to EOF the stream. So the reader at the back end would continue waiting for input.My first plan was to add a
shutdown
function toAsyncWriteStream
, but I couldn't downcastHostOutputStream
toAsyncWriteStream
, so that didn't work. So what I ended up with was:
- Have AsyncWriteStream accept the receiver end of a sync channel.
- Have AsyncWriteStream run a background task which waited on the receiver. When it received a message, it would enqueue a shutdown of the underlying
AsyncWrite
.- Have the manager of the outbound connection (in wasi-blobstore, an
outgoing-value
resource) hold the sender end of the channel.- When the guest called
outgoing-value::finish
, send a message via the sender.This seemed to work, although it's not been extensively tested.
As an interface, it's awkward, because for most operations you interact with
AsyncWriteStream/HostOutputStream
via methods, but for this one special case you interact with it via a sync channel. However, I don't see a way round that without adding a shutdown or close method to the WASIoutput-stream
resource. It's possible that WASI intends that shutdown be done by dropping theoutput-stream
resource, but in my testing this seemed to result in an abort rather than an EOF.So after talking to @alexcrichton I'm putting this up for discussion and in the hope that WASI folks can come up with a better solution. Please let me know if folks need clarifications around the problem I was trying to solve or the constraints that led me in this direction!
pchickey commented on PR #9680:
The prior art for shutdown is in the
wasi-sockets
implementation here: https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasi/src/tcp.rs#L641Rather than shutdown be a method on a stream, its a method on the parent of the stream, in this case the
tcp-socket
https://github.com/WebAssembly/wasi-sockets/blob/main/wit/tcp.wit#L361-L385.Can you restructure your blobstore interfaces to work in that manner, and then follow the architecture of wasmtime-wasi's sockets in your implementation?
pchickey edited a comment on PR #9680:
The prior art for shutdown is in the
wasi-sockets
implementation here: https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasi/src/tcp.rs#L641Rather than shutdown be a method on a stream, its a method on the parent of the stream, in this case the
tcp-socket
https://github.com/WebAssembly/wasi-sockets/blob/main/wit/tcp.wit#L361-L385.It looks like this
finish
mechanism on the parent resourceoutgoing-value
in wasi-blobstore follows the same approximate architecture. Can you follow the architecture used by wasmtime-wasi's sockets in your implementation?
pchickey edited a comment on PR #9680:
The prior art for shutdown is in the
wasi-sockets
implementation here: https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasi/src/tcp.rs#L641Rather than shutdown be a method on a stream, its a method on the parent of the stream, in this case the
tcp-socket
https://github.com/WebAssembly/wasi-sockets/blob/main/wit/tcp.wit#L361-L385.It looks like this
finish
mechanism on the parent resourceoutgoing-value
in wasi-blobstore follows an approximately similar architecture. Can you follow the architecture used by wasmtime-wasi's sockets in your implementation?
pchickey edited a comment on PR #9680:
The prior art for shutdown is in the
wasi-sockets
implementation here: https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasi/src/tcp.rs#L641Rather than shutdown be a method on a stream, its a method on the parent of the stream, in this case the
tcp-socket
https://github.com/WebAssembly/wasi-sockets/blob/main/wit/tcp.wit#L361-L385.It looks like this
finish
method consuming parent resourceoutgoing-value
in wasi-blobstore follows an approximately similar architecture. It has the added advantage that the output-stream is already guaranteed to be dropped at that point. Cn you follow the architecture used by wasmtime-wasi's sockets in your implementation?
itowlson commented on PR #9680:
Thanks for the pointers @pchickey. I'll try to implement that pattern.
Last updated: Dec 23 2024 at 12:05 UTC