expnkx labeled Issue #2864:
Thanks for filing a bug report! Please fill out the TODOs below.
Test Case
#include<fast_io.h>
int main()
{
print(fast_io::out(),"Hello World\n",fast_io::posix_clock_gettime(fast_io::posix_clock_id::realtime));
}Expected Results
It should output all results.
Actual Results
It only outputs the first one
Versions and Environment
Wasmtime version or commit: main
Operating system: windows
Architecture: x86_64
Extra Info
-stdlib=libstdc++
expnkx opened Issue #2864:
Thanks for filing a bug report! Please fill out the TODOs below.
Test Case
#include<fast_io.h>
int main()
{
print(fast_io::out(),"Hello World\n",fast_io::posix_clock_gettime(fast_io::posix_clock_id::realtime));
}Expected Results
It should output all results.
Actual Results
It only outputs the first one
Versions and Environment
Wasmtime version or commit: main
Operating system: windows
Architecture: x86_64
Extra Info
-stdlib=libstdc++
expnkx edited Issue #2864:
Thanks for filing a bug report! Please fill out the TODOs below.
Test Case
#include<fast_io.h>
int main()
{
print(fast_io::out(),"Hello World\n",fast_io::posix_clock_gettime(fast_io::posix_clock_id::realtime));
}Expected Results
It should output all results.
Actual Results
It only outputs the first one
Versions and Environment
Wasmtime version or commit: main
Operating system: windows
Architecture: x86_64
Extra Info
-stdlib=libstdc++
expnkx edited Issue #2864:
Thanks for filing a bug report! Please fill out the TODOs below.
Test Case
#include<fast_io.h> int main() { print(fast_io::out(),"Hello World\n",fast_io::posix_clock_gettime(fast_io::posix_clock_id::realtime)); }
Expected Results
It should output all results.
Actual Results
It only outputs the first one
Versions and Environment
Wasmtime version or commit: main
Operating system: windows
Architecture: x86_64
Extra Info
-stdlib=libstdc++
sunfishcode commented on Issue #2864:
As with POSIX
writev
,__wasi_fd_write
may write fewer bytes than requested. It should return the number of bytes read, so that you can call it again with the unwritten portion of the buffer(s).
expnkx commented on Issue #2864:
As with POSIX
writev
,__wasi_fd_write
may write fewer bytes than requested. It should return the number of bytes read, so that you can call it again with the unwritten portion of the buffer(s).The problem is that these approaches break the atomicity of syscalls. And it might in a situation it only writes zero-bytes example. Also it leads to binary bloat.
It requires a fix in wasmtime tbh.
https://github.com/expnkx/fast_io/blob/725695664091bc4e8fbb05df446aaad93642bbf5/include/fast_io_hosted/platforms/win32.h#L613
Here is my implementation on windows. Use LockFile to lock the file
expnkx edited a comment on Issue #2864:
As with POSIX
writev
,__wasi_fd_write
may write fewer bytes than requested. It should return the number of bytes read, so that you can call it again with the unwritten portion of the buffer(s).The problem is that these approaches break the atomicity of syscalls. And it might in a situation it only writes zero-bytes example, the users have no idea how to deal with it. Also, it leads to binary bloat.
It requires a fix in wasmtime tbh.
https://github.com/expnkx/fast_io/blob/725695664091bc4e8fbb05df446aaad93642bbf5/include/fast_io_hosted/platforms/win32.h#L613
Here is my implementation on windows. Use LockFile to lock the file
bjorn3 commented on Issue #2864:
What if part of the data could be written and then an error is returned? It would be incorrect to suggest that no data was written by returning an error. Returning only how many bytes werw actually written would still require the loop. The error could be EWOULDBLOCK for example which in case of async usage indicates that it will be possible in the future to write more data. In that case the application needs to know how many bytes were written and try again later. Moving thr write loop into the write/writev call also doesn't match the expectations of pretty much every unix application.
bjorn3 commented on Issue #2864:
And it might in a situation it only writes zero-bytes example, the users have no idea how to deal with it.
I don't think that is possible. For read itbis possible though and indicates EOF.
expnkx commented on Issue #2864:
And it might in a situation it only writes zero-bytes example, the users have no idea how to deal with it.
I don't think that is possible. For read itbis possible though and indicates EOF.
This is not readv or read. This is writev. It does not mean it is EOF.
expnkx edited a comment on Issue #2864:
And it might in a situation it only writes zero-bytes example, the users have no idea how to deal with it.
I don't think that is possible. For read itbis possible though and indicates EOF.
This is not readv or read. This is writev. It does not mean it is EOF.
Checking size is very bad idea imo for this since it can lead to block of the process.
bjorn3 edited a comment on Issue #2864:
And it might in a situation it only writes zero-bytes example, the users have no idea how to deal with it.
I don't think that is possible. For read it is possible though and indicates EOF.
sunfishcode commented on Issue #2864:
The code bloat is a consequence of POSIX here; no implementation on a POSIX host can guarantee that it'll write all of all the buffers atomically without being interrupted. There has to be a loop somewhere, and if we follow POSIX, we can't put the loop in the WASI implementation because that would make it non-atomic.
WASI is working on some new I/O facilities which don't follow POSIX as closely, and when those are available we may have other options here. But right now, we have
__wasi_fd_write
and it does follow POSIX, and POSIX requires you to have a loop if you want all the buffers printed.
expnkx commented on Issue #2864:
The code bloat is a consequence of POSIX here; no implementation on a POSIX host can guarantee that it'll write all of all the buffers atomically without being interrupted. There has to be a loop somewhere, and if we follow POSIX, we can't put the loop in the WASI implementation because that would make it non-atomic.
WASI is working on some new I/O facilities which don't follow POSIX as closely, and when those are available we may have other options here. But right now, we have
__wasi_fd_write
and it does follow POSIX, and POSIX requires you to have a loop if you want all the buffers printed.this has nothing to do with the code works on Linux but not on windows. Windows can work as long as you lock the file.
It is absolutely the project's fault, not POSIX. POSIX never said you can provide a buggy implementation.
expnkx edited a comment on Issue #2864:
The code bloat is a consequence of POSIX here; no implementation on a POSIX host can guarantee that it'll write all of all the buffers atomically without being interrupted. There has to be a loop somewhere, and if we follow POSIX, we can't put the loop in the WASI implementation because that would make it non-atomic.
WASI is working on some new I/O facilities which don't follow POSIX as closely, and when those are available we may have other options here. But right now, we have
__wasi_fd_write
and it does follow POSIX, and POSIX requires you to have a loop if you want all the buffers printed.this has nothing to do with the code works on Linux but not on windows. Windows can work as long as you lock the file. I write the same code too and I never have any issues on windows.
It is absolutely the project's fault, not POSIX. POSIX never said you can provide a buggy implementation.
expnkx edited a comment on Issue #2864:
The code bloat is a consequence of POSIX here; no implementation on a POSIX host can guarantee that it'll write all of all the buffers atomically without being interrupted. There has to be a loop somewhere, and if we follow POSIX, we can't put the loop in the WASI implementation because that would make it non-atomic.
WASI is working on some new I/O facilities which don't follow POSIX as closely, and when those are available we may have other options here. But right now, we have
__wasi_fd_write
and it does follow POSIX, and POSIX requires you to have a loop if you want all the buffers printed.this has nothing to do with the code works on Linux but not on windows. Windows can work as long as you lock the file. I write the same code too and I never have any issues on windows.
It is absolutely the project's fault, not POSIX. POSIX never said you can provide a buggy implementation.Atomicity of syscalls cannot be guaranteed with writev either. an example is a remote console or pipe on Linux. writev is not atomic either. However, file operations beside NFS are always atomic.
expnkx edited a comment on Issue #2864:
The code bloat is a consequence of POSIX here; no implementation on a POSIX host can guarantee that it'll write all of all the buffers atomically without being interrupted. There has to be a loop somewhere, and if we follow POSIX, we can't put the loop in the WASI implementation because that would make it non-atomic.
WASI is working on some new I/O facilities which don't follow POSIX as closely, and when those are available we may have other options here. But right now, we have
__wasi_fd_write
and it does follow POSIX, and POSIX requires you to have a loop if you want all the buffers printed.this has nothing to do with the code works on Linux but not on windows. Windows can work as long as you lock the file. I write the same code too and I never have any issues on windows.
It is absolutely the project's fault, not POSIX. POSIX never said you can provide a buggy implementation.Atomicity of syscalls cannot be guaranteed with writev either. an example is a remote console or pipe on Linux. writev is not atomic either. However, file operations besides NFS are always atomic.
The code works just fine on windows and it ensures atomicity for file operations.
https://github.com/expnkx/fast_io/blob/725695664091bc4e8fbb05df446aaad93642bbf5/include/fast_io_hosted/platforms/win32.h#L613
bjorn3 commented on Issue #2864:
The implementation is not buggy on windows. It can behave in exactly the same way on linux too. You are just lucky that your the amount of text you are printing is smaller than the pipe buffer size (64k on linux) and that your terminal immediately empties the pipe as soon as data gets received. If you were to write more than 64k at once, you are guaranteed to only get a partial write and when your terminal is too slow, you may get a partial write. This is just the semantics of the write function as defined by POSIX.
iximeow commented on Issue #2864:
And it might in a situation it only writes zero-bytes example
i also don't think this is possible - wasi-libc, wasmtime, and cap-std just pass the iovecs to Rust's stdlib. the
default_write_vectored
impl there selects the first _nonempty_ iovec for writing. ifwrite_vectored
returns zero it's because the system returned zero; either the iovec selected for writing was zero-length (and all iovs were zero-length), or a non-empty vector was selected for writing and the OS itself wrote zero bytes.
iximeow edited a comment on Issue #2864:
And it might in a situation it only writes zero-bytes example
i also don't think this is possible - wasi-libc, wasmtime, and cap-std currently just pass the iovecs to Rust's stdlib. the
default_write_vectored
impl there selects the first _nonempty_ iovec for writing. ifwrite_vectored
returns zero it's because the system returned zero; either the iovec selected for writing was zero-length (and all iovs were zero-length), or a non-empty vector was selected for writing and the OS itself wrote zero bytes.
expnkx commented on Issue #2864:
And it might in a situation it only writes zero-bytes example
i also don't think this is possible - wasi-libc, wasmtime, and cap-std currently just pass the iovecs to Rust's stdlib. the
default_write_vectored
impl there selects the first _nonempty_ iovec for writing. ifwrite_vectored
returns zero it's because the system returned zero; either the iovec selected for writing was zero-length (and all iovs were zero-length), or a non-empty vector was selected for writing and the OS itself wrote zero bytes.wraps around rust's stdlib are not something that is supposed to work with wasm imo. WASI is to abstract OS interface, not to abstract language interface.
expnkx edited a comment on Issue #2864:
And it might in a situation it only writes zero-bytes example
i also don't think this is possible - wasi-libc, wasmtime, and cap-std currently just pass the iovecs to Rust's stdlib. the
default_write_vectored
impl there selects the first _nonempty_ iovec for writing. ifwrite_vectored
returns zero it's because the system returned zero; either the iovec selected for writing was zero-length (and all iovs were zero-length), or a non-empty vector was selected for writing and the OS itself wrote zero bytes.wraps around rust's stdlib are not something that is supposed to work with wasm imo. WASI is to abstract OS interface, not to abstract language interface.
BTW I do see IO performance bottleneck with using Rust stdlib.
sunfishcode commented on Issue #2864:
POSIX says "If write() is interrupted by a signal after it successfully writes some data, it shall return the number of bytes written". So even on Linux, where things may appear to work, they're not guaranteed to work in all cases. The system call could still be interrupted in some circumstances, and could perform an incomplete write in some circumstances. Consequently, Wasm code needs a loop if it wants all the buffers printed.
In theory, a
__wasi_fd_write
call could return zero. But this shouldn't be any different from it returning any other number less than the requested number. If the wasm code has a loop, the zero case isn't special.WASI today is a synchronous API. It blocks. Many people are obviously interested in async I/O; various approaches are in development, but not available yet. So for now, it's a blocking API, and adding a loop around it doesn't introduce any new blocking.
expnkx commented on Issue #2864:
POSIX says "If write() is interrupted by a signal after it successfully writes some data, it shall return the number of bytes written". So even on Linux, where things may appear to work, they're not guaranteed to work in all cases. The system call could still be interrupted in some circumstances, and could perform an incomplete write in some circumstances. Consequently, Wasm code needs a loop if it wants all the buffers printed.
In theory, a
__wasi_fd_write
call could return zero. But this shouldn't be any different from it returning any other number less than the requested number. If the wasm code has a loop, the zero case isn't special.WASI today is a synchronous API. It blocks. Many people are obviously interested in async I/O; various approaches are in development, but not available yet. So for now, it's a blocking API, and adding a loop around it doesn't introduce any new blocking.
if the API keeps returning 0, you get a block.
sunfishcode commented on Issue #2864:
if the API keeps returning 0, you get a block.
That is correct. The API itself can also block.
sunfishcode commented on Issue #2864:
The implementation is working as intended. To write a complete string reliably with the current WASI APIs, it's necessary to wrap the
__wasi_fd_write
call in a loop, as libc does.
sunfishcode closed Issue #2864:
Thanks for filing a bug report! Please fill out the TODOs below.
Test Case
#include<fast_io.h> int main() { print(fast_io::out(),"Hello World\n",fast_io::posix_clock_gettime(fast_io::posix_clock_id::realtime)); }
Expected Results
It should output all results.
Actual Results
It only outputs the first one
Versions and Environment
Wasmtime version or commit: main
Operating system: windows
Architecture: x86_64
Extra Info
-stdlib=libstdc++
Last updated: Dec 23 2024 at 12:05 UTC