Wasmtime GitHub notifications bot (Jan 23 2022 at 03:35): Wasmtime GitHub notifications bot (Jan 23 2022 at 03:35):JoeStrout opened issue #3715:
Feature
Make wasmtime hip to a shebang at the first line of the file, e.g.:
#!/usr/local/bin/wasmtimefollowed by the text-format wasm code as usual. Currently, the shell runs wasmtime and feeds it the script as expected, but wasmtime chokes on the first line and fails to run the following script.
Benefit
This would allow us to write scripts that can be executed by simply setting the executable bit and invoking them, like any other shell script. It's a matter of convenience. I have a lot of scripts in a lot of different languages; I don't have to remember what tool to use to run them because they all understand shebang — except for wasmtime.
Implementation
Just special-case it: if the first line of the script file starts with "#!", then ignore it and continue with the rest of the file.
Alternatives
I could write, for every wasm script I want to run, a little 2-line shell script that runs wasmtime with the path to the actual script. That would be annoying on so many levels.
Wasmtime GitHub notifications bot (Jan 24 2022 at 15:23):alexcrichton commented on issue #3715:
Personally I don't feel that Wasmtime should support wasmtime-specific extensions to the text format. If this wants to be supported then I think that this should be added to the official grammar of text files in the wasm specification itself.
Wasmtime GitHub notifications bot (Jan 24 2022 at 16:08):JoeStrout commented on issue #3715:
I would argue that this is wasmtime's responsibility, as a command-line executable whose function is to run .wat files on the command line. This is something all other script runners (perl, python, miniscript, etc. etc.) do, and the shebang is not part of the grammar of those languages either; it's just something the script runners are prepared to deal with.
Conversely, a shebang doesn't make any sense in other contexts, e.g. in a .wat file that's part of a Visual Studio project. So I wouldn't expect to see it in the .wat spec itself. It _only_ makes sense for people who (like me) are trying to use wasmtime like other command-line scripts, so it ought to just be something that wasmtime does.
Wasmtime GitHub notifications bot (Jan 24 2022 at 16:20):bjorn3 commented on issue #3715:
Shebang's are definitively part of the grammar of a language if an interpreter for it can handle them.
Wasmtime GitHub notifications bot (Jan 25 2022 at 02:22):cfallin commented on issue #3715:
@JoeStrout FWIW I think the main issue with support in wasmtime for this would be that it could potentially normalize "not quite compliant"
.watfiles -- in other words, it encourages the existence of non-standards-compliant source that fails to parse in other tools/engines.But since this is Unix, there are probably ways to put together other tools to make a solution... one technique that might help is to write the wat content as a heredoc, something like the following (just tested):
#!/bin/sh wasmtime run /dev/stdin <<__END__ (module (import "wasi_snapshot_preview1" "proc_exit" (func (param i32))) (memory (export "memory") 0 1) (func (export "_start") i32.const 0 call 0)) __END__One could conceivably write a tool that converts
.watto.wat.shby wrapping it like this.One could probably also use the "custom binfmt" support in Linux to recognize the magic number in a
.wasmand invokewasmtimeas an interpreter; then a WASI .wasm would behave just like a native executable. (IIRC, distributions of qemu often set this up for ELF binaries of other architectures; that might be a place to start looking.)Anyway, just some ideas -- hope these help!
Wasmtime GitHub notifications bot (Jan 25 2022 at 02:39):iximeow commented on issue #3715:
(a little bit of pedantry, sorry: bash and python and friends do _not_ have any particular support for
#!, that's handled in the kernel as part ofexec. by happy ~coincidence~ design, those languages treat#as a comment and then ignore the shebang line when executed. thus wasmtime executing a shebang'd file would first require _ignoring a preceding#!.*\n_, then executing the provided.wattext. that's where the non-standards-compliance comes in, since .wat comments are;;rather than#.)
Wasmtime GitHub notifications bot (Jan 25 2022 at 12:25):tschneidereit commented on issue #3715:
Yeah, I don't think wasmtime should start accepting non-standard extensions to the file format. One way to do this if we somehow absolutely wanted to is to get the standard changed to support shebangs. That's what's happening in JavaScript.
I'm not sure that it makes sense to do this though: the
watformat really isn't meant to be used as a production format, and I don't think we'd be well served with making it easier to runwatfiles thanwasmfiles. And the binfmt support Chris mentions already exists.
Wasmtime GitHub notifications bot (Feb 19 2022 at 13:17):JoeStrout commented on issue #3715:
I'm surprised by the negative reactions. All I'm asking for is to skip the first line of the input if it starts with
#, rather than fail. If#had some _other_ valid usage here, we would have a problem as it might be difficult to disambiguate the user's intent. But we don't. I think you can confidently say that the user does not intend the result to be a screenful of error messages.It's not a big ask, and it's a substantial quality-of-life improvement, at least for some of us.
Wasmtime GitHub notifications bot (Feb 19 2022 at 14:26):tschneidereit commented on issue #3715:
The "big" part of the ask is to support a non-standard extension to a standardized format, that'd then mean that that format only runs in our runtime, but not others. The right way to go about adding something like this is to propose it in the WebAssembly Community Group.
As I said in my previous comment I also think that this isn't the right thing to do to begin with, but a sufficient number of people in the Community Group might disagree for it to be standardized, in which case we'd implement it.
Wasmtime GitHub notifications bot (May 20 2022 at 15:32):alexcrichton commented on issue #3715:
I'm going to close this form the discussion above. I think it's still reasonable to propose this extension to the wasm CG and the text format if you're interested though.
Wasmtime GitHub notifications bot (May 20 2022 at 15:32):alexcrichton closed issue #3715:
Feature
Make wasmtime hip to a shebang at the first line of the file, e.g.:
#!/usr/local/bin/wasmtimefollowed by the text-format wasm code as usual. Currently, the shell runs wasmtime and feeds it the script as expected, but wasmtime chokes on the first line and fails to run the following script.
Benefit
This would allow us to write scripts that can be executed by simply setting the executable bit and invoking them, like any other shell script. It's a matter of convenience. I have a lot of scripts in a lot of different languages; I don't have to remember what tool to use to run them because they all understand shebang — except for wasmtime.
Implementation
Just special-case it: if the first line of the script file starts with "#!", then ignore it and continue with the rest of the file.
Alternatives
I could write, for every wasm script I want to run, a little 2-line shell script that runs wasmtime with the path to the actual script. That would be annoying on so many levels.
Wasmtime GitHub notifications bot (Jan 21 2023 at 06:35):guest271314 commented on issue #3715:
Just encountered this limitation trying to run WebAssembly as a Native Messaging host. To me it seems
wasmtime_should_ be able to run using#!/usr/bin/env -Swhich works for Python, Node.js, Deno, QuickJS, et al. though throws forwasmtime#!/usr/bin/env wasmtime (module ;; Import the required fd_write WASI function which will write the given io vectors to stdout ;; The function signature for fd_write is: ;; (File Descriptor, *iovs, iovs_len, nwritten) -> Returns number of bytes written (import "wasi_unstable" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32))) (memory 1) (export "memory" (memory 0)) ;; Write 'hello world\n' to memory at an offset of 8 bytes ;; Note the trailing newline which is required for the text to appear (data (i32.const 8) "hello world\n") (func $main (export "_start") ;; Creating a new io vector within linear memory (i32.store (i32.const 0) (i32.const 8)) ;; iov.iov_base - This is a pointer to the start of the 'hello world\n' string (i32.store (i32.const 4) (i32.const 12)) ;; iov.iov_len - The length of the 'hello world\n' string (call $fd_write (i32.const 1) ;; file_descriptor - 1 for stdout (i32.const 0) ;; *iovs - The pointer to the iov array, which is stored at memory location 0 (i32.const 1) ;; iovs_len - We're printing 1 string stored in an iov - so one. (i32.const 20) ;; nwritten - A place in memory to store the number of bytes written ) drop ;; Discard the number of bytes written from the top of the stack ) )$ ./demo Error: failed to run main module `./demo` Caused by: 0: if you're trying to run a precompiled module, pass --allow-precompiled 1: expected `(` --> ./demo:1:1 | 1 | #!/usr/bin/env wasmtime | ^
Wasmtime GitHub notifications bot (Jan 21 2023 at 06:38):guest271314 commented on issue #3715:
that it could potentially normalize "not quite compliant"
.watfiles -- in other words, it encourages the existence of non-standards-compliant source that fails to parse in other tools/engines.But since this is Unix, there are probably ways to put together other tools to make a solution... one technique that might help is to write the wat content as a heredoc, something like the following (just tested):
```
#!/bin/shwasmtime run /dev/stdin <<__END__
(module
(import "wasi_snapshot_preview1" "proc_exit" (func (param i32)))
(memory (export "memory") 0 1)
(func (export "_start")
i32.const 0
call 0))
__END__
```How will that work when the WebAssembly module is expected to be reading stdin from an application, e.g., the browser?
Wasmtime GitHub notifications bot (Jan 21 2023 at 15:08):bjorn3 commented on issue #3715:
How will that work when the WebAssembly module is expected to be reading stdin from an application, e.g., the browser?
You can write the wat file to a temporary file and then run it. Or you could store the wat file next to the shell script that runs wasmtime. You will need to do that anyway if you want to compile the wat file to a wasm file when shipping the extension. (a wat file is several times bigger than the compiled wasm file)
Wasmtime GitHub notifications bot (Jan 21 2023 at 16:04):guest271314 commented on issue #3715:
You can write the wat file to a temporary file and then run it.
That doesn't work as described in this case https://github.com/bytecodealliance/wasmtime/issues/5614.
I tried both .wat and .wasm.
wasmtimeappears to be a special case executable that cannot be instructed to execute what is below the shebang.
Wasmtime GitHub notifications bot (Jan 21 2023 at 16:36):bjorn3 commented on issue #3715:
Did you try something like
#!/usr/bin/sh cat >foo.wat <<__END__ (module ;; Import the required fd_write WASI function which will write the given io vectors to stdout ;; The function signature for fd_write is: ;; (File Descriptor, *iovs, iovs_len, nwritten) -> Returns number of bytes written (import "wasi_unstable" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32))) (memory 1) (export "memory" (memory 0)) ;; Write 'hello world\n' to memory at an offset of 8 bytes ;; Note the trailing newline which is required for the text to appear (data (i32.const 8) "hello world\n") (func $main (export "_start") ;; Creating a new io vector within linear memory (i32.store (i32.const 0) (i32.const 8)) ;; iov.iov_base - This is a pointer to the start of the 'hello world\n' string (i32.store (i32.const 4) (i32.const 12)) ;; iov.iov_len - The length of the 'hello world\n' string (call $fd_write (i32.const 1) ;; file_descriptor - 1 for stdout (i32.const 0) ;; *iovs - The pointer to the iov array, which is stored at memory location 0 (i32.const 1) ;; iovs_len - We're printing 1 string stored in an iov - so one. (i32.const 20) ;; nwritten - A place in memory to store the number of bytes written ) drop ;; Discard the number of bytes written from the top of the stack ) ) __END__ wasmtime run foo.wat
Wasmtime GitHub notifications bot (Jan 21 2023 at 16:50):guest271314 commented on issue #3715:
Did you try something like
Just tried. The Native Messaging host exits.
Wasmtime GitHub notifications bot (Jan 21 2023 at 16:54):bjorn3 commented on issue #3715:
This specific example doesn't read from stdin and exits as soon as it wrote
hello world\nto stdout. As such it is expected that it exits. Furthermore the messages need to be in json format in both directions.
Wasmtime GitHub notifications bot (Jan 21 2023 at 16:55):bjorn3 edited a comment on issue #3715:
This specific example doesn't read from stdin and exits as soon as it wrote
hello world\nto stdout. As such it is expected that it exits. Furthermore the messages need to be in json format in both directions together with a length prefix (4 byte little endian integer I believe).
Wasmtime GitHub notifications bot (Jan 21 2023 at 16:56):guest271314 commented on issue #3715:
I uploaded the WASm file to https://webassembly.github.io/wabt/demo/wasm2wat/ and substituted that for the wat in your example.
I'm just trying to test WebAssembly version of a Native Messaging host. Thus I asked the WebAssembly experts how to do that after not being successful using
wasmtimeand learning thatwasmtimedoes not support shebang line.
Wasmtime GitHub notifications bot (Jan 21 2023 at 17:00):guest271314 commented on issue #3715:
@bjorn3 I know for a fact that both https://github.com/guest271314/native-messaging-c and https://github.com/guest271314/native-messaging-cpp work as expected. I compiled the C version to WASM using WASI SDK. I downloaded and installed
wasmtime. Now I am stuck trying to run the executable.
Wasmtime GitHub notifications bot (Jan 21 2023 at 17:12):guest271314 commented on issue #3715:
It shouldn't be this complicated that we have to use a shell script to run .wasm and .wat. when we have
wasmtimethat _should_ be capable of just interpreting a shebang lineand running the code below that line.At least provide users with the _option_ to use the shebang line.
I mean the line to install
wasmtimeitself uses Bash.curl https://wasmtime.dev/install.sh -sSf | bashExamples of using shebang for Python, Node.js, Deno, QuickJS, Bun Native Messaging hosts:
#!/usr/bin/env -S python3 -u#!/usr/bin/env -S ./node --max-old-space-size=6 --jitless --expose-gc --v8-pool-size=1#!/usr/bin/env -S ./deno run --v8-flags="--expose-gc,--jitless"#!/usr/bin/env -S ./qjs --std#!/usr/bin/env -S ./bun run --no-install --hot
Wasmtime GitHub notifications bot (Jan 21 2023 at 17:47):bjorn3 commented on issue #3715:
#!/usr/bin/env -S python3 -uIn Python
#indicates a comment, so no syntax extension is needed here.
#!/usr/bin/env -S ./node --max-old-space-size=6 --jitless --expose-gc --v8-pool-size=1
#!/usr/bin/env -S ./deno run --v8-flags="--expose-gc,--jitless"
#!/usr/bin/env -S ./qjs --std
#!/usr/bin/env -S ./bun run --no-install --hotThese have decided to extend javascript in a way incompatible with javascript engines not included in this list like browsers, which is not something wasmtime wants to do.
At least provide users with the option to use the shebang line.
As per https://github.com/bytecodealliance/wasmtime/issues/3715#issuecomment-1020216100 you should suggest adding it to the official wat specification, not as a wasmtime specific extension.
@bjorn3 I know for a fact that both https://github.com/guest271314/native-messaging-c and https://github.com/guest271314/native-messaging-cpp work as expected. I compiled the C version to WASM using WASI SDK. I downloaded and installed wasmtime. Now I am stuck trying to run the executable.
I don't think this issue is related at all to the shebang:
If I try
printf '\0\0\0\x02{}' | ./nm_c, I get\0\0\0\x02{}as response back followed by an infinite list of null bytes (as you didn't add an exit condition when 0 bytes are read, which indicates EOF). However when using the wasm version I get\0\0\0\x02{}in repeat. This is caused by an off-by-one error when allocating the message buffer. The following patch fixes this mistake:diff --git a/nm_c.c b/nm_c.c index a1afda1..e328d20 100644 --- a/nm_c.c +++ b/nm_c.c @@ -26,7 +26,7 @@ uint8_t* getMessage(size_t *inputLength) { // `message[0]`). // `sizeof(*message)` will hence return the size in bytes of the type // `message` points at, which means it's equivalent to `sizeof(uint_8)`. - uint8_t *message = calloc(messageLength, sizeof(*message)); + uint8_t *message = calloc(messageLength+1, sizeof(*message)); result = fread(message, sizeof(*message), messageLength, stdin); // `inputLength` is a pointer, so we store the length at the memory address it // points at. This way we return 2 values at once from a function!I suspect that applying this patch will fix the communication issue that lead to the native messaging host exiting.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:04):guest271314 commented on issue #3715:
I can do this just fine with the C Native Messaging host
globalThis.name = chrome.runtime.getManifest().short_name; globalThis.port = chrome.runtime.connectNative(globalThis.name); port.onMessage.addListener((message) => console.log(message)); port.onDisconnect.addListener((p) => console.log(chrome.runtime.lastError)); port.postMessage('');Thanks for the patch nonetheless. Kindly file a PR explaining why, what, how so we know exactly what is going on on the record of the repository. I am not a C or C++ expert. I am far more fluent in JavaScript. I had to get help to create the C and C++ versions.
As per https://github.com/bytecodealliance/wasmtime/issues/3715#issuecomment-1020216100 you should suggest adding it to the official wat specification, not as a wasmtime specific extension.
I think we should be able to use
/usr/bin/envwith .wat or .wasm files where we callwastimeas the environment.I suspect that applying this patch will fix the communication issue that lead to the native messaging host exiting.
Unfortunately that didn't change anything relevant to the Native Messaging host.
How is .wat relevant to the ability to call
wasmtime?I'm just trying to test WebAssembly as a Native Messaging host using the officially endorsed runtime. If you try the above hosts they each echo back the input from the browser. I have no idea how to do that using WebAssembly or
wasmtime. This is the first time I've actually successfully compiled a .wasm file.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:08):guest271314 commented on issue #3715:
I applied the patch, recompiled the C source code to WASM and the host still exited - when using the
catand redirection approach.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:12):guest271314 commented on issue #3715:
@bjorn3 If/when you have the time and are so inclined perhaps try testing using
wasmtimeyourself; c. to the other hosts I've linked to above. Perhaps then you will see where I'm at and why I commented here and filed an issue. The instructions are here https://github.com/guest271314/native-messaging-nodejs.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:17):bjorn3 commented on issue #3715:
Thanks for the patch nonetheless. Kindly file a PR explaining why, what, how so we know exactly what is going on on the record of the repository. I am not a C or C++ expert. I am far more fluent in JavaScript. I had to get help to create the C and C++ versions.
Sure, https://github.com/guest271314/native-messaging-c/pull/2.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:21):bjorn3 commented on issue #3715:
@bjorn3 If/when you have the time and are so inclined perhaps try testing using wasmtime yourself; c. to the other hosts I've linked to above. Perhaps then you will see where I'm at and why I commented here and filed an issue. The instructions are here https://github.com/guest271314/native-messaging-nodejs.
I will try it.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:24):guest271314 commented on issue #3715:
@bjorn3 If/when you have the time and are so inclined perhaps try testing using wasmtime yourself; c. to the other hosts I've linked to above. Perhaps then you will see where I'm at and why I commented here and filed an issue. The instructions are here https://github.com/guest271314/native-messaging-nodejs.
I will try it.
Don't hesitate to ask if you have any questions getting the host running.
I look forward to see how you will get
wasmtimeworking to use .wat or .wasm as a WebAssembly Native Messaging host.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:38):bjorn3 commented on issue #3715:
Got https://github.com/guest271314/native-messaging-c on native working after I moved it out of the
/tmpdir and made sure to use a trailing slash for the extension specifier innm_c.json. I did get a message with an array of exactly 0x33333 zero elements. Is this expected?
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:42):bjorn3 commented on issue #3715:
For me an
nm_c.shwith the following contents worked fine:#!/usr/bin/env bash wasmtime run nm_c.wasmMake sure to make the shell script executable though. Otherwise chromium says "Native host has exited.".
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:44):guest271314 commented on issue #3715:
Got https://github.com/guest271314/native-messaging-c on native working after I moved it out of the
/tmpdir and made sure to use a trailing slash for the extension specifier innm_c.json. I did get a message with an array of exactly 0x33333 zero elements. Is this expected?We pass 1MB to the host (
JSON.stringify(new Array(290715)).length // 1453576) and get that 1MB back.
Wasmtime GitHub notifications bot (Jan 21 2023 at 18:55):guest271314 commented on issue #3715:
For me an
nm_c.shwith the following contents worked fine:```shell
#!/usr/bin/env bashwasmtime run nm_c.wasm
```Make sure to make the shell script executable though. Otherwise chromium says "Native host has exited.".
Interesting. Still exiting here. You got the array back? Can you kindly post the nm_c_webassembly.json you copied to NativeMessagingHosts and manifest.json you use here or to a gist? Did you use WASI SDK to compile to WASM?
Wasmtime GitHub notifications bot (Jan 21 2023 at 19:48):guest271314 commented on issue #3715:
@bjorn3 This is what I get using C Native Messaging host
This is what I get using
#!/usr/bin/env bash wasmtime run nm_c.wasm
Wasmtime GitHub notifications bot (Jan 21 2023 at 20:12):JoeStrout commented on issue #3715:
Interesting to see other people running into this unnecessary hurdle. I doubt you'll get much traction, though; it seems the people in charge of wasmtime are sticklers more interested in their own notions of purity than in making a tool actually useful.
This will, one by one, drive away everybody who doesn't agree with them (like me; I gave up on WASM and wasmtime a while back, in part because of this). Then they'll look around at the people who are left, all nodding in agreement, and think they were justified in their stubbornness. That's just how these things go.
Wasmtime GitHub notifications bot (Jan 21 2023 at 20:34):tschneidereit locked issue #3715:
Feature
Make wasmtime hip to a shebang at the first line of the file, e.g.:
#!/usr/local/bin/wasmtimefollowed by the text-format wasm code as usual. Currently, the shell runs wasmtime and feeds it the script as expected, but wasmtime chokes on the first line and fails to run the following script.
Benefit
This would allow us to write scripts that can be executed by simply setting the executable bit and invoking them, like any other shell script. It's a matter of convenience. I have a lot of scripts in a lot of different languages; I don't have to remember what tool to use to run them because they all understand shebang — except for wasmtime.
Implementation
Just special-case it: if the first line of the script file starts with "#!", then ignore it and continue with the rest of the file.
Alternatives
I could write, for every wasm script I want to run, a little 2-line shell script that runs wasmtime with the path to the actual script. That would be annoying on so many levels.
Wasmtime GitHub notifications bot (Jan 21 2023 at 20:39):tschneidereit commented on issue #3715:
I locked this topic as all the discussion starting yesterday is off-topic. As laid out last year, the wasmtime project is dedicated to implementing standardized features, and will not accept non-standards, non-interoperable features. This isn't a matter of purity, but a matter of wanting to foster an ecosystem that's wider than this one project.
I'd again encourage those who want this feature to propose it to the WebAssembly Community Group as the right forum for discussions about changes to the standards themselves. If and when such a proposal gains traction there, we'll happily implement it in wasmtime.
(@bjorn3: thank you very much for your diligent work on trying to resolve users' problems—it's much appreciated! ❤️)
Last updated: Dec 23 2024 at 12:05 UTC
