Stream: git-wasmtime

Topic: wasmtime / issue #6741 Redesign Wasmtime's CLI flags

Wasmtime GitHub notifications bot (Jul 17 2023 at 23:35):

alexcrichton opened issue #6741:

Currently the set of CLI flags added to Wasmtime has been ad-hoc and generally "throw another option in there" style historically, which is great for easily adding new options (which happens a fair bit), but I personally feel like there have been some growing pains that may be worthwhile to address sooner rather than later.

Some concerns I have with the current CLI are:

• Discoverability is hard right now as --help or -h prints out quite a lot of information.
• Not a lot of single-letter flags are in use so CLI commands tend to be quite long
• It's tough to review "at a glance" what Wasmtime's capabilities are on the CLI through the help page since there's so much to sift through
• Some options have little papercuts like trying to remember --dir vs --mapdir, there's no way to inherit the caller's environment or a single env var from the environment easily, etc.
• Options can be randomly --disable-foo or --enable-foo and it's not clear why which is which when you're approaching the CLI for the first time.

To be clear none of these are dealbreakers by any means. While I think the Wasmtime CLI is important the embedding API are where things "get serious" so it's ok for the CLI to not be perfect. That being said folks often interact with Wasmtime through the CLI to start off with and it's also mega-useful during development and debugging and such.

So given all the above I'd like to propose a (hopefully) brief bikeshed about the Wasmtime CLI. I'd like to ideally address the above concerns and provide guidelines of how to add new options into the future. This will hopefully clean up our CLI, make it a bit more ergonomic to use, all while not hindering our ability to easily add various knobs here and there as they're implemented.

At a high level my proposal is "let's do what rustc does". I'm very biased in that regard as I helped design rustc's CLI as well. Some things I like about rustc though are:

• The output of rustc -h fits more-or-less on one terminal. This serves as a good index from which you can learn more.
• There are "option groups" that are behind other flags, such as rustc -C foo or rustc -Z foo. These single-letter capital flags are easy to type and easy to remember and serve as a good place to house sub-options.
• More information can be read through rustc -C help which provides an window into all the codegen capabilities of rustc
• Instead of --enable-foo or --disable-foo most rustc options are -C foo for --enable-foo or -C foo=no for --disable-foo. This makes it easy for callers to forcibly enable/disable something and you don't have to worry about what the defaults are since as a user you typically want to turn something on or off and you don't care too much about whatever the default happens to be.

My proposal here for Wasmtime's option groups are:

Codegen options (-C or --codegen)

• All --disable-foo flags become -C foo=no
• All --enable-foo flags become -C foo
• -C foo is shorthand for -C foo=yes
• *=yes can support y, yes, true, etc
• *=no can support n, no, false, etc

• Non-boolean flags support -C name=val where val is parsed as whatever
  name wants.

• Can support comma-separation as well such as -C foo,bar or -C foo=n,bar

old cli flag	new cli flag
--compiler <COMPILER>	-C compiler=..
-O, --optimize	-C opt-level=N / -O
--opt-level <LEVEL>	-C opt-level=N
--config <CONFIG_PATH>	-C cache-config=..
--disable-address-map	-C address-map=no
--disable-cache	-C cache=no
--disable-parallel-compilation	-C parallel-compilation=no
--epoch-interruption	-C epoch-interruption
-g	-C debuginfo / -g
--dynamic-memory-guard-size <SIZE>	-C dynamic-memory-guard-size=...
--static-memory-forced	-C static-memory-forced
--static-memory-guard-size <SIZE>	-C static-memory-guard-size=N
--static-memory-maximum-size <MAXIMUM>	-C static-memory-maximum-size=N
--relaxed-simd-deterministic	-C relaxed-simd-deterministic
--cranelift-enable <SETTING>	-C cranelift-NAME
--enable-cranelift-debug-verifier	-C cranelift-debug-verifier
--enable-cranelift-nan-canonicalization	-C cranelift-nan-canonicalization
--cranelift-set <NAME=VALUE>	-C cranelift-NAME=VALUE

Runtime options (-R or --runtime)

• Various *-unknown-* options are all -R unknown-* now.
• Same system for boolean flags as -C options

old cli flag	new cli flag
--allow-unknown-exports	-R unknown-exports-allow
--trap-unknown-imports	-R unknown-imports-trap
--default-values-unknown-imports	-R unknown-imports-default
--coredump-on-trap <PATH>	-R coredump=..
--disable-logging	-R logging=no
--disable-memory-init-cow	-R memory-init-cow=no
--dynamic-memory-reserved-for-growth <SIZE>	-R dynamic-memory-reserved-for-growth=...
--fuel <N>	-R fuel=N
--log-to-files	-R log-to-files
--max-instances <MAX_INSTANCES>	-R max-instances=N
--max-memories <MAX_MEMORIES>	-R max-memories=N
--max-memory-size <BYTES>	-R max-memory-size=N
--max-table-elements <MAX_TABLE_ELEMENTS>	-R max-table-elements=N
--max-tables <MAX_TABLES>	-R max-tables=N
--max-wasm-stack <MAX_WASM_STACK>	-R max-wasm-stack=N
--pooling-allocator	-R pooling-allocator
--trap-on-grow-failure	-R trap-on-grow-failure
--wasm-timeout <TIME>	-R timeout=N

WASI options (-W or --wasi)

• Separate from runtime options as this'll have WASI-proposal-specific
  configuration options.

• Leave --env as its own standalone flag

• Add support for --env FOO which inherits the env var FOO from the outer
  process.

old cli flag	new cli flag
--listenfd	-W listenfd
--tcplisten <SOCKET ADDRESS>	-W tcplisten=...
--wasi-modules <MODULE,MODULE,...>	-W common
--wasi-modules <MODULE,MODULE,...>	-W experimental-threads
--dir <DIRECTORY>	-W dir=..
--mapdir <GUEST_DIR::HOST_DIR>	-W dir=a::b
--env <NAME=VAL>	--env ...

Other flags

These are the other flags supported by Wasmtime which don't necessarily fit into the groups above. They're either important enough I think they should stick around at the top level or they have enough of their own "sub-syntax" that I'm not sure they fit well in the groups above.

• --wasm-features <FEATURE,FEATURE,...> - perhaps add something like a -F
  shortcut, I find it a bit annoying to always type this out. Support --wasm-features help for the extended help section on this.

• --preload <NAME=MODULE_PATH> - allow omitting NAME and infer it by default
  from MODULE_PATH.

• --profile <STRATEGY> - leave as-is

• --invoke <FUNCTION> - leave as-is
• --allow-precompiled - leave as-is
• --target <TARGET> - leave as-is
• --output <OUTPUT> - leave as-is
• --emit-clif <PATH> - leave as-is

---

I'm curious what others think about this! If it's broadly amenable and folks are ok with the breakage I'm happy to implement this myself, I don't think it'll take all that long (perhaps just a day or so)

Wasmtime GitHub notifications bot (Jul 18 2023 at 10:25):

sunfishcode commented on issue #6741:

Overall I like the ideas here; my main concern is that an end user may not always know whether something is a "codegen" option or a "runtime" option, for example epoch interruption or memory configuration. I don't have a fully formed idea yet, but I wonder if it would make sense to instead group the options by something like:

• Performance tuning: optimization level, caching, parallel compilation, memory init cow, etc.
• Wasm semantics: NaN canonicalization, deterministic relaxed simd, max memory size etc., trap on grow failure, etc.
• Debugging: -g, logging, profiling, etc.
• WASI flags as you say

What do you think?

Wasmtime GitHub notifications bot (Jul 18 2023 at 20:12):

alexcrichton commented on issue #6741:

Nah yeah I agree. My best strawman for an ordering like you're suggesting is something along the lines of:

• -P - performance tuning
• -C - codegen
• -D - debugging
• -W - wasm semantics
• -S - WASI & system config

<details>

old cli flag	new cli flag
-O, --optimize	-P opt-level=N / -O
--opt-level <LEVEL>	-P opt-level=N
--dynamic-memory-guard-size <SIZE>	-P dynamic-memory-guard-size=...
--static-memory-forced	-P static-memory-forced
--static-memory-guard-size <SIZE>	-P static-memory-guard-size=N
--static-memory-maximum-size <MAXIMUM>	-P static-memory-maximum-size=N
--dynamic-memory-reserved-for-growth <SIZE>	-P dynamic-memory-reserved-for-growth=...
--pooling-allocator	-P pooling-allocator
--disable-memory-init-cow	-P memory-init-cow=no
--compiler <COMPILER>	-C compiler=..
--enable-cranelift-debug-verifier	-C cranelift-debug-verifier
--cranelift-enable <SETTING>	-C cranelift-NAME
--cranelift-set <NAME=VALUE>	-C cranelift-NAME=VALUE
--config <CONFIG_PATH>	-C cache-config=..
--disable-cache	-C cache=no
--disable-parallel-compilation	-C parallel-compilation=no
-g	-D debuginfo / -g
--disable-address-map	-D address-map=no
--disable-logging	-D logging=no
--log-to-files	-D log-to-files
--coredump-on-trap <PATH>	-D coredump=..
--relaxed-simd-deterministic	-W relaxed-simd-deterministic
--enable-cranelift-nan-canonicalization	-W cranelift-nan-canonicalization
--fuel <N>	-W fuel=N
--epoch-interruption	-W epoch-interruption
--allow-unknown-exports	-W unknown-exports-allow
--trap-unknown-imports	-W unknown-imports-trap
--default-values-unknown-imports	-W unknown-imports-default
--max-instances <MAX_INSTANCES>	-W max-instances=N
--max-memories <MAX_MEMORIES>	-W max-memories=N
--max-memory-size <BYTES>	-W max-memory-size=N
--max-table-elements <MAX_TABLE_ELEMENTS>	-W max-table-elements=N
--max-tables <MAX_TABLES>	-W max-tables=N
--max-wasm-stack <MAX_WASM_STACK>	-W max-wasm-stack=N
--trap-on-grow-failure	-W trap-on-grow-failure
--wasm-timeout <TIME>	-W timeout=N
--listenfd	-S listenfd
--tcplisten <SOCKET ADDRESS>	-S tcplisten=...
--wasi-modules <MODULE,MODULE,...>	-S common
--wasi-modules <MODULE,MODULE,...>	-S experimental-threads
--dir <DIRECTORY>	-S dir=..
--mapdir <GUEST_DIR::HOST_DIR>	-S dir=a::b

</details>

One thing that feels a bit odd though is that I was hoping for a clean split where wasmtime compile would be a strict subset of wasmtime run options or something like that. For example there's no need for wasmtime compile to support -S or -D options in theory. That's what I was roughly hoping for above where wasmtime compile would drop all the -R options but keep all the -C options.

Here it's not so straightforward though because fuel is both a wasm semantics option and a compilation option. Other wasm semantic options like max-memory-size don't affect the compilation though. This may not be a great thing to optimize for though and the more I think about it the more it feels sort of arbitrary.

In any case I like the idea of the categories you're thinking of too, would you move any of the above options around though?

Wasmtime GitHub notifications bot (Jul 19 2023 at 17:35):

sunfishcode commented on issue #6741:

Some bikeshed suggestions:

• Name the performance tuning flag -O, as in -O pooling-allocator and so on, because that's familiar to me from common compiler flags, though I don't know if that will have the same association for others.
• Rename cranelift-nan-canonicalization to just nan-canonlicalization and maybe have winch fail if that flag is given and it isn't supported yet. Because from an end-user perspective, this is a semantics thing, not a cranelift thing.

The compile vs. run split is a good point. My best idea is to have compile accept all the same options as run, and record those options along with the output so that run can default to them. run could allow some of those flags to be overridden, but not all, because some will be baked into the compiled code. But it's awkward for users to figure out which flags need to be specified at compile and which can be overridden at run.

But maybe that's ok? Maybe the priority should be to make the simple run user experience the simplest, and organizing the options in terms of tuning/semantics/debugging/etc. seems best for that?

Wasmtime GitHub notifications bot (Jul 19 2023 at 18:06):

fitzgen commented on issue #6741:

Don't feel like joining this particular bike shed, but one TODO item to record for posterity if we change the flags to enable different Wasm proposals is we will want to send a PR to update https://webassembly.org/roadmap/ as well.

Wasmtime GitHub notifications bot (Jul 19 2023 at 18:54):

alexcrichton commented on issue #6741:

I like the idea of -O for tuning, yeah, and renaming to nan-canonicalization sounds good to me. I also agree it's probably best to optimize for run and compile can do what's necessary to support the flags, so given that I think the proposal would be:

old cli flag	new cli flag
-O, --optimize	-O opt-level=N
--opt-level <LEVEL>	-O opt-level=N
--dynamic-memory-guard-size <SIZE>	-O dynamic-memory-guard-size=...
--static-memory-forced	-O static-memory-forced
--static-memory-guard-size <SIZE>	-O static-memory-guard-size=N
--static-memory-maximum-size <MAXIMUM>	-O static-memory-maximum-size=N
--dynamic-memory-reserved-for-growth <SIZE>	-O dynamic-memory-reserved-for-growth=...
--pooling-allocator	-O pooling-allocator
--disable-memory-init-cow	-O memory-init-cow=no
--compiler <COMPILER>	-C compiler=..
--enable-cranelift-debug-verifier	-C cranelift-debug-verifier
--cranelift-enable <SETTING>	-C cranelift-NAME
--cranelift-set <NAME=VALUE>	-C cranelift-NAME=VALUE
--config <CONFIG_PATH>	-C cache-config=..
--disable-cache	-C cache=no
--disable-parallel-compilation	-C parallel-compilation=no
-g	-D debuginfo / -g
--disable-address-map	-D address-map=no
--disable-logging	-D logging=no
--log-to-files	-D log-to-files
--coredump-on-trap <PATH>	-D coredump=..
--relaxed-simd-deterministic	-W relaxed-simd-deterministic
--enable-cranelift-nan-canonicalization	-W nan-canonicalization
--fuel <N>	-W fuel=N
--epoch-interruption	-W epoch-interruption
--allow-unknown-exports	-W unknown-exports-allow
--trap-unknown-imports	-W unknown-imports-trap
--default-values-unknown-imports	-W unknown-imports-default
--max-instances <MAX_INSTANCES>	-W max-instances=N
--max-memories <MAX_MEMORIES>	-W max-memories=N
--max-memory-size <BYTES>	-W max-memory-size=N
--max-table-elements <MAX_TABLE_ELEMENTS>	-W max-table-elements=N
--max-tables <MAX_TABLES>	-W max-tables=N
--max-wasm-stack <MAX_WASM_STACK>	-W max-wasm-stack=N
--trap-on-grow-failure	-W trap-on-grow-failure
--wasm-timeout <TIME>	-W timeout=N
--listenfd	-S listenfd
--tcplisten <SOCKET ADDRESS>	-S tcplisten=...
--wasi-modules <MODULE,MODULE,...>	-S common
--wasi-modules <MODULE,MODULE,...>	-S experimental-threads
--dir <DIRECTORY>	-S dir=..
--mapdir <GUEST_DIR::HOST_DIR>	-S dir=a::b

Wasmtime GitHub notifications bot (Jul 19 2023 at 19:05):

tschneidereit commented on issue #6741:

Also don't feel like I need to share my favorite shed coloring scheme, but a question: is it possible to have help output for specific flags? I.e., can I do wasmtime help run -O (or wasmtime help -O) and get output just for all -O flags?

Wasmtime GitHub notifications bot (Jul 19 2023 at 19:06):

tschneidereit commented on issue #6741:

And another question: can we in redesigning this look into making flags available as env vars as well where reasonable? Ideally in a way that works for libwasmtime embeddings, not just the CLI? Obviously that doesn't make sense for most options, but it'd be excellent for debug/profiling things, for example.

Wasmtime GitHub notifications bot (Jul 19 2023 at 22:49):

alexcrichton commented on issue #6741:

I'm not sure of a way to configure clap to do that, but I've also found that if I'm looking for a specific option then wasmtime --help | less and searching for foo is good enough for exploring a specific option. Or alternatively with the groups above it'd be wasmtime -W help or similar.

And another question: can we in redesigning this look into making flags available as env vars as well where reasonable? Ideally in a way that works for libwasmtime embeddings, not just the CLI?

On one hand this is easy because I think clap exposes the ability to do this. On the other hand this isn't easy because embeddings aren't using clap they're using Config. I do agree though that for profiling and debugging options env vars are mega-useful since you don't have to thread them around everywhere.

Wasmtime GitHub notifications bot (Aug 29 2023 at 18:08):

alexcrichton commented on issue #6741:

I've made a PR for this at https://github.com/bytecodealliance/wasmtime/pull/6925

Wasmtime GitHub notifications bot (Oct 23 2023 at 10:54):

tschneidereit commented on issue #6741:

I guess we can close this?

Wasmtime GitHub notifications bot (Oct 23 2023 at 14:33):

alexcrichton commented on issue #6741:

Ah yes, thanks!

Wasmtime GitHub notifications bot (Oct 23 2023 at 14:33):

alexcrichton closed issue #6741:

Currently the set of CLI flags added to Wasmtime has been ad-hoc and generally "throw another option in there" style historically, which is great for easily adding new options (which happens a fair bit), but I personally feel like there have been some growing pains that may be worthwhile to address sooner rather than later.

Some concerns I have with the current CLI are:

• Discoverability is hard right now as --help or -h prints out quite a lot of information.
• Not a lot of single-letter flags are in use so CLI commands tend to be quite long
• It's tough to review "at a glance" what Wasmtime's capabilities are on the CLI through the help page since there's so much to sift through
• Some options have little papercuts like trying to remember --dir vs --mapdir, there's no way to inherit the caller's environment or a single env var from the environment easily, etc.
• Options can be randomly --disable-foo or --enable-foo and it's not clear why which is which when you're approaching the CLI for the first time.

To be clear none of these are dealbreakers by any means. While I think the Wasmtime CLI is important the embedding API are where things "get serious" so it's ok for the CLI to not be perfect. That being said folks often interact with Wasmtime through the CLI to start off with and it's also mega-useful during development and debugging and such.

So given all the above I'd like to propose a (hopefully) brief bikeshed about the Wasmtime CLI. I'd like to ideally address the above concerns and provide guidelines of how to add new options into the future. This will hopefully clean up our CLI, make it a bit more ergonomic to use, all while not hindering our ability to easily add various knobs here and there as they're implemented.

At a high level my proposal is "let's do what rustc does". I'm very biased in that regard as I helped design rustc's CLI as well. Some things I like about rustc though are:

• The output of rustc -h fits more-or-less on one terminal. This serves as a good index from which you can learn more.
• There are "option groups" that are behind other flags, such as rustc -C foo or rustc -Z foo. These single-letter capital flags are easy to type and easy to remember and serve as a good place to house sub-options.
• More information can be read through rustc -C help which provides an window into all the codegen capabilities of rustc
• Instead of --enable-foo or --disable-foo most rustc options are -C foo for --enable-foo or -C foo=no for --disable-foo. This makes it easy for callers to forcibly enable/disable something and you don't have to worry about what the defaults are since as a user you typically want to turn something on or off and you don't care too much about whatever the default happens to be.

My proposal here for Wasmtime's option groups are:

Codegen options (-C or --codegen)

• All --disable-foo flags become -C foo=no
• All --enable-foo flags become -C foo
• -C foo is shorthand for -C foo=yes
• *=yes can support y, yes, true, etc
• *=no can support n, no, false, etc

• Non-boolean flags support -C name=val where val is parsed as whatever
  name wants.

• Can support comma-separation as well such as -C foo,bar or -C foo=n,bar

old cli flag	new cli flag
--compiler <COMPILER>	-C compiler=..
-O, --optimize	-C opt-level=N / -O
--opt-level <LEVEL>	-C opt-level=N
--config <CONFIG_PATH>	-C cache-config=..
--disable-address-map	-C address-map=no
--disable-cache	-C cache=no
--disable-parallel-compilation	-C parallel-compilation=no
--epoch-interruption	-C epoch-interruption
-g	-C debuginfo / -g
--dynamic-memory-guard-size <SIZE>	-C dynamic-memory-guard-size=...
--static-memory-forced	-C static-memory-forced
--static-memory-guard-size <SIZE>	-C static-memory-guard-size=N
--static-memory-maximum-size <MAXIMUM>	-C static-memory-maximum-size=N
--relaxed-simd-deterministic	-C relaxed-simd-deterministic
--cranelift-enable <SETTING>	-C cranelift-NAME
--enable-cranelift-debug-verifier	-C cranelift-debug-verifier
--enable-cranelift-nan-canonicalization	-C cranelift-nan-canonicalization
--cranelift-set <NAME=VALUE>	-C cranelift-NAME=VALUE

Runtime options (-R or --runtime)

• Various *-unknown-* options are all -R unknown-* now.
• Same system for boolean flags as -C options

old cli flag	new cli flag
--allow-unknown-exports	-R unknown-exports-allow
--trap-unknown-imports	-R unknown-imports-trap
--default-values-unknown-imports	-R unknown-imports-default
--coredump-on-trap <PATH>	-R coredump=..
--disable-logging	-R logging=no
--disable-memory-init-cow	-R memory-init-cow=no
--dynamic-memory-reserved-for-growth <SIZE>	-R dynamic-memory-reserved-for-growth=...
--fuel <N>	-R fuel=N
--log-to-files	-R log-to-files
--max-instances <MAX_INSTANCES>	-R max-instances=N
--max-memories <MAX_MEMORIES>	-R max-memories=N
--max-memory-size <BYTES>	-R max-memory-size=N
--max-table-elements <MAX_TABLE_ELEMENTS>	-R max-table-elements=N
--max-tables <MAX_TABLES>	-R max-tables=N
--max-wasm-stack <MAX_WASM_STACK>	-R max-wasm-stack=N
--pooling-allocator	-R pooling-allocator
--trap-on-grow-failure	-R trap-on-grow-failure
--wasm-timeout <TIME>	-R timeout=N

WASI options (-W or --wasi)

• Separate from runtime options as this'll have WASI-proposal-specific
  configuration options.

• Leave --env as its own standalone flag

• Add support for --env FOO which inherits the env var FOO from the outer
  process.

old cli flag	new cli flag
--listenfd	-W listenfd
--tcplisten <SOCKET ADDRESS>	-W tcplisten=...
--wasi-modules <MODULE,MODULE,...>	-W common
--wasi-modules <MODULE,MODULE,...>	-W experimental-threads
--dir <DIRECTORY>	-W dir=..
--mapdir <GUEST_DIR::HOST_DIR>	-W dir=a::b
--env <NAME=VAL>	--env ...

Other flags

These are the other flags supported by Wasmtime which don't necessarily fit into the groups above. They're either important enough I think they should stick around at the top level or they have enough of their own "sub-syntax" that I'm not sure they fit well in the groups above.

• --wasm-features <FEATURE,FEATURE,...> - perhaps add something like a -F
  shortcut, I find it a bit annoying to always type this out. Support --wasm-features help for the extended help section on this.

• --preload <NAME=MODULE_PATH> - allow omitting NAME and infer it by default
  from MODULE_PATH.

• --profile <STRATEGY> - leave as-is

• --invoke <FUNCTION> - leave as-is
• --allow-precompiled - leave as-is
• --target <TARGET> - leave as-is
• --output <OUTPUT> - leave as-is
• --emit-clif <PATH> - leave as-is

---

I'm curious what others think about this! If it's broadly amenable and folks are ok with the breakage I'm happy to implement this myself, I don't think it'll take all that long (perhaps just a day or so)

Last updated: Nov 22 2024 at 16:03 UTC