Benoît C. (Jun 26 2021 at 21:23): Benoît C. (Jun 26 2021 at 21:23):Hello!
I'm coming from the Helix plugins system proposal issue (https://github.com/helix-editor/helix/issues/122).
To put it into context, I want to build a MVP of this, but I'm mostly hesitating on which runtime should be embedded.
Here are a few constraints:
We'll be using WASI.
We should be able to perform tasks in parallel.
- Plugins should be able to spawn threads and call host functions later.
E.g.: a ripgrep plugin should spawn a thread to execute ripgrep and collect the results back into the UI without causing the editor to freeze.
- Plugins should be able to run in parallel while sharing some states.
E.g.: a keybinds manager plugin must provide some way to add new bindings, and the manager itself need to be able to call functions from other plugins.
Not absolutely necessary, but I would like to support a Ahead-of-Time workflow as an optional optimization. For this Lucet looks promising.
E.g.: as a plugin developer, I want my plugin to be usable without recompiling Helix binary. However, as a Helix developer I might want to embed most useful plugins out of the box. Additionally, as an end user, I might want to recompile Helix binary myself with additional plugins I'm commonly using.
As far as I know (note that I don't know much yet haha), both Wasmtime (+ Lucet) and Wasmer allows me to do this right now, but I can't decide which one is best suited.
I'm also interested in using witx-bindgen to generate API in various languages. README says to reach out here to get more information first, so I'm interested in that too.
Till Schneidereit (Jun 28 2021 at 15:01):Hi Benoît!
I think Wasmtime would be well suited for what you're describing. As I mentioned on the issue thread, our recently overhauled embedding API makes it very robust for use in multi-threaded, async runtime environments, which IIUC is a key requirement for you.
Here are a few constraints:
We'll be using WASI.
Obviously that's something we support :smile:
I assume you'll want to also expose other APIs though, for which witx-bindgen might come in very handy. You can describe the APIs using a simple IDL, and then generate high-level bindings for both the host and content. Currently this works for Rust on both sides (using Wasmtime as the host), and JS on the host side (for use in browsers and Node/Deno), but we're working on adding bindings for other languages, so your plugins could have a nice toolchain even when written in languages other than Rust.
We should be able to perform tasks in parallel.
I assume you'll want to use an async runtime for that, which Wasmtime supports very well—see above.
- Plugins should be able to spawn threads and call host functions later. E.g.: a ripgrep plugin should spawn a thread to execute ripgrep and collect the results back into the UI without causing the editor to freeze.
As long as these function calls can happen asynchronously, that shouldn't be a problem. Everything else is less straightforward: there's not yet a standard for doing shared-memory multi-threading outside the browser.
- Plugins should be able to run in parallel while sharing some states.
Do you need fully shared state for this, or the ability to propagate state changes? The latter should be readily doable today, but the former would require some kind of shared-memory multi-threading, see above.
This is true for all runtimes, I should note, and not some limitation of Wasmtime.
E.g.: a keybinds manager plugin must provide some way to add new bindings, and the manager itself need to be able to call functions from other plugins. Not absolutely necessary, but I would like to support a Ahead-of-Time workflow as an optional optimization. For this Lucet looks promising.
Wasmtime has a wasmtime compile subcommand for the same purpose. We've by now integrated all functionality from Lucet that Wasmtime didn't used to have, and are fully focusing on Wasmtime going forward.
E.g.: as a plugin developer, I want my plugin to be usable without recompiling Helix binary. However, as a Helix developer I might want to embed most useful plugins out of the box. Additionally, as an end user, I might want to recompile Helix binary myself with additional plugins I'm commonly using.
That should be easy to support, yes.
As far as I know (note that I don't know much yet haha), both Wasmtime (+ Lucet) and Wasmer allows me to do this right now, but I can't decide which one is best suited.
Ultimately that's a choice you'll have to make, of course, but we're happy to answer any questions you might have, and help you understand what Wasmtime does or doesn't offer :smile:
Benoît C. (Jun 28 2021 at 20:02):Hi Till!
Thank you for the detailed answer!
As I mentioned on the issue thread, our recently overhauled embedding API makes it very robust for use in multi-threaded, async runtime environments, which IIUC is a key requirement for you.
I've been looking a bit deeper at the API, and it indeed looks very good! That's definitely a winning point. I couldn't yet find the equivalent in wasmer doc.
Do you need fully shared state for this, or the ability to propagate state changes?
Good point. It's not required to support _fully_ shared state, we mostly need the ability to propagate state changes at some point.
The latter should be readily doable today, but the former would require some kind of shared-memory multi-threading, see above.
For the latter, do you refer to the custom struct provided as second argument of Store::new?
I was considering using this to give a try.
I assume you'll want to also expose other APIs though, for which witx-bindgen might come in very handy. You can describe the APIs using a simple IDL, and then generate high-level bindings for both the host and content.
Yes, this looks very promising!
I understand in the current state of witx-bindgen, we could have a workflow using it for both our editor (host) and plugins (content) written in Rust. Our intention is indeed to make sure that, at term, users are not locked with a single extension language, and it doesn't have to be perfect from the beginning. As such, we're perfectly fine with starting by officially supporting Rust only at first and make that list grow at the same pace as witx-bindgen.
I suppose you already plan on generating interfaces for languages such as C/C++ and Go, but I also noticed AssemblyScript mentioned in wasmtime documentation. Can I assume AssemblyScript interface generation is also planned at some point?
We were considering multiple languages including AssemblyScript to embed a wasm scripting language at some point in the future and witx-bindgen support will be a big factor in the decision process.
Wasmtime has a
wasmtime compilesubcommand for the same purpose. We've by now integrated all functionality from Lucet that Wasmtime didn't used to have, and are fully focusing on Wasmtime going forward.
Oh! I missed that. This is just perfect.
Till Schneidereit (Jun 28 2021 at 20:10):I'm glad to hear this is helpful, and that things look promising!
For the latter, do you refer to the custom struct provided as second argument of Store::new?
I was considering using this to give a try.
Oh, I meant in-content shared memory, but you're right: for many uses being able to access mutable state in the host bindings would probably work just fine. And yes, that's definitely something you can do with the embedding API and the right synchronization primitives.
I suppose you already plan on generating interfaces for languages such as C/C++ and Go, but I also noticed AssemblyScript mentioned in wasmtime documentation. Can I assume AssemblyScript interface generation is also planned at some point?
We don't have any firm plans for it, but it seems highly plausible that there will be AssemblyScript bindings for witx-bindgen :smile:
Benoît C. (Jun 28 2021 at 20:30):Oh, I meant in-content shared memory, but you're right: for many uses being able to access mutable state in the host bindings would probably work just fine.
You probably know a better way then haha
I was thinking about providing host functions to get access to various states across plugins here (in-content shared memory / states); and that's probably a pattern I'll use at some point, but for my own information, what did you have in mind?
We don't have any firm plans for it, but it seems highly plausible that there will be AssemblyScript bindings for witx-bindgen :smile:
Glad to hear that :smiley:
I'll be following the progresses!
Till Schneidereit (Jun 28 2021 at 21:33):You probably know a better way then haha I was thinking about providing host functions to get access to various states across plugins here (in-content shared memory / states); and that's probably a pattern I'll use at some point, but for my own information, what did you have in mind?
I think where possible it'd be easiest to have this state live host-side, and have the host functions provide access to it for all plugins. That way you also don't run the risk of a plugin corrupting this state, and then others (perhaps silently) breaking because of it
Alex Crichton (Jun 28 2021 at 21:41):On the wasm-content side, witx-bindgen and the general scheme around interface types and the canonical ABI is all being designed explicitly to support multiple languages on all sides, meaning that a wasm module using interface types can be written in any language and the runtime consuming the wasm module can also be any supporting runtime (e.g. wasmtime, a browser, node, ...). Currently I've implemented host support in the form of Rust wasmtime bindings and JS web/node/deno bindings. For the compiled-to-wasm side I've only implemented Rust so far, but I hope to implement at least C in the near future as at least a proof-of-concept. The intention is that writing a language binding generator isn't the hardest thing in the world (but also not a trivial task), so other languages can be done as they come up as well.
Benoît C. (Jul 05 2021 at 00:22):Thank you both for the answers so far! I'm looking forward to the future of witx-bindgen. Until there is an official way to integrate the tool in our toolchain (probably similar to cbindgen with build.rs?), I think we'll generate and commit the files generated by the CLI tool to our VCS :smile:
Last updated: Nov 22 2024 at 16:03 UTC