Stream: git-wasmtime

Topic: wasmtime / issue #7925 Add component adaptor for wasi-nn

view this post on Zulip Wasmtime GitHub notifications bot (Feb 13 2024 at 13:38):

danbev opened issue #7925:

Feature

Add component adaptor for wasi-nn (wasi_ephemeral_nn).

Benefit

Currently when trying to use wasm-tools component new with a core wasm module leads to the following error:
```console
$ wasm-tools component new -vvv ./target/wasm32-wasi/release/inf_wasi.wasm \
--adapt wit-lib/wasi_snapshot_preview1.reactor.wasm \
-o target/inf-wasi-component.wasm
...

error: failed to encode a component from module

Caused by:
0: module requires an import interface named wasi_ephemeral_nn
`` My thinking here is that adding an adapter similar to wasi_snapshot_preview1 but for wasi_ephemeral_nn` would enable the component to be created.

Implementation

I don't have any suggestions here but if I can get some guidance I'd be willing to try to implement this.

Alternatives

N/A

view this post on Zulip Wasmtime GitHub notifications bot (Feb 13 2024 at 17:26):

pchickey commented on issue #7925:

I don't think a component adapter is the path to go for wasi-nn. The component adapter was created to solve a very specific problem: we needed to create components out of modules that had some of their imports created in difficult to modify (especially before standardization is complete) upstream source trees such as wasi-libc and rust's std libraries. Because we couldn't upgrade those module imports to use e.g. a wit-bindgen! macro to generate component imports, we created an adapter to go between them, but in practice the adapter is very difficult to maintain because it needs to be written in a subset of Rust that doesn't result in any table or data sections existing in the linked module, and also the adapter is in the process of becoming obsolete, now that WASI 0.2 is stabilized, because we are actively upstreaming the changes in libc and std to use wit-bindgen component bindings directly.

Wasi-nn doesnt have those problems, that I am aware of: users of wasi-nn have much easier paths to upgrading their imports of wasi-nn from the (now legacy) witx abi module imports to wit-bindgen component imports. Have you tried to upgrade your wasi-nn user code in that way?

view this post on Zulip Wasmtime GitHub notifications bot (Feb 14 2024 at 06:31):

danbev closed issue #7925:

Feature

Add component adaptor for wasi-nn (wasi_ephemeral_nn).

Benefit

Currently when trying to use wasm-tools component new with a core wasm module leads to the following error:
```console
$ wasm-tools component new -vvv ./target/wasm32-wasi/release/inf_wasi.wasm \
--adapt wit-lib/wasi_snapshot_preview1.reactor.wasm \
-o target/inf-wasi-component.wasm
...

error: failed to encode a component from module

Caused by:
0: module requires an import interface named wasi_ephemeral_nn
`` My thinking here is that adding an adapter similar to wasi_snapshot_preview1 but for wasi_ephemeral_nn` would enable the component to be created.

Implementation

I don't have any suggestions here but if I can get some guidance I'd be willing to try to implement this.

Alternatives

N/A

view this post on Zulip Wasmtime GitHub notifications bot (Feb 14 2024 at 06:31):

danbev commented on issue #7925:

@pchickey Thanks for the detailed explanation about the adapter, I appreciate it!

users of wasi-nn have much easier paths to upgrading their imports of wasi-nn from the (now legacy) witx abi module imports to wit-bindgen component imports. Have you tried to upgrade your wasi-nn user code in that way?

I actually have not tried that which is a little embarrassing. I just ran into the error and it looked like the same issue I had seen before and I assumed it would have to be handled in the same way. I'll upgrade wasi-nn and try that out. Thanks!

view this post on Zulip Wasmtime GitHub notifications bot (Feb 14 2024 at 17:13):

pchickey commented on issue #7925:

Don't be embarrassed, this stuff is brand new and there are nowhere near enough docs. Happy to help.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 19 2025 at 11:35):

adiibanez commented on issue #7925:

@pchickey what is the current state of support for wasi-nn in wasmtime?

trying to run https://github.com/LlamaEdge/LlamaEdge on wasmtime resp. wasmex ( elixir runtime based on wasmtime )

But I'm having a hard time figuring out how to configure / init. I'm using wasmtime 30.0.2 (398694a59 2025-02-25) on osx

With wasmedge the initialization looks like this: 

wasmedge version 0.14.1
 (plugin "wasi_logging") version 0.1.0.0
/Users/adrianibanez/.wasmedge/lib/../plugin/libwasmedgePluginWasiNN.dylib (plugin "wasi_nn") version 0.1.15.0
adrianibanez@MacBookPro elixir_desktop_lvn % wasmedge --dir .:. --nn-preload default:GGML:AUTO:Llama-3.2-1B-Instruct-Q5_K_M.gguf llama-chat.wasm -p llama-3-chat

[INFO] llama-chat version: 0.16.11
[INFO] Model name: default
[INFO] Model alias: default
[INFO] Prompt template: llama-3-chat
[INFO] Context size: 512
[INFO] Number of tokens to predict: -1
[INFO] Number of layers to run on the GPU: 100
[INFO] Threads: 2
[INFO] Batch size for prompt processing: 512
[INFO] Temperature for sampling: 1
[INFO] Penalize repeat sequence of tokens: 1.1
[INFO] Presence penalty (0.0 = disabled): 0
[INFO] Frequency penalty (0.0 = disabled): 0
[INFO] BNF-like grammar:
[INFO] Enable prompt log: false
[INFO] Enable plugin log: false
[INFO] Temperature for sampling: 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 512
[INFO] Wasi-nn-ggml plugin: b4875 (commit 7841fc72)

================================== Running in interactive mode. ===================================

    - Press [Ctrl+C] to interject at any time.
    - Press [Return] to end the input.
    - For multi-line inputs, end each line with '\' and press [Return] to get another line.


[You]:
What do you know about wasi-nn support in wasmtime?

[Bot]:
I can provide general information about the concept of "Wasn't I right?" or "Wise one" in the context of Wasimite (also known as Wasmime) in Wasmime, but it seems you are referring to a game that uses this name.

In Wasimite, wasi-nn support is an auto-heal ability that summons a wispy duplicate of the player's character, which can attack and deal damage. It's often used for situations where the player might need additional protection or when they're low on health without leaving their current location.

[You]: Sure ...

view this post on Zulip Wasmtime GitHub notifications bot (Mar 19 2025 at 11:36):

adiibanez edited a comment on issue #7925:

@pchickey what is the current state of support for wasi-nn in wasmtime?

trying to run https://github.com/LlamaEdge/LlamaEdge on wasmtime resp. wasmex ( elixir runtime based on wasmtime )

But I'm having a hard time figuring out how to configure / init. I'm using wasmtime 30.0.2 (398694a59 2025-02-25) on osx

wasmtime --env MODEL=Llama-3.2-1B-Instruct-Q5_K_M.gguf \
         --env PROMPT="What is the capital of France?" \
         llama-chat.wasm
Error: failed to run main module `llama-chat.wasm`

Caused by:
    0: failed to instantiate "llama-chat.wasm"
    1: unknown import: `wasi_ephemeral_nn::compute` has not been defined
adrianibanez@MacBookPro elixir_desktop_lvn %

With wasmedge the initialization looks like this: 

wasmedge version 0.14.1
 (plugin "wasi_logging") version 0.1.0.0
/Users/adrianibanez/.wasmedge/lib/../plugin/libwasmedgePluginWasiNN.dylib (plugin "wasi_nn") version 0.1.15.0
adrianibanez@MacBookPro elixir_desktop_lvn % wasmedge --dir .:. --nn-preload default:GGML:AUTO:Llama-3.2-1B-Instruct-Q5_K_M.gguf llama-chat.wasm -p llama-3-chat

[INFO] llama-chat version: 0.16.11
[INFO] Model name: default
[INFO] Model alias: default
[INFO] Prompt template: llama-3-chat
[INFO] Context size: 512
[INFO] Number of tokens to predict: -1
[INFO] Number of layers to run on the GPU: 100
[INFO] Threads: 2
[INFO] Batch size for prompt processing: 512
[INFO] Temperature for sampling: 1
[INFO] Penalize repeat sequence of tokens: 1.1
[INFO] Presence penalty (0.0 = disabled): 0
[INFO] Frequency penalty (0.0 = disabled): 0
[INFO] BNF-like grammar:
[INFO] Enable prompt log: false
[INFO] Enable plugin log: false
[INFO] Temperature for sampling: 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 512
[INFO] Wasi-nn-ggml plugin: b4875 (commit 7841fc72)

================================== Running in interactive mode. ===================================

    - Press [Ctrl+C] to interject at any time.
    - Press [Return] to end the input.
    - For multi-line inputs, end each line with '\' and press [Return] to get another line.


[You]:
What do you know about wasi-nn support in wasmtime?

[Bot]:
I can provide general information about the concept of "Wasn't I right?" or "Wise one" in the context of Wasimite (also known as Wasmime) in Wasmime, but it seems you are referring to a game that uses this name.

In Wasimite, wasi-nn support is an auto-heal ability that summons a wispy duplicate of the player's character, which can attack and deal damage. It's often used for situations where the player might need additional protection or when they're low on health without leaving their current location.

[You]: Sure ...

view this post on Zulip Wasmtime GitHub notifications bot (Mar 19 2025 at 15:49):

abrown commented on issue #7925:

I'm afraid Wasmtime and WasmEdge's command line flags will differ too much here for that example to work. All of Wasmtime's WASI-related flag live under --wasi or -S; you could run wasmtime -S help and take note of the nn-related flags to configure wasi-nn with Wasmtime. As for the current state of wasi-nn in Wasmtime, it contains backends for OpenVINO, ONNX, PyTorch, and WinML, so you'll be able to run models that run directly on those ML frameworks, but probably not the backend you need to run that GGUF model (I believe there's a GGML backend floating around somewhere but has not been upstreamed). IIRC, @jianjunz has a demo somewhere that demonstrates running an LLM similar to what you're trying to do but that lives somewhere else; if you're interested, we can coordinate on Zulip.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 28 2025 at 10:16):

jianjunz commented on issue #7925:

IIRC, @jianjunz has a demo somewhere that demonstrates running an LLM similar to what you're trying to do but that lives somewhere else;

My draft code for ggml support is uploaded to https://github.com/jianjunz/wasmtime/tree/ggml. It's only tested with phi-3 mini.

Last updated: Apr 16 2025 at 22:03 UTC