Stream: wasi

Topic: ✔ Filesystem access from component in Rust host

Christoph Brewing (Aug 06 2024 at 10:31):

My current use-case anticipates some Go code which is supposed to be called as a WASM component and eventually has to read out a text file.

I have no problems in case the code is wrapped into a main() function and built and called like this:

tinygo build -o reader.wasm -target=wasip2 read.go
wasmtime run --dir=/path-to-host-folder::/ reader.wasm /text-file.txt

However, for some reason I do not succeed to access any folder or file in case the code is wrapped in a function, say display, exported via an interface, wrapped in a componenet and called from a Rust host with wasmtime = { version = "23.0.1", features = ["component-model"]}. No matter what path or file I try to access, I get a "file does not exist".

My question is: Do I have to take into account anything special (maybe in the Rust host) in order to access the host's filesystem apart from mounting the folder of interest in the Guest?

My build commands are as follows:

# generate bindings
wit-bindgen tiny-go ./read.wit --world reader --out-dir=gen

# build for wasi target
tinygo build -o read.wasm -target=wasi read.go

# embed WIT and componentize
export COMPONENT_ADAPTER_REACTOR=./wasi_snapshot_preview1.reactor.wasm
wasm-tools component embed --world reader ./read.wit read.wasm -o read.embed.wasm
wasm-tools component new -o read.component.wasm --adapt wasi_snapshot_preview1="$COMPONENT_ADAPTER_REACTOR" read.embed.wasm

# virtualize wasi and mount filesystem
wasi-virt read.component.wasm --mount /=/path-to-host-folder -o virt.read.wasm

# execute virt.read.wasm in Rust host
# where the Go (now WASM) code does a `dir, err := os.Open("/")`

Pat Hickey (Aug 06 2024 at 17:25):

there shouldn't be anything required in the host for wasi-virt to provide the filesystem, no - this is probably a wasi-virt bug since replacing that filesystem with wasmtime run --dir... works

Pat Hickey (Aug 06 2024 at 17:33):

one thing you can perhaps do to debug this is set WASMTIME_DEBUG=wasmtime_wasi=trace and see if any filesystem ops are hitting the host. afaik theres no way to get a log of what operations are hitting virt but its been a minute since i played with wasi-virt

Christoph Brewing (Aug 07 2024 at 07:45):

Thank you very much for your guidance @Pat Hickey. I was pretty lost with that topic.

WASMTIME_DEBUG=wasmtime_wasi=trace does not show any additional output on my screen, neither in debug nor in release, see also this build script.

I will open an issue at WASI-virt then.

In order to be better able to discuss this topic (also in WASI-Virt), I have prepared a somewhat minimal example:

1. Component creation
2. Component consumption

Christoph Brewing (Aug 07 2024 at 09:56):

In WASI-Virt, there just came up the argument that the host has to grant access, indeed. I just did not find any relevant configuration parameters for wasmtime, yet.

Christoph Brewing (Aug 07 2024 at 12:27):

Based on the explanation and some further research regarding the runtime, wasmtime, I found preopened_dir() and implemented it accordingly. Helas, I still cannot access the filesystem from the component. What I have is

Go: dir, err := os.Open("/data")
wasi-virt: wasi-virt reader.component.wasm --mount /data=./ -o reader.wasm
Rust host: let wasi = WasiCtxBuilder::new().preopened_dir("./", "/data", DirPerms::all(), FilePerms::all())?.build();

What is said about the virtualization by wasi-virt is

Virtualized files from local filesystem:

  - /data/README.md : ./README.md
  - /data/binaries : ./binaries
  - [..]
  - /data/scripts : ./scripts
  - /data/scripts/build.sh : ./scripts/build.sh

Is this setup supposed to match?

Richard Backhouse (Aug 07 2024 at 14:20):

There has been problems in the past with Tinygo and filesystem access.

The problem was with the version of the WASI SDK being used. I have been patching my tinygo runtime to use a different SDK version and was able to get filesystem access working.

I actually just tried the new Tingo v0.32.0 but the problem was still there. I guess they have not updated the version of the SDK they used.

This is the issue I opened https://github.com/bytecodealliance/wasmtime/issues/6914

Richard Backhouse (Aug 07 2024 at 15:32):

I should qualify that I mean wasi-libc version and not the wasi-sdk. version that tinygo uses.

You can see here that tinygo is still using a SDK 20 version

https://github.com/tinygo-org/tinygo/tree/release/lib

It needs to be 21 and above to ensure the explicit initialization for preopens happens

Christoph Brewing (Aug 08 2024 at 04:38):

Thank you very much for your illustration!

I am actually on tinygo version 0.33.0-dev. It still uses the same wasi-lbc. The reason for dev was that I wanted to try wasip2. I did not have problems to access the filesystem in the following way:

tinygo build -o reader.wasm -target=wasip2 read.go
wasmtime run --dir=/path-to-host-folder::/ reader.wasm /text-file.txt

I am not sure if it makes sense but maybe this is the way to go then.

Richard Backhouse (Aug 08 2024 at 09:36):

If the read.go has a main() then it will work as that triggers to call to _start and that does the preopens initialization.

The problem is only with components as _start does not get called.

Christoph Brewing (Aug 08 2024 at 09:39):

I totally agree. Unfortunately, I am up to the second use-case ... I just do not succeed in creating a successful example for the second use-case, that is: I want to expose arbitrary interfaces/functions from a (Go based) component as I am used to from my Rust toolchain - with the slight difference that my Go code needs access to the filesystem this time.

At least, I am in train to gather what I have in order to show what does not work. I will paste in the link for reference later. Perhaps, there is a workaround.

Richard Backhouse (Aug 08 2024 at 09:50):

I was able to move forward with getting components to work by patching in a newer version of wasi-libc in the tinygo I'm using.

This worked for Tinygo v0.31.1 patching in a V21 wasi-libc obtained from a downloaded wasi-sdk.

Download wasi-sdk Version 21 from https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-21
Replace the wasi-libc directory in tingo/lib with version from wasi-sdk.

• Use the wasi-sdk/share/wasi-sysroot dir.
• Rename to sysroot and just include the "include" and "lib" directory from it.
• Also delete the "wasm32-wasi-threads" from "lib"

I found the directory structure of wasi-sdk 22 onwards has changed a bit and so far I've not figured out exactly whats need copying. But then I haven't spent much time on it yet.

Christoph Brewing (Aug 08 2024 at 10:02):

Thank you, sounds like the current plan-A!

My non-working example targeting wasip2 can be found in the fourth-component.

There is another one in the second-component implementing a main().

Notification Bot (Aug 08 2024 at 10:25):

Christoph Brewing has marked this topic as resolved.

Randy Reddig (Sep 13 2024 at 21:25):

FWIW, TinyGo -target=wasip2 implements WASI 0.2 directly, and not via wasi-libc: https://github.com/tinygo-org/tinygo/tree/dev/src/internal/wasi

Ralph (Sep 14 2024 at 14:08):

wast trying this out yesterday and it worked great

Last updated: Nov 22 2024 at 16:03 UTC