Stream: wasi

Topic: bundling into .wasm?

Richard Zak (Aug 22 2022 at 15:44):

@Luke Wagner I understand that you know about how to bundle files into a Wasm file. I'm trying to figure out how to bundle Python scripts into a Python.wasm binary. Could you point me in the right direction?

Lann Martin (Aug 22 2022 at 15:47):

AFAIK it hasn't been implemented or even really spec'd out yet.

Lann Martin (Aug 22 2022 at 15:51):

The rough idea is to embed files into custom sections in a way that they can be losslessly converted to and from a file system tree, but I don't think we've gotten much more concrete than that

Richard Zak (Aug 22 2022 at 16:01):

I've seen wasi-vfs https://github.com/kateinoigakukun/wasi-vfs which is used with Ruby.wasm https://github.com/ruby/ruby.wasm but I haven't been able to replicate with Python. Is the proposed design for Wasm/wasi like Ruby's wasi-vfs? It's a static lib which gets compiled in, and the wasi-vfs tool is able to add files/directories.

Lann Martin (Aug 22 2022 at 16:08):

If I'm not mistaken, wasi-vfs constructs an in-memory filesystem which is copied into a normal data segment. It doesn't require any cooperation from the runtime, unlike the custom sections approach

Lann Martin (Aug 22 2022 at 16:09):

(but I haven't used it, so hopefully someone more knowledgeable than I can confirm)

Luke Wagner (Aug 22 2022 at 17:31):

I think there are two subtly-different cases here:

1. bundling "static assets" that you want to be programmatically available to the running wasm code through maybe a traditional filesystem or maybe something more explicit like, say, the asset references proposal in JS
2. bundling externally-viewed metadata (that is not accessible to the running wasm code) into a .wasm file (primarily via custom sections)

Both are valuable, but the toolchain story is quite different. If you're talking about bundling Python scripts, that sounds like case #1. What I'd like to see here is that we can leverage the component model's linking support to build a language-independent tool that can virtualize a WASI filesystem in terms of data sections (and possibly other non-filesystem interfaces, e.g., a blob store interface). The rough workflow would be that you (1) create a general Python component that imports the WASI filesystem interface to access the scripts as normal files, (2) call the virtualizer tool, passing in the Python component, the static asset directory, and lastly an /etc/fstab-like file that says "mount the static assets at this directory in the VFS" (thereby allowing multiple kinds of mounts, like Emscripten allows today). The result would be a compound component that imports the original (shareable) Python component and embeds the static assets in data sections along with (also shareable) VFS code that implements the WASI filesystem interface imported by the Python component. As an optional step (3), you could use wizer to create a post-Python-parsing snapshot, removing Python-parsing from the runtime-instantiation path.

Unfortunately, we haven't started building this virtualizer tool, though. Now that we have real component-model support in Wasmtime and producer tooling for generating and linking components wit-component+component-compose, it would make sense to start this soon as a new Bytecode Alliance tool to add to the suite.

In the meantime, wizer can be used on core wasm to achieve a similar effect today, reading all the files before the snapshot, effectively using snapshotting as the tool for putting static assets into data sections. Nick Fitzgerald is a good person to talk to about this.

Yuta Saito (Aug 24 2022 at 22:41):

@Richard Zak Hi, Richard. I'm the author of wasi-vfs, and I'm interested in your Python use case. Could you share me how you played with wasi-vfs? I'd love to help you :)

Richard Zak (Aug 24 2022 at 22:43):

I used wasi-vfs to bundle Ruby's standard scripts and a script I wanted to run so I could run it with Enarx https://github.com/enarx/enarx. I haven't been successful with wasi-vfs & Python yet, but @bstrie and I are working on it. I ran into errors when trying to get a Wasi port of Python to link with libwasi_vfs.a and haven't tried again. But I think it's the only way forward for us, at least for now.

Richard Zak (Aug 24 2022 at 22:45):

@bstrie There are two "bstrie" accounts for some reason, tagging both.

Yuta Saito (Aug 24 2022 at 22:53):

Ok, if you could report the error to the wasi-vfs repo, it would be very appreciated :)

Pete Vetere (Aug 25 2022 at 15:29):

I was able to successfully build WASI Python with the VFS. I cloned tiran's cpython fork (https://github.com/tiran/cpython.git) and added a couple of hacks, as shown in this diff (obviously, update the paths for your own environment):

https://github.com/pvetere/cpython/commit/e291f4e8a656ddf8b75445290d826e6703e5776

Then, from the repo's root, I did the following:

1. Build WASI-enabled Python:

Tools/wasm/wasm_build.py wasi compile

2. Create the runtime structure in ./pyroot:

cd builddir/wasi && make install && cd ../..

3. Pack the runtime libs into the Wasm module. Run the following command from the root of the repo. It may take a minute or two:

wasi-vfs pack builddir/wasi/python.wasm --mapdir lib::$(pwd)/pyroot -o builddir/wasi/python-with-vfs.wasm

4. Now you should be able to run python in Wasm with the VFS, like in the following example. The --dir . option is necessary to give the right capabilities to wasmtime, but the libs are all read from the VFS.

% wasmtime run --dir . -- builddir/wasi/python-with-vfs.wasm
Python 3.12.0a0 (heads/main-dirty:32ac98e899, Aug 25 2022, 10:59:15) [Clang 13.0.0 (https://github.com/llvm/llvm-project fd1d8c2f04dde23bee0fb3a7d069 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import json
>>>

Hope this helps!

Last updated: Dec 23 2024 at 12:05 UTC