Ramon Klass (Apr 19 2024 at 12:16): Ramon Klass (Apr 19 2024 at 12:16):hi @Joel Dice I just wanted to ask if there are news regarding .so files and wasi python, I'm pretty sure wasi-sdk 21 and 22 already have all changes from your fork. Componentize-py still is using your python fork from 4 months ago, are there any news since then?
Ramon Klass (Apr 19 2024 at 12:17):my use case is a rust binary that uses python internally as a scripting language, so I can't use componentize-py, and upstream cpython can't load the wheel .so files
Joel Dice (Apr 19 2024 at 14:45):Hi @Ramon Klass. You're correct that wasi-sdk 21 and 22 both have stub dlopen support which libraries like CPython can be built against. However, the stubs are trivial and just return errors unconditionally; the intention is that the stub libdl.so would be replaced with a working version at runtime (or component creation time).
I've spoken with Brett Cannon and others about what it would look like to upstream my (fairly trivial) patch to CPython to enable dlopen support, and but we didn't come to any definite conclusion. The issue is that there's currently no way to test dlopen in CPython without pulling in a dependency on either wasm-tools or componentize-py, neither of which Brett felt were appropriate. There's also the issue of messaging to users: unlike traditional platforms, WASI doesn't have a standard for dlopen in the sense of an open-world, truly dynamic, whatever-you-want, whenever-you-want runtime code loading mechanism. Instead, you need to use e.g. wasm-tools component link to bundle the libraries ahead of time and create the illusion of runtime code loading (which is really just symbolically resolving code that's already loaded). Communicating to users that they can use .so files but not the way they're used to will be a challenge.
One possibility is to add dlopen or equivalent to WASI and implement it in WASI-capable runtimes like Wasmtime. That could be a tough sell, though, given that an open-world version of dlopen (like fork) would be "magic" and fundamentally require special host powers. I.e. it couldn't be virtualized and would thus be unlike any other WASI standard. @Luke Wagner suggested that we could standardize it, but outside of WASI. I'm not sure what umbrella it would fall under, though. Core Wasm?
Anyway, to address your specific problem: I think I understand why you can't use componentize-py -- it's because you've got a Rust program that happens to use Python internally, but it's the Rust code that provides the interface to the host and outside world, correct? If that's the case, I'd suggest you use wasm-tools component link directly in combination with a libpython312.so built from my fork. That will require building your Rust code as a .so also, which is a bit arcane to do at the moment, but certainly possible (and is what componentize-py does internally).
Ramon Klass (Apr 19 2024 at 14:48):thank you for the detailed answer, is there any documentation or examples of the arcane incantations required?
Joel Dice (Apr 19 2024 at 14:48):Last updated: Nov 22 2024 at 16:03 UTC