Stream: SIG-Guest-Languages

Topic: Python subgroup


view this post on Zulip Joel Dice (Aug 31 2023 at 15:32):

The Python subgroup will be meeting at today at https://meet.jit.si/PythonComponentTooling. Agenda is here: https://hackmd.io/hazcX6vkT0e-90nqKOvEeg. It's pretty open-ended; my main goal is to decide what projects to tackle next now that componentize-py is starting to mature.

Join a WebRTC video conference powered by the Jitsi Videobridge
or

view this post on Zulip Joel Dice (Sep 14 2023 at 18:15):

The Python subgroup will be meeting at at https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09. Agenda (sparse as it is) is here: https://hackmd.io/mLLo1CduQLeGk9h05WGFAw. Feel free to add items.

or

view this post on Zulip Joel Dice (Sep 27 2023 at 22:34):

@Hood Chatham apparently all SciPy tests are passing on LFortran: https://bytecodealliance.zulipchat.com/#narrow/stream/206238-general/topic/Can.20WASM.20be.20compiled.20to.20a.20standalone.20binary.3F/near/393576378

view this post on Zulip Hood Chatham (Sep 28 2023 at 00:32):

I think if I read that comment correctly, he means that he can "fully compile several of the Fortran modules" of scipy "and all tests pass" for the modules that can be compiled. But it does sound like it's coming along very well =)

view this post on Zulip Hood Chatham (Sep 28 2023 at 00:34):

As far as I know, LFORTRAN still can't quite generate position independent code but I'm pretty sure it's due to some trivial mistake

view this post on Zulip Hood Chatham (Sep 28 2023 at 00:34):

I was able to produce PIC binaries by first asking it to emit llvm IR and then compiling that

view this post on Zulip Hood Chatham (Sep 28 2023 at 00:35):

In other news, it looks like pmatos maybe fixed the problems with wasm table access builtins and PIC:
https://github.com/llvm/llvm-project/issues/65191#issuecomment-1737244995

Example: Source file: static __externref_t table[0]; __externref_t test_builtin_wasm_table_get() { return __builtin_wasm_table_get(table, 0); } Compiling as follows works: clang -c -target wasm32 -...

view this post on Zulip Joel Dice (Sep 28 2023 at 17:02):

We'll be meeting at today. Agenda and notes here: https://hackmd.io/uhmQG41TS0mRtOTP9ikbUA. https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

or

view this post on Zulip Brett Cannon (Sep 28 2023 at 17:35):

I'll probably miss today's meeting. My update is I'm working on a script to make building CPython easier.

view this post on Zulip Brett Cannon (Oct 04 2023 at 21:37):

CPython 3.12.0 WASI build https://github.com/brettcannon/cpython-wasi-build/releases/tag/v3.12.0

Unofficial WASI builds of CPython. Contribute to brettcannon/cpython-wasi-build development by creating an account on GitHub.

view this post on Zulip Scott Waye (Oct 05 2023 at 14:56):

How is Python generating binding for WIT ?

view this post on Zulip Joel Dice (Oct 05 2023 at 15:07):

CPython currently targets WASI Preview 1 (via wasi-sdk), which is not based on WIT. If you want to target WASI Preview 2 (which is based on WIT), you can use https://github.com/bytecodealliance/componentize-py (which uses CPython but also supports targeting arbitrary WIT worlds)

Contribute to bytecodealliance/componentize-py development by creating an account on GitHub.

view this post on Zulip Scott Waye (Oct 05 2023 at 17:14):

I see, so you have a code generator in Rust but you decided not to create a backend to wit-bindgen, is that right?

view this post on Zulip Joel Dice (Oct 05 2023 at 17:21):

Correct. The main reason for that is most of the code componentize-py generates is Wasm, not Python. The intention is that part could be reused for other high-level languages.

view this post on Zulip Joel Dice (Oct 05 2023 at 17:22):

(i.e. all the lifting and lowering code is Wasm, with calls out to CPython to construct and deconstruct Python objects)

view this post on Zulip Scott Waye (Oct 05 2023 at 17:51):

Interesting, thanks. Do you know if javascript it the same by any chance?

view this post on Zulip Joel Dice (Oct 05 2023 at 17:55):

No, I think componentize-js currently generates JS code for lifting and lowering. If @Guy Bedford agrees, we could split the Wasm lift/lower generator out of componentize-py and reuse it in componentize-js. It would be a pretty big refactor, though.

view this post on Zulip Joel Dice (Oct 05 2023 at 17:58):

componentize-js is similar to componentize-py in that it has its own binding generator which is not part of wit-bindgen, if that's what you meant to ask.

view this post on Zulip Joel Dice (Oct 05 2023 at 18:00):

I think the trend is that each guest language tool for producing components will generate bindings its own way rather than have wit-bindgen do it. They'll all presumably use wit-parser and maybe share other code, but I don't think wit-bindgen will necessarily be a central repository for all bindings generators.

view this post on Zulip Scott Waye (Oct 05 2023 at 18:20):

Ok, there is a preference from the dotnet runtime team to follow a protobuf/protoc approach, like wit-bindgen. The c# approach is currently at this crossroads so I'm trying to gather opinions so thanks for the input. If anyone wants to chip in directly: https://github.com/dotnet/runtimelab/issues/2409

Hi, I started a discussion on https://bytecodealliance.zulipchat.com/#narrow/stream/394175-SIG-Guest-Languages about where to host a place where the interested parties can discuss the way forward f...

view this post on Zulip Joel Dice (Oct 12 2023 at 13:43):

Does anyone have any agenda items for today's Python subgroup meeting? If not, I was thinking of canceling it.

view this post on Zulip Joel Dice (Oct 12 2023 at 17:58):

I went ahead and cancelled it. My only update is that componentize-py now supports resources. Haven't been doing anything Python related since then.

view this post on Zulip Joel Dice (Oct 26 2023 at 16:26):

We'll be meeting at today. Agenda here: https://hackmd.io/Wu5-BHBfTdq_gri90BdRiw. Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

or

view this post on Zulip Joel Dice (Oct 26 2023 at 19:01):

...er, maybe. Having technical issues logging into Zoom under the BA account. The credentials are no longer showing up in my BA 1password account.

view this post on Zulip Joel Dice (Oct 26 2023 at 19:07):

Sorry, it might not happen after all. I'm not able to log in to the host account, and having trouble finding anyone who can help on short notice.

view this post on Zulip Brett Cannon (Oct 26 2023 at 19:15):

My update was going to be wasi.py requires a quick validation by me on macOS and then adding support to specify the command for the WASI runtime used at Python build time (wasmtime 14 moved that from a nice-to-have to a requirement).

view this post on Zulip Joel Dice (Oct 26 2023 at 19:15):

We're in now!

view this post on Zulip Joel Dice (Nov 09 2023 at 00:17):

If I'm reading this right (I'll admit I've only skimmed it), SciPy can now be built by the LLVM 17 release of Flang for Windows and pass 100% of the test suite: https://labs.quansight.org/blog/building-scipy-with-flang. Might be worth using Flang to target Wasm and see what happens.

Moving SciPy to Meson meant finding a different Fortran compiler on Windows, which was particularly tricky to pull off for conda-forge. This blog tells the story about how things looked pretty grim for the Python 3.12 release, and how things ended up working out just in the nick of time.

view this post on Zulip Joel Dice (Nov 09 2023 at 16:32):

We'll meet at today. Could be a short one if there's not a lot to discuss. Feel free to add to the agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

or

view this post on Zulip Brett Cannon (Nov 22 2023 at 17:36):

I'm assuming there's no meeting tomorrow due to US Thanksgiving?

view this post on Zulip Joel Dice (Nov 22 2023 at 17:37):

Correct; I'll cancel it.

view this post on Zulip Joel Dice (Dec 07 2023 at 16:17):

We'll be meeting at today. Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09 Agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA

or

view this post on Zulip Joel Dice (Jan 04 2024 at 16:04):

Does anyone have agenda items for today's Python meeting? If not, I'll cancel it.

view this post on Zulip Joel Dice (Jan 18 2024 at 16:13):

Anyone have agenda items for today's Python meeting?

view this post on Zulip Ralph (Jan 18 2024 at 17:40):

how great you all are?

view this post on Zulip Ralph (Jan 18 2024 at 17:41):

prolly too late now, but.....

view this post on Zulip Joel Dice (Jan 18 2024 at 19:12):

I'm going to go ahead and cancel the meeting since there are no agenda items and I enjoy cancelling meetings.

Relatedly: In the past few meetings we _didn't_ cancel, we mostly discussed the various things we _should_ do but which nobody currently has bandwidth to work on:

In short, we're stalled right now -- lots of work to do, but no bandwidth to do it. Until we move out of that state, I don't expect we'll have much to discuss.

view this post on Zulip Ralph (Jan 18 2024 at 19:34):

I have a sneaking suspicion that there will be a lull having finished the project work... while people breathe and test things out.

view this post on Zulip Ralph (Jan 18 2024 at 19:35):

I built some samples and passed them around and had some real success on jco and wasmtime and spin for example, and the more that happens the more there will be some volunteers for the next tasks, porting wheels and so forth....

view this post on Zulip Ralph (Jan 18 2024 at 19:36):

I think the ms python peeps are going to be motivated, so I hope to help enable them to realize that they can really help push things forward by helping.

view this post on Zulip Ralph (Jan 18 2024 at 19:36):

<crossed fingers here />

view this post on Zulip Ralph (Jan 18 2024 at 19:36):

you have issues for steps marked out?

view this post on Zulip Ralph (Jan 18 2024 at 19:36):

I'll try to find people!

view this post on Zulip Joel Dice (Jan 18 2024 at 20:00):

componentize-py has an up-to-date issue list, and I'm always happy to mentor new devs: https://github.com/bytecodealliance/componentize-py/issues

Otherwise, the big effort is porting packages to WASI, and I'm not sure how to mark that out; i.e. which packages to prioritize. I'm still quite new to the whole Python ecosystem, so I don't even know what the most popular packages are. I guess we could make a checklist out of the list of packages Pyodide supports (https://github.com/pyodide/pyodide/tree/main/packages), but again I'm not sure how to order it by priority.

Contribute to bytecodealliance/componentize-py development by creating an account on GitHub.

view this post on Zulip Ralph (Jan 19 2024 at 09:57):

good chance Brett will know, and the pythonistas here. I'll ask around

view this post on Zulip Brett Cannon (Jan 22 2024 at 20:02):

Ralph said:

good chance Brett will know, and the pythonistas here. I'll ask around

I do know and there's no need to ask around unless you're after conda because otherwise all roads lead to me. :sweat_smile: There aren't issues because I'm still focusing towards tier 2 support. After that is seeing if WASI-SDK 21 has what's necessary for dlopen() and figuring out how to tell the community to get the packages to work. Then we have to decide how the wheel platform tags will function (both what they specify and how to tell if a platform supports the tag), especially with wasi-libc having no forwards/backwards-compatibility. Once that's all done we can the ask to allow for uploads to PyPI and hopefully get various build backends to support WASI.

view this post on Zulip Joel Dice (Jan 22 2024 at 20:44):

wasi-sdk 21 does have shared library and dlopen support, so we can check that off the list. Even once we have all the decisions made and infrastructure in place, though, each porting each individual package will require effort. Some of them will just work with no code changes or WASI-specific hacks. As Pyodide has shown, though, many will require a lot more effort, and I expect that upstream maintainers won't want to take that on themselves.

view this post on Zulip Ralph (Jan 23 2024 at 10:16):

wasi-libc support resources will appear; it's becomign clear to a lot of people that this is a path that is needed.

view this post on Zulip Brett Cannon (Jan 23 2024 at 22:54):

Joel Dice said:

As Pyodide has shown, though, many will require a lot more effort, and I expect that upstream maintainers won't want to take that on themselves.

Two things on that. One, part of that will be due to things like no socket support, so we may not be entirely in the same spot as them depending on what WASI-libc covers. And two, I have some :carrot: ideas to entice projects into caring by solving some long-standing Python ecosystem issues that WebAssembly and WASI are uniquely positioned to help w/. :wink:

view this post on Zulip Brett Cannon (Jan 26 2024 at 21:01):

Just a quick note that wasmtime 17 (and earlier) don't pass Python's test suite under preview 2. I have filed the requisite bugs, but I wanted to save others time in case people rushed out to try preview 2 w/ CPython and noticed the test failures.

view this post on Zulip Joel Dice (Feb 01 2024 at 16:03):

I'm cancelling today's meeting (again) since many of us are attending the BA contributors' summit today.

view this post on Zulip Brett Cannon (Feb 07 2024 at 00:41):

https://github.com/python/devguide/pull/1273 is my PR documenting how to build CPython for WASI. Once this gets merged we will be down to just clean-up PRs to move WASI to tier 2 support!

📚 Documentation preview 📚: https://cpython-devguide--1273.org.readthedocs.build/

view this post on Zulip Ramon Klass (Feb 07 2024 at 00:49):

very exciting :)

view this post on Zulip Brett Cannon (Feb 08 2024 at 21:40):

https://peps.python.org/pep-0011/#tier-2

view this post on Zulip Joel Dice (Feb 15 2024 at 15:06):

For the first time in 2 months, today's meeting is not cancelled.

See you then!

or

view this post on Zulip Joel Dice (Feb 16 2024 at 18:53):

@Brett Cannon I don't know if this is useful for you at all, but here are the terrible hacks I've applied to CPython 3.12.1 to enable .so and socket support: https://github.com/dicej/cpython/commit/118e9d8eeabb6345d57295f6cec1616694eacbc2

Signed-off-by: Joel Dice <joel.dice@fermyon.com> enable testing sockets on WASI Signed-off-by: Joel Dice <joel.dice@fermyon.com>

view this post on Zulip Ralph (Feb 18 2024 at 15:59):

love me some bad, bad hacks that work

view this post on Zulip Lann Martin (Feb 18 2024 at 18:21):

let terrible_dotnet_hack = detect_dotnet(&wasm_bytes);

This "temporary" fix (in some internal code) is over a year old now!

view this post on Zulip Joel Dice (Feb 28 2024 at 17:32):

Anybody have agenda items for tomorrow's meeting? If not I'll cancel it.

BTW, a quick update on my action item from the last meeting: I spoke with Dan Gohman about improving throughput on wasi-libc PRs, and our plan is essentially this: when a PR goes idle for a while (i.e. a week with no comments) without being merged after all the feedback has been addressed, I'll make a "final request for comments", and if there are none after several days, we'll merge it. This seems to be working well so far.

Regarding the other big question about maintaining binary compatibility across wasi-libc releases: no news there, i.e. nobody has stepped up to take (or fund) that responsibility yet.

view this post on Zulip Brett Cannon (Feb 28 2024 at 19:14):

My agenda item would be https://bytecodealliance.zulipchat.com/#narrow/stream/394175-SIG-Guest-Languages/topic/All.20known.20issues.20for.20CPython.20under.20wasmtime.2018.20.26.20WASI.20SDK.2021 , but we also don't need to have a meeting just for that.

view this post on Zulip Joel Dice (Feb 28 2024 at 21:07):

@Brett Cannon If we were to meet and discuss that (which I'd be happy to do), what would the goal(s) be? Increase visibility for those issues and invite feedback? Resolve open questions related to those issues?

view this post on Zulip Ralph (Feb 28 2024 at 21:20):

I might swing by to understand the area. I need to find some help here.

view this post on Zulip Joel Dice (Feb 28 2024 at 21:30):

Sounds good; I won't cancel it, then.

view this post on Zulip Joel Dice (Feb 29 2024 at 15:30):

Meeting info for today:

or

view this post on Zulip Brett Cannon (Feb 29 2024 at 16:53):

Joel Dice said:

Brett Cannon If we were to meet and discuss that (which I'd be happy to do), what would the goal(s) be? Increase visibility for those issues and invite feedback? Resolve open questions related to those issues?

Visibility, status update as to where CPython stands for WASI 0.2, and just in case someone knows I'm overlooking something.

view this post on Zulip Joel Dice (Mar 14 2024 at 14:02):

Any objections to me cancelling today's meeting? Looks like a few of our regular attendees can't make it, and we don't have anything on the agenda anyway.

view this post on Zulip Joel Dice (Mar 27 2024 at 21:57):

Anyone have agenda items for tomorrow's meeting? If not, we can just do an asynchronous status update here.

view this post on Zulip Joel Dice (Mar 28 2024 at 15:26):

I'm going to cancel the meeting since there are no agenda items (and since I _love_ cancelling meetings).
My update:

I sync, you sync, we all sync for async! Contribute to dicej/isyswasfa development by creating an account on GitHub.
Contribute to bytecodealliance/componentize-py development by creating an account on GitHub.

view this post on Zulip Joel Dice (Apr 11 2024 at 16:00):

No agenda again this week, so I'll cancel today's meeting if there are no objections.

view this post on Zulip Joel Dice (Apr 25 2024 at 13:20):

Anyone have agenda items for today's meeting?

view this post on Zulip Joel Dice (Apr 25 2024 at 18:28):

Crickets; I'll cancel it.

view this post on Zulip Joel Dice (Jul 11 2024 at 18:17):

For the first time in a while, we're having a meeting today at (i.e. about 45 minutes from now).

or

view this post on Zulip Brett Cannon (Jul 11 2024 at 20:35):

I think my most immediate question is if I use the new wasm32-wasi-preview2 target triple (although @Joel Dice made it sound like the triple was actually wasm23-wasip2), I get sockets, which is great! But compiling for a component instead of a module; I'm not sure how that affects me. I still want a python.wasm that I can pass to e.g. wasmtime to run the compiled interpreter. Does this change? And I assume it doesn't affect wanting to statically compile .o files for things like zlib so that I can statically link it?

view this post on Zulip Joel Dice (Jul 11 2024 at 20:39):

When you target wasm32-wasip2 (which is now the official name; wasm32-wasi-preview2 is no longer used anywhere, AFAIK), python.wasm will be a component, which should run fine using Wasmtime, node+jco, browser+jco, or any other WASIp2-capable runtime. It won't run on any runtime which only supports WASIp1, though.

And, yes, statically linking .o and .a files should continue to work like it always has.

view this post on Zulip Brett Cannon (Jul 11 2024 at 20:49):

FYI the wasm32-wasi-preview2 target is specifically mentioned at https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-22 because of you :wink:

What's Changed build and test wasm32-wasi-preview2 target by @dicej in #370 doc: remember to tag wasi-libc during a release by @abrown in #374 update wasi-libc and Makefile to fix <fstream> by @di...

view this post on Zulip Joel Dice (Jul 11 2024 at 20:51):

Yeah, that PR landed before the name change, which happened later in the WASI-SDK 22 development process.

view this post on Zulip Brett Cannon (Jul 11 2024 at 20:58):

https://github.com/python/cpython/issues/121634 is tracking trying out the new target triple (after I make sure WASI SDK 22 builds fine under wasm32-wasi).

https://bytecodealliance.zulipchat.com/#narrow/stream/394175-SIG-Guest-Languages/topic/Python.20subgroup/near/450807393 Requires #120371 since WASI SDK 22 is required. But this supposedly has some ...

view this post on Zulip Joel Dice (Jul 11 2024 at 21:00):

Note that you can always support both wasm32-wasip1 and wasm32-wasip2 in CPython if desired (i.e. as separate build targets). We decided not to do that for .NET because it compounded the maintenance burden, but maybe it wouldn't be so bad for CPython?

view this post on Zulip Brett Cannon (Jul 12 2024 at 18:39):

I assume I will support both (for now) by making the target triple an option for my build script

view this post on Zulip Victor Adossi (Jul 22 2024 at 15:47):

Hey all (and maybe @Joel Dice, though I assume :palm_tree: means vacation! ) -- I have some friends over SONY Midokura over here in :flag_japan: that are trying to do python WebAssembly things, and they're doing it the hard way -- compiling MicroPython w/ wasi-sdk and trying to work out all the hard parts themselves.

I know there was a TON of hard work in this area, but before I give them the links I have to sort of show the walk through/progression of shared everything linking and the road to componentize-py/work that's been done on it since (that I didn't do!), I wanted to ask if there's a good existing resource that describes everything concisely?

I'd like to do just a little bit better than pointing them to https://www.fermyon.com/blog/introducing-componentize-py and https://github.com/bytecodealliance/componentize-py and recent PRs to upstream python, etc.

Componentize-Py is a new tool for packaging Python apps and their dependencies as WebAssembly components.
Contribute to bytecodealliance/componentize-py development by creating an account on GitHub.

view this post on Zulip Lann Martin (Jul 22 2024 at 16:02):

This is another resource: https://component-model.bytecodealliance.org/language-support/python.html

view this post on Zulip Victor Adossi (Jul 22 2024 at 16:06):

Thanks for that link, will definitely include it. They're quite technical as they've started and had some success with the MicroPython build so was hoping for just a few resources that also... throw them off the deep end, in a matter of speaking

view this post on Zulip Brett Cannon (Jul 22 2024 at 17:22):

@Victor Adossi how are they dealing with the long jump issue as my understanding is that the WASI SDK needs the exceptions proposal to land for that to work (i.e. I tried this last year :sweat_smile:)?

And yes, Joel is the best person to answer all of this. But if they are using MicroPython then how would linking come into play since I didn't think MicroPython had extension module support?

view this post on Zulip Joel Dice (Jul 22 2024 at 17:56):

Yes, I'm on vacation and typing this on my phone. Briefly: I spoke at length with a couple of people from Midokura at Wasmcon last year, and @YAMAMOTO Takashi is familiar with the work I've done to support shared libraries in wasi-sdk, so I think they already know a lot of the technical details. My understanding is that they're using Wamr, which does not yet support WASIp2 or the component model, which may help explain the approach they're currently taking.

view this post on Zulip Joel Dice (Jul 22 2024 at 17:56):

BTW, @Luke Wagner wrote this up about linking in the CM, for reference: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Linking.md

Repository for design and specification of the Component Model - WebAssembly/component-model

view this post on Zulip Victor Adossi (Jul 23 2024 at 04:58):

Hey thanks for the thoughtful responses!

@Brett Cannon I'm not sure exactly what which long jump issue this is referring to, but maybe it's the implementation done by Yamato-san here: https://github.com/WebAssembly/wasi-libc/pull/483 (this was mentioned & demo'd in a talk I saw last night!)

I was trying to discuss less linking specifically and more getting things working in python land in general. I know less about python and even less about micropython but was hoping for resources that sound like "X didn't work so then I Y'd but there was a problem with Z" kind of write-up that could be illuminating for people working with a python-flavored toolchain.

at Joel :palm_tree: (but so this doesn't ping)

Apologies! I actually saw a presentation on the work that Yamato-san did just last night :) And you're absolutely right about their approach, I need to ask more about what they're running into, maybe I'll check out the embedded WG.

I want to be clear that it wasn't quite about linking specifically so much as I wanted any more documentation on how it all sort of fits together (high level and low level), and possibly the messy kind -- I should have asked more specifically about the challenges they were running into so I could be more specific!

Add setjmp/longjump support based on Wasm EH proposal. It's provided as a separate library (libsetjmp) from libc so that runtimes w/o EH support can still load libc.so. To use this setjmp/longjmp i...

view this post on Zulip Brett Cannon (Jul 23 2024 at 18:03):

@Victor Adossi Thanks for the link! That's exactly the "long jump" issue I was referring to. Do you know if there are plans to upstream the changes to MicroPython (they have Emscripten support in the repo, so I suspect they would take WASI as well).

And https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md might be the low level dynamic linking docs you were looking for.

Conventions supporting interoperatibility between tools working with WebAssembly. - WebAssembly/tool-conventions

view this post on Zulip Victor Adossi (Jul 24 2024 at 03:57):

Hey @Brett Cannon no problem! I don't know if they plan on upstreaming, but I know they're working with MicroPython right now... I'll shoot over a question and certainly @YAMAMOTO Takashi will know way more than I do.

Thanks for that dynamic linking doc -- most of that (and how it integrates with non-emscripten LLVM WASM builds) is over my head but certainly useful for others. Reviewing the componentize-py code would probably help me understand more as well.

view this post on Zulip Joel Dice (Jul 29 2024 at 14:40):

Yeah, componentize-py needs more docs, including an ARCHITECTURE.md (hence this issue). Meanwhile, this presentation might be helpful.

This should include a CONTRIBUTING.md file, plus an ARCHITECTURE.md describing the high-level architecture of the project. In addition, each source file needs doc and implementation comments.
Introducing Componentize-Py joel.dice@fermyon.com Help Python developers participate in the Wasm component ecosystem Luke spoke about yesterday. Target arbitrary WIT worlds using Python

view this post on Zulip Brett Cannon (Aug 08 2024 at 18:21):

I'm assuming there's no meeting today since nothing has been mentioned about it?

view this post on Zulip Joel Dice (Aug 08 2024 at 19:09):

Argh, sorry I forgot about this. I'm in Germany for an offsite this week and won't be able to host. I just cancelled it (yes, 9 minutes after it was supposed to start).

view this post on Zulip Brett Cannon (Aug 15 2024 at 18:31):

Joel Dice said:

Note that you can always support both wasm32-wasip1 and wasm32-wasip2 in CPython if desired (i.e. as separate build targets). We decided not to do that for .NET because it compounded the maintenance burden, but maybe it wouldn't be so bad for CPython?

So I got wasm32-wasip2 building, but I didn't try turning any new features like sockets on either (i.e. I got the wasi.py script that drives builds for CPython to allow specifying the host's target triple). But upon reflection, supporting both wasm32-wasi and wasm32-wasip2 as separate targets is a pain as Python's tier support is like Rust (which is my doing :sweat_smile:), so it would be a new platform requiring a buildbot test machine, sponsorship, etc.

I assume a wasm32-wasi and a wasm32-wasip1 build are not interchangeable if you stick to only the parts that are in both (I know the APIs are different, so I'm talking from a feature perspective)?

view this post on Zulip Ramon Klass (Aug 15 2024 at 18:36):

I'm sure I know less about the new targets than you but when they were announced, wasip1 was called a rename of wasi, has that changed?

ref: https://blog.rust-lang.org/2024/04/09/updates-to-rusts-wasi-targets.html

Empowering everyone to build reliable and efficient software.

view this post on Zulip Joel Dice (Aug 15 2024 at 18:39):

AFAIK wasm32-wasi and wasm32-wasip1 are interchangeable in the sense they should both generate the exact same output.

view this post on Zulip Joel Dice (Aug 15 2024 at 18:40):

I.e. the latter is just a new name for the former.

view this post on Zulip Ramon Klass (Aug 15 2024 at 18:45):

while we're here, @Joel Dice would it be helpful if I try to update wasi-sdk and use Brett's build instead of maybe_make_cpython in componentize-py or do you want to do that yourself? I'd like to help

view this post on Zulip Joel Dice (Aug 15 2024 at 18:49):

@Ramon Klass This is my CPython patch -- it enables WASI sockets and shared library support. If there's a route to upstream either of those features in the near future, great; otherwise, I think we'll need to stick with my fork. The shared library support in particular is essential given how componentize-py works internally.

Signed-off-by: Joel Dice <joel.dice@fermyon.com> enable testing sockets on WASI Signed-off-by: Joel Dice <joel.dice@fermyon.com>

view this post on Zulip Ramon Klass (Aug 15 2024 at 18:50):

does the same apply to the wasi-wheels repo?

view this post on Zulip Joel Dice (Aug 15 2024 at 18:52):

Hmm, now I'm trying to remember what the cpython submodule in the wasi-wheels repo was for...

view this post on Zulip Joel Dice (Aug 15 2024 at 18:55):

looks like we use it to specify CROSS_PREFIX when building the various wheels. Maybe those builds require a libpython312.so to be present? Or maybe not? Feel free to switch it to a pre-built CPython for WASI -- maybe it will work?

view this post on Zulip Ramon Klass (Aug 15 2024 at 18:58):

ok then I'll start with wasi-wheels; are .a files from wasi-sdk23 compatible to wasi-sdk24 or should I always use the same wasi-sdk as build.py?

view this post on Zulip Joel Dice (Aug 15 2024 at 19:02):

Yes, AFAIK they should be compatible.

view this post on Zulip Ramon Klass (Aug 15 2024 at 19:14):

also @Brett Cannon I saw you want to include zlib into the build but what about other libraries? Specifically openssl, some libraries (in my case httpx) just assume that import _ssl will never fail, which is sad since you can't even import httpx because of that

view this post on Zulip Brett Cannon (Aug 15 2024 at 19:17):

Joel Dice said:

Ramon Klass This is my CPython patch -- it enables WASI sockets and shared library support. If there's a route to upstream either of those features in the near future, great; otherwise, I think we'll need to stick with my fork. The shared library support in particular is essential given how componentize-py works internally.

So that patch is half outdated, half doesn't work. :sweat_smile: The stuff to turn off the tests I already have upstream at Joel's request so that "dynamic" loading via imports work w/o patching, but the stuff that rips out the stack-related flags fails when Python is compiled under --with-pydebug which is necessary for CI testing.

I don't know about the sys/ioctl.h portion as every test in test_socket either errors out, gets skipped, or fails (lack of threading seems to be the main culprit for the failures, but I admittedly didn't dig into it).

If people want to help upstream stuff I'm happy to work w/ them.

view this post on Zulip Brett Cannon (Aug 15 2024 at 19:19):

Ramon Klass said:

also Brett Cannon I saw you want to include zlib into the build but what about other libraries? Specifically openssl, some libraries (in my case httpx) just assume that import _ssl will never fail, which is sad since you can't even import httpx because of that

I have been told OpenSSL won't really work as it requires constant time instructions for cryptography reasons and WebAssembly doesn't have that. For the rest I was hoping to get them working one by one.

view this post on Zulip Joel Dice (Aug 15 2024 at 19:19):

ssl is its own can of worms given the lack of constant time operations in Wasm and the risk of side channel attacks. I wouldn't recommend using OpenSSL on Wasm, even if it technically "works".

view this post on Zulip Ramon Klass (Aug 15 2024 at 19:25):

I see the point with "it wouldn't be secure either way", it's just a bit rough that just importing a package but never using it already kills the componentization, but I guess that's kind of the python ecosystem's fault, openssl was always optional, there should be guards around the import

view this post on Zulip Ramon Klass (Aug 15 2024 at 19:29):

just found that someone did in fact set up a build repo for openssl-wasi ;) https://github.com/jedisct1/openssl-wasm

OpenSSL 3 compiled for WebAssembly/WASI (up-to-date, maintained) - jedisct1/openssl-wasm

view this post on Zulip Ramon Klass (Aug 15 2024 at 19:34):

(why yes, I totally trust that you patch rand_lib.c without explanation)

view this post on Zulip Ramon Klass (Aug 15 2024 at 19:35):

but thank you both, a definite answer on the ssl thing is good to have even if it's wontfix with good reason :)

view this post on Zulip Joel Dice (Aug 15 2024 at 20:20):

FWIW, this is one of the reasons for the wasi:sockets/tls interface I proposed a while ago. That will probably be more useful for languages with higher-level abstractions for TLS (e.g. .NET/System.Net.Security.SslStream, Rust/native-tls, JS/node:tls), but might eventually benefit Python as well.

(Pre-)Proposal: wasi:sockets/tls joel.dice@fermyon.com

view this post on Zulip Ramon Klass (Aug 15 2024 at 20:22):

that's a promising path forward :) python could use this to implement a different version of the _ssl module once it's accepted

view this post on Zulip Ralph (Aug 16 2024 at 12:22):

Ramon Klass said:

just found that someone did in fact set up a build repo for openssl-wasi ;) https://github.com/jedisct1/openssl-wasm

this particular spike didn't get too far, and I'm not sure it's the one to think about.

view this post on Zulip Ramon Klass (Aug 16 2024 at 12:29):

yes the wording in the README is rather unclear about the security implications their patches introduce, I think putting efforts toward a wit interface that allows using host tls is the best way forward

one of the main issues I see is, people right now don't just want to have TLS inside components, people probably would expect that they can compile their C code to wasi and run it, and that C code will probably #include openssl, so would that mean wasi-sdk needs a openssl-to-wasitls bridge that can be linked?

view this post on Zulip Ralph (Aug 16 2024 at 12:30):

both approaches will be needed, but the question now is how to specify and provide resources for such a wasi:tls/ssl

view this post on Zulip Ralph (Aug 16 2024 at 12:30):

questions abound yet

view this post on Zulip Bailey Hayes (Aug 19 2024 at 21:16):

Joel Dice said:

AFAIK wasm32-wasi and wasm32-wasip1 are interchangeable in the sense they should both generate the exact same output.

A lil late, but wanted to chime in that there is a hope that we could use wasm32-wasi as wasi 1.0 in the future.

view this post on Zulip Milan (Sep 07 2024 at 20:37):

Joel Dice said:

Know where I can find the state of different packages that have wasi wheels published? Also curious if the recent LFortran wasm target improvements are helping for a path for scipy, etc: https://lfortran.org/blog/2024/05/fortran-on-web-using-lfortran/

view this post on Zulip Ramon Klass (Sep 07 2024 at 20:45):

there is a sandbox for building wasi wheels here https://github.com/dicej/wasi-wheels
it's a bit barebones and spits out zip files you need to extract into your site-pckages but all the needed parts are there, I'd also be interested in coordinated work on this :)

Python wheels built for WASI. Contribute to dicej/wasi-wheels development by creating an account on GitHub.

view this post on Zulip Ramon Klass (Sep 07 2024 at 20:47):

I have a proof of concept that turns the output into a NAME-wasi-X.Y.Z.whl that depends on NAME-X.Y.Z and only adds the additional .so files but I haven't converted all the build.shs to that yet

view this post on Zulip Joel Dice (Sep 10 2024 at 20:05):

We have a meeting scheduled for Thursday. I've still been focused on .NET stuff, so don't have much to report on the Python side. Feel free to add agenda items here.

or

view this post on Zulip Brett Cannon (Sep 11 2024 at 17:55):

My only update would be going over https://opensource.snarky.ca/Python/WASI which I will also be presenting at the plumber's summit, so I'm happy to talk about it, but also understand not having a meeting if it isn't enough.

view this post on Zulip Joel Dice (Sep 11 2024 at 18:39):

Let's go ahead and meet. I'll put that (i.e. Brent's WASI todo list) on the agenda, and I can also talk briefly about a couple of recent and in-progress contributions to componentize-py.

view this post on Zulip Brett Cannon (Sep 12 2024 at 19:10):

Is the meeting still happening? I'm in the lobby and Zoom is saying it's waiting for the host to start the meeting.

view this post on Zulip Joel Dice (Sep 12 2024 at 19:11):

oh crap, I had an alarm set, but then got caught up in a conversation

view this post on Zulip Joel Dice (Sep 12 2024 at 19:11):

will start it asap

view this post on Zulip Joel Dice (Sep 12 2024 at 19:11):

(need to dig up the password)

view this post on Zulip Joel Dice (Sep 12 2024 at 19:11):

sorry!

view this post on Zulip Joel Dice (Oct 10 2024 at 15:21):

Python meeting today at . Feel free to add items to the agenda. Zoom link: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

or

view this post on Zulip Brett Cannon (Oct 10 2024 at 17:10):

If it's just us and you don't have anything to talk about, @Joel Dice , we can cancel as beyond the 3.13.0 release I haven't had WASI time lately.

view this post on Zulip Joel Dice (Oct 10 2024 at 17:12):

I was just going to point to https://github.com/WebAssembly/component-model/issues/401 in case you or anyone else hasn't seen it yet. Otherwise, I've got no news either, so I'm fine with canceling.

At the BA summit this past weekend I discussed with a few folks about what it might look like to implement dlopen from C in the component model. What follows is a rough sketch about how this might ...

view this post on Zulip Joel Dice (Oct 10 2024 at 17:31):

I went ahead and cancelled it in favor of async discussion here as needed.

view this post on Zulip Joel Dice (Nov 13 2024 at 19:41):

We have agenda items for the meeting tomorrow (). If you're not otherwise occupied with e.g. KubeCon, please join us.

or

view this post on Zulip Brett Cannon (Nov 14 2024 at 17:06):

FYI I plan to be at the meeting but there's a slight chance I will be a little late

view this post on Zulip Joel Dice (Nov 14 2024 at 19:52):

Meeting starts in about 8 minutes: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

view this post on Zulip Joel Dice (Dec 11 2024 at 14:42):

Any agenda items for the meeting tomorrow? I don't have anything, myself. @Ben Brandt mentioned he's been working on improved dependency management, but he won't be able to make it to tomorrow's meeting; he expects to have something to present in January, though.

view this post on Zulip Ben Brandt (Dec 11 2024 at 15:03):

Yes, I can plan to present in January for sure. Progress has been a bit slow, but more because of lack of time. Brett's pointers last times were super helpful and I feel optimistic with some time over the holidays I can have something meaningful to present next month :+1:

view this post on Zulip Joel Dice (Dec 11 2024 at 15:16):

No pressure at all; enjoy the holiday season!

view this post on Zulip Brett Cannon (Dec 11 2024 at 17:28):

I don't have anything.

view this post on Zulip Joel Dice (Dec 11 2024 at 17:29):

Okay, I'll go ahead and cancel it.


Last updated: Dec 23 2024 at 12:05 UTC