Stream: SIG-Guest-Languages

Topic: Python subgroup

Joel Dice (Aug 31 2023 at 15:32):

The Python subgroup will be meeting at 2023-08-31T13:00:00-06:00 today at https://meet.jit.si/PythonComponentTooling. Agenda is here: https://hackmd.io/hazcX6vkT0e-90nqKOvEeg. It's pretty open-ended; my main goal is to decide what projects to tackle next now that componentize-py is starting to mature.

Joel Dice (Sep 14 2023 at 18:15):

The Python subgroup will be meeting at 2023-09-14T13:00:00-06:00 at https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09. Agenda (sparse as it is) is here: https://hackmd.io/mLLo1CduQLeGk9h05WGFAw. Feel free to add items.

Joel Dice (Sep 27 2023 at 22:34):

@Hood Chatham apparently all SciPy tests are passing on LFortran: https://bytecodealliance.zulipchat.com/#narrow/stream/206238-general/topic/Can.20WASM.20be.20compiled.20to.20a.20standalone.20binary.3F/near/393576378

Hood Chatham (Sep 28 2023 at 00:32):

I think if I read that comment correctly, he means that he can "fully compile several of the Fortran modules" of scipy "and all tests pass" for the modules that can be compiled. But it does sound like it's coming along very well =)

Hood Chatham (Sep 28 2023 at 00:34):

As far as I know, LFORTRAN still can't quite generate position independent code but I'm pretty sure it's due to some trivial mistake

Hood Chatham (Sep 28 2023 at 00:34):

I was able to produce PIC binaries by first asking it to emit llvm IR and then compiling that

Hood Chatham (Sep 28 2023 at 00:35):

In other news, it looks like pmatos maybe fixed the problems with wasm table access builtins and PIC:
https://github.com/llvm/llvm-project/issues/65191#issuecomment-1737244995

Joel Dice (Sep 28 2023 at 17:02):

We'll be meeting at 2023-09-28T13:00:00-06:00 today. Agenda and notes here: https://hackmd.io/uhmQG41TS0mRtOTP9ikbUA. https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Brett Cannon (Sep 28 2023 at 17:35):

I'll probably miss today's meeting. My update is I'm working on a script to make building CPython easier.

Brett Cannon (Oct 04 2023 at 21:37):

CPython 3.12.0 WASI build https://github.com/brettcannon/cpython-wasi-build/releases/tag/v3.12.0

Scott Waye (Oct 05 2023 at 14:56):

How is Python generating binding for WIT ?

Joel Dice (Oct 05 2023 at 15:07):

CPython currently targets WASI Preview 1 (via wasi-sdk), which is not based on WIT. If you want to target WASI Preview 2 (which is based on WIT), you can use https://github.com/bytecodealliance/componentize-py (which uses CPython but also supports targeting arbitrary WIT worlds)

Scott Waye (Oct 05 2023 at 17:14):

I see, so you have a code generator in Rust but you decided not to create a backend to wit-bindgen, is that right?

Joel Dice (Oct 05 2023 at 17:21):

Correct. The main reason for that is most of the code componentize-py generates is Wasm, not Python. The intention is that part could be reused for other high-level languages.

Joel Dice (Oct 05 2023 at 17:22):

(i.e. all the lifting and lowering code is Wasm, with calls out to CPython to construct and deconstruct Python objects)

Scott Waye (Oct 05 2023 at 17:51):

Interesting, thanks. Do you know if javascript it the same by any chance?

Joel Dice (Oct 05 2023 at 17:55):

No, I think componentize-js currently generates JS code for lifting and lowering. If @Guy Bedford agrees, we could split the Wasm lift/lower generator out of componentize-py and reuse it in componentize-js. It would be a pretty big refactor, though.

Joel Dice (Oct 05 2023 at 17:58):

componentize-js is similar to componentize-py in that it has its own binding generator which is not part of wit-bindgen, if that's what you meant to ask.

Joel Dice (Oct 05 2023 at 18:00):

I think the trend is that each guest language tool for producing components will generate bindings its own way rather than have wit-bindgen do it. They'll all presumably use wit-parser and maybe share other code, but I don't think wit-bindgen will necessarily be a central repository for all bindings generators.

Scott Waye (Oct 05 2023 at 18:20):

Ok, there is a preference from the dotnet runtime team to follow a protobuf/protoc approach, like wit-bindgen. The c# approach is currently at this crossroads so I'm trying to gather opinions so thanks for the input. If anyone wants to chip in directly: https://github.com/dotnet/runtimelab/issues/2409

Joel Dice (Oct 12 2023 at 13:43):

Does anyone have any agenda items for today's Python subgroup meeting? If not, I was thinking of canceling it.

Joel Dice (Oct 12 2023 at 17:58):

I went ahead and cancelled it. My only update is that componentize-py now supports resources. Haven't been doing anything Python related since then.

Joel Dice (Oct 26 2023 at 16:26):

We'll be meeting at 2023-10-26T13:00:00-06:00 today. Agenda here: https://hackmd.io/Wu5-BHBfTdq_gri90BdRiw. Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Joel Dice (Oct 26 2023 at 19:01):

...er, maybe. Having technical issues logging into Zoom under the BA account. The credentials are no longer showing up in my BA 1password account.

Joel Dice (Oct 26 2023 at 19:07):

Sorry, it might not happen after all. I'm not able to log in to the host account, and having trouble finding anyone who can help on short notice.

Brett Cannon (Oct 26 2023 at 19:15):

My update was going to be wasi.py requires a quick validation by me on macOS and then adding support to specify the command for the WASI runtime used at Python build time (wasmtime 14 moved that from a nice-to-have to a requirement).

Joel Dice (Oct 26 2023 at 19:15):

We're in now!

Joel Dice (Nov 09 2023 at 00:17):

If I'm reading this right (I'll admit I've only skimmed it), SciPy can now be built by the LLVM 17 release of Flang for Windows and pass 100% of the test suite: https://labs.quansight.org/blog/building-scipy-with-flang. Might be worth using Flang to target Wasm and see what happens.

Joel Dice (Nov 09 2023 at 16:32):

We'll meet at 2023-11-09T13:00:00-07:00 today. Could be a short one if there's not a lot to discuss. Feel free to add to the agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Brett Cannon (Nov 22 2023 at 17:36):

I'm assuming there's no meeting tomorrow due to US Thanksgiving?

Joel Dice (Nov 22 2023 at 17:37):

Correct; I'll cancel it.

Joel Dice (Dec 07 2023 at 16:17):

We'll be meeting at 2023-12-07T13:00:00-07:00 today. Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09 Agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA

Joel Dice (Jan 04 2024 at 16:04):

Does anyone have agenda items for today's Python meeting? If not, I'll cancel it.

Joel Dice (Jan 18 2024 at 16:13):

Anyone have agenda items for today's Python meeting?

Ralph (Jan 18 2024 at 17:40):

how great you all are?

Ralph (Jan 18 2024 at 17:41):

prolly too late now, but.....

Joel Dice (Jan 18 2024 at 19:12):

I'm going to go ahead and cancel the meeting since there are no agenda items and I enjoy cancelling meetings.

Relatedly: In the past few meetings we _didn't_ cancel, we mostly discussed the various things we _should_ do but which nobody currently has bandwidth to work on:

• Build and publish WASI wheels for popular packages to a public index so devs can pip install them easily
  - Porting those packages can range from pretty trivial (e.g. a package with a simple, already very portable C native extention) to extraordinarily difficult (e.g. SciPy and anything else involving Fortran).
  - This is highly parallelizable work, but also an enormous amount of effort if we want to support e.g. all the packages Pyodide supports
• Improve the componentize-py dev experience (e.g. automatically create a virtual environment, install all required packages to it, and build the component)

In short, we're stalled right now -- lots of work to do, but no bandwidth to do it. Until we move out of that state, I don't expect we'll have much to discuss.

Ralph (Jan 18 2024 at 19:34):

I have a sneaking suspicion that there will be a lull having finished the project work... while people breathe and test things out.

Ralph (Jan 18 2024 at 19:35):

I built some samples and passed them around and had some real success on jco and wasmtime and spin for example, and the more that happens the more there will be some volunteers for the next tasks, porting wheels and so forth....

Ralph (Jan 18 2024 at 19:36):

I think the ms python peeps are going to be motivated, so I hope to help enable them to realize that they can really help push things forward by helping.

Ralph (Jan 18 2024 at 19:36):

<crossed fingers here />

Ralph (Jan 18 2024 at 19:36):

you have issues for steps marked out?

Ralph (Jan 18 2024 at 19:36):

I'll try to find people!

Joel Dice (Jan 18 2024 at 20:00):

componentize-py has an up-to-date issue list, and I'm always happy to mentor new devs: https://github.com/bytecodealliance/componentize-py/issues

Otherwise, the big effort is porting packages to WASI, and I'm not sure how to mark that out; i.e. which packages to prioritize. I'm still quite new to the whole Python ecosystem, so I don't even know what the most popular packages are. I guess we could make a checklist out of the list of packages Pyodide supports (https://github.com/pyodide/pyodide/tree/main/packages), but again I'm not sure how to order it by priority.

Ralph (Jan 19 2024 at 09:57):

good chance Brett will know, and the pythonistas here. I'll ask around

Brett Cannon (Jan 22 2024 at 20:02):

Ralph said:

good chance Brett will know, and the pythonistas here. I'll ask around

I do know and there's no need to ask around unless you're after conda because otherwise all roads lead to me. :sweat_smile: There aren't issues because I'm still focusing towards tier 2 support. After that is seeing if WASI-SDK 21 has what's necessary for dlopen() and figuring out how to tell the community to get the packages to work. Then we have to decide how the wheel platform tags will function (both what they specify and how to tell if a platform supports the tag), especially with wasi-libc having no forwards/backwards-compatibility. Once that's all done we can the ask to allow for uploads to PyPI and hopefully get various build backends to support WASI.

Joel Dice (Jan 22 2024 at 20:44):

wasi-sdk 21 does have shared library and dlopen support, so we can check that off the list. Even once we have all the decisions made and infrastructure in place, though, each porting each individual package will require effort. Some of them will just work with no code changes or WASI-specific hacks. As Pyodide has shown, though, many will require a lot more effort, and I expect that upstream maintainers won't want to take that on themselves.

Ralph (Jan 23 2024 at 10:16):

wasi-libc support resources will appear; it's becomign clear to a lot of people that this is a path that is needed.

Brett Cannon (Jan 23 2024 at 22:54):

Joel Dice said:

As Pyodide has shown, though, many will require a lot more effort, and I expect that upstream maintainers won't want to take that on themselves.

Two things on that. One, part of that will be due to things like no socket support, so we may not be entirely in the same spot as them depending on what WASI-libc covers. And two, I have some :carrot: ideas to entice projects into caring by solving some long-standing Python ecosystem issues that WebAssembly and WASI are uniquely positioned to help w/. :wink:

Brett Cannon (Jan 26 2024 at 21:01):

Just a quick note that wasmtime 17 (and earlier) don't pass Python's test suite under preview 2. I have filed the requisite bugs, but I wanted to save others time in case people rushed out to try preview 2 w/ CPython and noticed the test failures.

Joel Dice (Feb 01 2024 at 16:03):

I'm cancelling today's meeting (again) since many of us are attending the BA contributors' summit today.

Brett Cannon (Feb 07 2024 at 00:41):

https://github.com/python/devguide/pull/1273 is my PR documenting how to build CPython for WASI. Once this gets merged we will be down to just clean-up PRs to move WASI to tier 2 support!

Ramon Klass (Feb 07 2024 at 00:49):

very exciting :)

Brett Cannon (Feb 08 2024 at 21:40):

https://peps.python.org/pep-0011/#tier-2

Joel Dice (Feb 15 2024 at 15:06):

For the first time in 2 months, today's meeting is not cancelled.

• Time: 2024-02-15T13:00:00-07:00
• Agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
• Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

See you then!

Joel Dice (Feb 16 2024 at 18:53):

@Brett Cannon I don't know if this is useful for you at all, but here are the terrible hacks I've applied to CPython 3.12.1 to enable .so and socket support: https://github.com/dicej/cpython/commit/118e9d8eeabb6345d57295f6cec1616694eacbc2

Ralph (Feb 18 2024 at 15:59):

love me some bad, bad hacks that work

Lann Martin (Feb 18 2024 at 18:21):

let terrible_dotnet_hack = detect_dotnet(&wasm_bytes);

This "temporary" fix (in some internal code) is over a year old now!

Joel Dice (Feb 28 2024 at 17:32):

Anybody have agenda items for tomorrow's meeting? If not I'll cancel it.

BTW, a quick update on my action item from the last meeting: I spoke with Dan Gohman about improving throughput on wasi-libc PRs, and our plan is essentially this: when a PR goes idle for a while (i.e. a week with no comments) without being merged after all the feedback has been addressed, I'll make a "final request for comments", and if there are none after several days, we'll merge it. This seems to be working well so far.

Regarding the other big question about maintaining binary compatibility across wasi-libc releases: no news there, i.e. nobody has stepped up to take (or fund) that responsibility yet.

Brett Cannon (Feb 28 2024 at 19:14):

My agenda item would be https://bytecodealliance.zulipchat.com/#narrow/stream/394175-SIG-Guest-Languages/topic/All.20known.20issues.20for.20CPython.20under.20wasmtime.2018.20.26.20WASI.20SDK.2021 , but we also don't need to have a meeting just for that.

Joel Dice (Feb 28 2024 at 21:07):

@Brett Cannon If we were to meet and discuss that (which I'd be happy to do), what would the goal(s) be? Increase visibility for those issues and invite feedback? Resolve open questions related to those issues?

Ralph (Feb 28 2024 at 21:20):

I might swing by to understand the area. I need to find some help here.

Joel Dice (Feb 28 2024 at 21:30):

Sounds good; I won't cancel it, then.

Joel Dice (Feb 29 2024 at 15:30):

Meeting info for today:

• Time: 2024-02-29T13:00:00-07:00
• Agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
• Zoom: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Brett Cannon (Feb 29 2024 at 16:53):

Joel Dice said:

Brett Cannon If we were to meet and discuss that (which I'd be happy to do), what would the goal(s) be? Increase visibility for those issues and invite feedback? Resolve open questions related to those issues?

Visibility, status update as to where CPython stands for WASI 0.2, and just in case someone knows I'm overlooking something.

Joel Dice (Mar 14 2024 at 14:02):

Any objections to me cancelling today's meeting? Looks like a few of our regular attendees can't make it, and we don't have anything on the agenda anyway.

Joel Dice (Mar 27 2024 at 21:57):

Anyone have agenda items for tomorrow's meeting? If not, we can just do an asynchronous status update here.

Joel Dice (Mar 28 2024 at 15:26):

I'm going to cancel the meeting since there are no agenda items (and since I _love_ cancelling meetings).
My update:

• Released componentize-py 0.13, which includes isyswasfa support (which I plan to replace with proper CM async support once matching support is available in wasm-tools, wasmtime, etc.), as well as a --stub-wasi option for eliminating any WASI imports (see example).
• Tried to help @Kyle Gray get Pandas working inside a component; failed due to trouble with Cython-generated C++ code, which we had to hand-edit to make compatible with -fno-exceptions but still had issues which might be cross-compiling related: https://bytecodealliance.zulipchat.com/#narrow/stream/219900-wasi/topic/Attempting.20to.20use.20pandas.20with.20componentize-py. Perhaps someone with more Cython experience can help? Might also be worth looking at what Pyodide does to support Pandas.

Joel Dice (Apr 11 2024 at 16:00):

No agenda again this week, so I'll cancel today's meeting if there are no objections.

Joel Dice (Apr 25 2024 at 13:20):

Anyone have agenda items for today's meeting?

Joel Dice (Apr 25 2024 at 18:28):

Crickets; I'll cancel it.

Joel Dice (Jul 11 2024 at 18:17):

For the first time in a while, we're having a meeting today at 2024-07-11T13:00:00-06:00 (i.e. about 45 minutes from now).

• Link: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09
• Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA

Brett Cannon (Jul 11 2024 at 20:35):

I think my most immediate question is if I use the new wasm32-wasi-preview2 target triple (although @Joel Dice made it sound like the triple was actually wasm23-wasip2), I get sockets, which is great! But compiling for a component instead of a module; I'm not sure how that affects me. I still want a python.wasm that I can pass to e.g. wasmtime to run the compiled interpreter. Does this change? And I assume it doesn't affect wanting to statically compile .o files for things like zlib so that I can statically link it?

Joel Dice (Jul 11 2024 at 20:39):

When you target wasm32-wasip2 (which is now the official name; wasm32-wasi-preview2 is no longer used anywhere, AFAIK), python.wasm will be a component, which should run fine using Wasmtime, node+jco, browser+jco, or any other WASIp2-capable runtime. It won't run on any runtime which only supports WASIp1, though.

And, yes, statically linking .o and .a files should continue to work like it always has.

Brett Cannon (Jul 11 2024 at 20:49):

FYI the wasm32-wasi-preview2 target is specifically mentioned at https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-22 because of you :wink:

Joel Dice (Jul 11 2024 at 20:51):

Yeah, that PR landed before the name change, which happened later in the WASI-SDK 22 development process.

Brett Cannon (Jul 11 2024 at 20:58):

https://github.com/python/cpython/issues/121634 is tracking trying out the new target triple (after I make sure WASI SDK 22 builds fine under wasm32-wasi).

Joel Dice (Jul 11 2024 at 21:00):

Note that you can always support both wasm32-wasip1 and wasm32-wasip2 in CPython if desired (i.e. as separate build targets). We decided not to do that for .NET because it compounded the maintenance burden, but maybe it wouldn't be so bad for CPython?

Brett Cannon (Jul 12 2024 at 18:39):

I assume I will support both (for now) by making the target triple an option for my build script

Victor Adossi (Jul 22 2024 at 15:47):

Hey all (and maybe @Joel Dice, though I assume :palm_tree: means vacation! ) -- I have some friends over SONY Midokura over here in :flag_japan: that are trying to do python WebAssembly things, and they're doing it the hard way -- compiling MicroPython w/ wasi-sdk and trying to work out all the hard parts themselves.

I know there was a TON of hard work in this area, but before I give them the links I have to sort of show the walk through/progression of shared everything linking and the road to componentize-py/work that's been done on it since (that I didn't do!), I wanted to ask if there's a good existing resource that describes everything concisely?

I'd like to do just a little bit better than pointing them to https://www.fermyon.com/blog/introducing-componentize-py and https://github.com/bytecodealliance/componentize-py and recent PRs to upstream python, etc.

Lann Martin (Jul 22 2024 at 16:02):

This is another resource: https://component-model.bytecodealliance.org/language-support/python.html

Victor Adossi (Jul 22 2024 at 16:06):

Thanks for that link, will definitely include it. They're quite technical as they've started and had some success with the MicroPython build so was hoping for just a few resources that also... throw them off the deep end, in a matter of speaking

Brett Cannon (Jul 22 2024 at 17:22):

@Victor Adossi how are they dealing with the long jump issue as my understanding is that the WASI SDK needs the exceptions proposal to land for that to work (i.e. I tried this last year :sweat_smile:)?

And yes, Joel is the best person to answer all of this. But if they are using MicroPython then how would linking come into play since I didn't think MicroPython had extension module support?

Joel Dice (Jul 22 2024 at 17:56):

Yes, I'm on vacation and typing this on my phone. Briefly: I spoke at length with a couple of people from Midokura at Wasmcon last year, and @YAMAMOTO Takashi is familiar with the work I've done to support shared libraries in wasi-sdk, so I think they already know a lot of the technical details. My understanding is that they're using Wamr, which does not yet support WASIp2 or the component model, which may help explain the approach they're currently taking.

Joel Dice (Jul 22 2024 at 17:56):

BTW, @Luke Wagner wrote this up about linking in the CM, for reference: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Linking.md

Victor Adossi (Jul 23 2024 at 04:58):

Hey thanks for the thoughtful responses!

@Brett Cannon I'm not sure exactly what which long jump issue this is referring to, but maybe it's the implementation done by Yamato-san here: https://github.com/WebAssembly/wasi-libc/pull/483 (this was mentioned & demo'd in a talk I saw last night!)

I was trying to discuss less linking specifically and more getting things working in python land in general. I know less about python and even less about micropython but was hoping for resources that sound like "X didn't work so then I Y'd but there was a problem with Z" kind of write-up that could be illuminating for people working with a python-flavored toolchain.

at Joel :palm_tree: (but so this doesn't ping)

Apologies! I actually saw a presentation on the work that Yamato-san did just last night :) And you're absolutely right about their approach, I need to ask more about what they're running into, maybe I'll check out the embedded WG.

I want to be clear that it wasn't quite about linking specifically so much as I wanted any more documentation on how it all sort of fits together (high level and low level), and possibly the messy kind -- I should have asked more specifically about the challenges they were running into so I could be more specific!

Brett Cannon (Jul 23 2024 at 18:03):

@Victor Adossi Thanks for the link! That's exactly the "long jump" issue I was referring to. Do you know if there are plans to upstream the changes to MicroPython (they have Emscripten support in the repo, so I suspect they would take WASI as well).

And https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md might be the low level dynamic linking docs you were looking for.

Victor Adossi (Jul 24 2024 at 03:57):

Hey @Brett Cannon no problem! I don't know if they plan on upstreaming, but I know they're working with MicroPython right now... I'll shoot over a question and certainly @YAMAMOTO Takashi will know way more than I do.

Thanks for that dynamic linking doc -- most of that (and how it integrates with non-emscripten LLVM WASM builds) is over my head but certainly useful for others. Reviewing the componentize-py code would probably help me understand more as well.

Joel Dice (Jul 29 2024 at 14:40):

Yeah, componentize-py needs more docs, including an ARCHITECTURE.md (hence this issue). Meanwhile, this presentation might be helpful.

Brett Cannon (Aug 08 2024 at 18:21):

I'm assuming there's no meeting today since nothing has been mentioned about it?

Joel Dice (Aug 08 2024 at 19:09):

Argh, sorry I forgot about this. I'm in Germany for an offsite this week and won't be able to host. I just cancelled it (yes, 9 minutes after it was supposed to start).

Brett Cannon (Aug 15 2024 at 18:31):

Joel Dice said:

Note that you can always support both wasm32-wasip1 and wasm32-wasip2 in CPython if desired (i.e. as separate build targets). We decided not to do that for .NET because it compounded the maintenance burden, but maybe it wouldn't be so bad for CPython?

So I got wasm32-wasip2 building, but I didn't try turning any new features like sockets on either (i.e. I got the wasi.py script that drives builds for CPython to allow specifying the host's target triple). But upon reflection, supporting both wasm32-wasi and wasm32-wasip2 as separate targets is a pain as Python's tier support is like Rust (which is my doing :sweat_smile:), so it would be a new platform requiring a buildbot test machine, sponsorship, etc.

I assume a wasm32-wasi and a wasm32-wasip1 build are not interchangeable if you stick to only the parts that are in both (I know the APIs are different, so I'm talking from a feature perspective)?

Ramon Klass (Aug 15 2024 at 18:36):

I'm sure I know less about the new targets than you but when they were announced, wasip1 was called a rename of wasi, has that changed?

ref: https://blog.rust-lang.org/2024/04/09/updates-to-rusts-wasi-targets.html

Joel Dice (Aug 15 2024 at 18:39):

AFAIK wasm32-wasi and wasm32-wasip1 are interchangeable in the sense they should both generate the exact same output.

Joel Dice (Aug 15 2024 at 18:40):

I.e. the latter is just a new name for the former.

Ramon Klass (Aug 15 2024 at 18:45):

while we're here, @Joel Dice would it be helpful if I try to update wasi-sdk and use Brett's build instead of maybe_make_cpython in componentize-py or do you want to do that yourself? I'd like to help

Joel Dice (Aug 15 2024 at 18:49):

@Ramon Klass This is my CPython patch -- it enables WASI sockets and shared library support. If there's a route to upstream either of those features in the near future, great; otherwise, I think we'll need to stick with my fork. The shared library support in particular is essential given how componentize-py works internally.

Ramon Klass (Aug 15 2024 at 18:50):

does the same apply to the wasi-wheels repo?

Joel Dice (Aug 15 2024 at 18:52):

Hmm, now I'm trying to remember what the cpython submodule in the wasi-wheels repo was for...

Joel Dice (Aug 15 2024 at 18:55):

looks like we use it to specify CROSS_PREFIX when building the various wheels. Maybe those builds require a libpython312.so to be present? Or maybe not? Feel free to switch it to a pre-built CPython for WASI -- maybe it will work?

Ramon Klass (Aug 15 2024 at 18:58):

ok then I'll start with wasi-wheels; are .a files from wasi-sdk23 compatible to wasi-sdk24 or should I always use the same wasi-sdk as build.py?

Joel Dice (Aug 15 2024 at 19:02):

Yes, AFAIK they should be compatible.

Ramon Klass (Aug 15 2024 at 19:14):

also @Brett Cannon I saw you want to include zlib into the build but what about other libraries? Specifically openssl, some libraries (in my case httpx) just assume that import _ssl will never fail, which is sad since you can't even import httpx because of that

Brett Cannon (Aug 15 2024 at 19:17):

Joel Dice said:

Ramon Klass This is my CPython patch -- it enables WASI sockets and shared library support. If there's a route to upstream either of those features in the near future, great; otherwise, I think we'll need to stick with my fork. The shared library support in particular is essential given how componentize-py works internally.

So that patch is half outdated, half doesn't work. :sweat_smile: The stuff to turn off the tests I already have upstream at Joel's request so that "dynamic" loading via imports work w/o patching, but the stuff that rips out the stack-related flags fails when Python is compiled under --with-pydebug which is necessary for CI testing.

I don't know about the sys/ioctl.h portion as every test in test_socket either errors out, gets skipped, or fails (lack of threading seems to be the main culprit for the failures, but I admittedly didn't dig into it).

If people want to help upstream stuff I'm happy to work w/ them.

Brett Cannon (Aug 15 2024 at 19:19):

Ramon Klass said:

also Brett Cannon I saw you want to include zlib into the build but what about other libraries? Specifically openssl, some libraries (in my case httpx) just assume that import _ssl will never fail, which is sad since you can't even import httpx because of that

I have been told OpenSSL won't really work as it requires constant time instructions for cryptography reasons and WebAssembly doesn't have that. For the rest I was hoping to get them working one by one.

Joel Dice (Aug 15 2024 at 19:19):

ssl is its own can of worms given the lack of constant time operations in Wasm and the risk of side channel attacks. I wouldn't recommend using OpenSSL on Wasm, even if it technically "works".

Ramon Klass (Aug 15 2024 at 19:25):

I see the point with "it wouldn't be secure either way", it's just a bit rough that just importing a package but never using it already kills the componentization, but I guess that's kind of the python ecosystem's fault, openssl was always optional, there should be guards around the import

Ramon Klass (Aug 15 2024 at 19:29):

just found that someone did in fact set up a build repo for openssl-wasi ;) https://github.com/jedisct1/openssl-wasm

Ramon Klass (Aug 15 2024 at 19:34):

(why yes, I totally trust that you patch rand_lib.c without explanation)

Ramon Klass (Aug 15 2024 at 19:35):

but thank you both, a definite answer on the ssl thing is good to have even if it's wontfix with good reason :)

Joel Dice (Aug 15 2024 at 20:20):

FWIW, this is one of the reasons for the wasi:sockets/tls interface I proposed a while ago. That will probably be more useful for languages with higher-level abstractions for TLS (e.g. .NET/System.Net.Security.SslStream, Rust/native-tls, JS/node:tls), but might eventually benefit Python as well.

Ramon Klass (Aug 15 2024 at 20:22):

that's a promising path forward :) python could use this to implement a different version of the _ssl module once it's accepted

Ralph (Aug 16 2024 at 12:22):

Ramon Klass said:

just found that someone did in fact set up a build repo for openssl-wasi ;) https://github.com/jedisct1/openssl-wasm

this particular spike didn't get too far, and I'm not sure it's the one to think about.

Ramon Klass (Aug 16 2024 at 12:29):

yes the wording in the README is rather unclear about the security implications their patches introduce, I think putting efforts toward a wit interface that allows using host tls is the best way forward

one of the main issues I see is, people right now don't just want to have TLS inside components, people probably would expect that they can compile their C code to wasi and run it, and that C code will probably #include openssl, so would that mean wasi-sdk needs a openssl-to-wasitls bridge that can be linked?

Ralph (Aug 16 2024 at 12:30):

both approaches will be needed, but the question now is how to specify and provide resources for such a wasi:tls/ssl

Ralph (Aug 16 2024 at 12:30):

questions abound yet

Bailey Hayes (Aug 19 2024 at 21:16):

Joel Dice said:

AFAIK wasm32-wasi and wasm32-wasip1 are interchangeable in the sense they should both generate the exact same output.

A lil late, but wanted to chime in that there is a hope that we could use wasm32-wasi as wasi 1.0 in the future.

Milan (Sep 07 2024 at 20:37):

Joel Dice said:

• Build and publish WASI wheels for popular packages to a public index so devs can pip install them easily
  - Porting those packages can range from pretty trivial (e.g. a package with a simple, already very portable C native extention) to extraordinarily difficult (e.g. SciPy and anything else involving Fortran).

Know where I can find the state of different packages that have wasi wheels published? Also curious if the recent LFortran wasm target improvements are helping for a path for scipy, etc: https://lfortran.org/blog/2024/05/fortran-on-web-using-lfortran/

Ramon Klass (Sep 07 2024 at 20:45):

there is a sandbox for building wasi wheels here https://github.com/dicej/wasi-wheels
it's a bit barebones and spits out zip files you need to extract into your site-pckages but all the needed parts are there, I'd also be interested in coordinated work on this :)

Ramon Klass (Sep 07 2024 at 20:47):

I have a proof of concept that turns the output into a NAME-wasi-X.Y.Z.whl that depends on NAME-X.Y.Z and only adds the additional .so files but I haven't converted all the build.shs to that yet

Joel Dice (Sep 10 2024 at 20:05):

We have a meeting scheduled for Thursday. I've still been focused on .NET stuff, so don't have much to report on the Python side. Feel free to add agenda items here.

Brett Cannon (Sep 11 2024 at 17:55):

My only update would be going over https://opensource.snarky.ca/Python/WASI which I will also be presenting at the plumber's summit, so I'm happy to talk about it, but also understand not having a meeting if it isn't enough.

Joel Dice (Sep 11 2024 at 18:39):

Let's go ahead and meet. I'll put that (i.e. Brent's WASI todo list) on the agenda, and I can also talk briefly about a couple of recent and in-progress contributions to componentize-py.

Brett Cannon (Sep 12 2024 at 19:10):

Is the meeting still happening? I'm in the lobby and Zoom is saying it's waiting for the host to start the meeting.

Joel Dice (Sep 12 2024 at 19:11):

oh crap, I had an alarm set, but then got caught up in a conversation

Joel Dice (Sep 12 2024 at 19:11):

will start it asap

Joel Dice (Sep 12 2024 at 19:11):

(need to dig up the password)

Joel Dice (Sep 12 2024 at 19:11):

sorry!

Joel Dice (Oct 10 2024 at 15:21):

Python meeting today at 2024-10-10T13:00:00-06:00 . Feel free to add items to the agenda. Zoom link: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Brett Cannon (Oct 10 2024 at 17:10):

If it's just us and you don't have anything to talk about, @Joel Dice , we can cancel as beyond the 3.13.0 release I haven't had WASI time lately.

Joel Dice (Oct 10 2024 at 17:12):

I was just going to point to https://github.com/WebAssembly/component-model/issues/401 in case you or anyone else hasn't seen it yet. Otherwise, I've got no news either, so I'm fine with canceling.

Joel Dice (Oct 10 2024 at 17:31):

I went ahead and cancelled it in favor of async discussion here as needed.

Joel Dice (Nov 13 2024 at 19:41):

We have agenda items for the meeting tomorrow (2024-11-14T13:00:00-07:00). If you're not otherwise occupied with e.g. KubeCon, please join us.

Brett Cannon (Nov 14 2024 at 17:06):

FYI I plan to be at the meeting but there's a slight chance I will be a little late

Joel Dice (Nov 14 2024 at 19:52):

Meeting starts in about 8 minutes: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Joel Dice (Dec 11 2024 at 14:42):

Any agenda items for the meeting tomorrow? I don't have anything, myself. @Ben Brandt mentioned he's been working on improved dependency management, but he won't be able to make it to tomorrow's meeting; he expects to have something to present in January, though.

Ben Brandt (Dec 11 2024 at 15:03):

Yes, I can plan to present in January for sure. Progress has been a bit slow, but more because of lack of time. Brett's pointers last times were super helpful and I feel optimistic with some time over the holidays I can have something meaningful to present next month :+1:

Joel Dice (Dec 11 2024 at 15:16):

No pressure at all; enjoy the holiday season!

Brett Cannon (Dec 11 2024 at 17:28):

I don't have anything.

Joel Dice (Dec 11 2024 at 17:29):

Okay, I'll go ahead and cancel it.

Joel Dice (Jan 09 2025 at 19:09):

Meeting today at 2025-01-09T13:00:00-07:00 (i.e. in about an hour).
Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom link: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Ben Brandt (Jan 16 2025 at 22:20):

I managed to get cpython 3.13 building with the wasi script! :tada:
https://github.com/benbrandt/wasi-wheels/pull/28

With host triple wasm32-wasip2.

It doesn't work for 3.12 because it doesn't have the script, but I can just call 2 different functions.

Still using a fork to enable the --enable-wasm-dynamic-linking flag though.

Anyway with that I will go to bed, and pick up moving the pydantic builds happening in CI tomorrow hopefully. Thanks for all of the help/work @Brett Cannon

Ralph (Jan 17 2025 at 13:23):

indeed, thanks @Brett Cannon !!!!

Ben Brandt (Jan 20 2025 at 21:24):

Ok, I have the builds working with (I think) the correct platform tag and building in CI via workflow dispatch. I'm slowly building them all.

Would you be able to look at one of them (like this one: https://github.com/benbrandt/wasi-wheels/releases/tag/pydantic-core%2Fv2.20.0) and let me know if it looks right?

If so, we can start using one of these internally for more testing as well.

Ben Brandt (Jan 20 2025 at 21:26):

If this all looks correct, I'll then work on generating the index from the releases.
And then I can tackle a second dependency (most likely numpy or some other one that isn't maturin based)

Brett Cannon (Jan 21 2025 at 18:51):

Ben Brandt said:

Would you be able to look at one of them

What are you specifically wanting looked at? The file name looks good. Basically if pip will install them they look good. :wink:

Brett Cannon (Jan 21 2025 at 18:52):

Ben Brandt said:

It doesn't work for 3.12 because it doesn't have the script

If you're talking about wasi.py, you can probably just copy it from main from the CPython repo and use it (you might need to copy one or two other files as well like the site config, but that's more of a WASI SDK thing).

Ben Brandt (Jan 22 2025 at 08:15):

ok great, yes it all seems tobe working on our end. thanks!

Ben Brandt (Feb 01 2025 at 07:35):

Ok I have a basic index being generated! https://benbrandt.github.io/wasi-wheels/

Need to do some more testing with pip to make sure it all works, but making progress!

Ben Brandt (Feb 10 2025 at 19:54):

Are people still meeting this week? Happy to demo my current progress and get feedback.
I have the builds + index generation fully automated in CI at this point. I still only have pydantic-core, but when I have some time I will work on adding some of the existing wheels from Joel's repo

Joel Dice (Feb 10 2025 at 19:55):

Yeah, I'd love to see a demo!

Brett Cannon (Feb 11 2025 at 19:07):

I'm still up for meeting.

Ralph (Feb 12 2025 at 14:02):

is it on? I'd love to see as well if it didn't happen yesterday :-)

Ramon Klass (Feb 12 2025 at 14:03):

I think the meetings are always on Thursdays, so it should be tomorrow

Joel Dice (Feb 12 2025 at 14:55):

Yup: 2025-02-13T13:00:00-07:00
https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09
https://hackmd.io/3ZktfALiSZWn41ad7y3gnA

Brett Cannon (Mar 05 2025 at 18:10):

FYI I'm on vacation next week, so I will most likely miss the meeting.

Ben Brandt (Mar 05 2025 at 20:42):

I will be traveling for a conference. I might be able to make it... but probably best to count me out to be safe

Joel Dice (Mar 05 2025 at 20:44):

I'll probably go ahead and cancel it. We can do a one-off meeting later when everybody's available if we have things to discuss.

Ben Brandt (Apr 08 2025 at 12:22):

Are we meeting this week?
If so, I'd have the following items:

1. Where / what to document around wasi-sdk versions for each Python version, and what our decision is here
2. If it makes sense to target wasi-wheels as a more "official" project and what it would take to get there?
3. Concrete detail: noticing some (surprising) large wasm sizes from componentize-py, and curious to know where this is coming from

Can also wait until next month if people don't have time this week.

Joel Dice (Apr 08 2025 at 16:43):

I'm in for this week. Regarding binary sizes, I posted this elsewhere a while ago, in case it's useful:

I did some research to determine a size breakdown for a "hello world" wasi-http component generated by componentize-py.
The TL;DR is that the stripped binary is about 18MB, 12MB of which is the data section with the pre-init memory snapshot, and the rest of which are the various modules, the largest of which is libpython312.so at about 4MB.
A stock, stripped libpython312.so is about 8MB, 4MB of which is data section(s) which get replaced with the memory snapshot when we pre-init.
The upshot is that we could save about 6MB per component by importing the modules it needs, but we'd still be looking at 12MB minimum component sizes. To reduce sizes further, we'd need some way to package the .py, .pyc, and related standard library files into a composable VFS component we could publish to a registry and reuse. And we'd also need a standard way to export a pre-init function that can be run by the host (e.g. at deployment time) to minimize cold-start times since we wouldn't be doing pre-init at build time anymore.

The numbers change when you start adding in large third-party dependencies, of course. And e.g. numpy brings its own .so modules which could be reused.

BTW, the stripped Python component is 6MB (down from 18MB) when using zstd

Brett Cannon (Apr 08 2025 at 19:35):

I can make Thursday.

As for shrinking the binary, the .so shouldn't be needed and really is only for the embedding case. There are also probably other files that are more for compiling extensions than for running Python itself.

For the stdlib, we could work to strip stuff out more aggressively unless Joel is already leaving out parts that won't work. Otherwise the stdlib is portable and just needs to be found at ../lib/python3.X/site-packages.

As for pre-init, I didn't even know that Joel was doing that and I don't know what function is being called for that. But I will say that towards the bottom of my WASI todo list was trying to refactor CPython to make it easier to pre-init depending on the situation (e.g. locked down environment, generic to any machine, etc.).

Ben Brandt (Apr 09 2025 at 09:44):

Thanks guys, this confirms the behavior I was seeing, and I think it is fine. What is more interesting is seeing memory usage if we cache the InstancePre's in wasmtime.

Our components are roughly ~60MB from componentize-py, with pydantic + some SDK code. The first component takes ~1GB, and then each subsequent one is ~130MB if we keep the InstancePre across invocations.

So it seems there is a heavy cost for loading the first one, but then wasmtime is somehow reusing this memory across components even?

Joel Dice (Apr 09 2025 at 13:49):

@Alex Crichton can correct me if I'm wrong, but my recollection is that Wasmtime uses a copy-on-write mmap mapping of the data segment(s) of the module(s) inside a component from the compiled cwasm into memory, and that mapping is used to populate each instance's linear memory. Depending on how you're measuring memory usage, that would contribute to what you're seeing.

Alex Crichton (Apr 09 2025 at 14:47):

That's correct yeah, although may not account for everything? Multiple instances sharing the same InstancePre will share both executable code and data segments, but note that data segments will be copied if they're modified (e.g. the copy-on-write part of cow), so it's only inert pages that aren't modified past instantiation that will end up sharing memory.

Alex Crichton (Apr 09 2025 at 14:48):

I'm not sure how a 60M component would take 1G of memory, nor how each instantiation afterwards would take 130M, as that sort of looks like little sharing is happening anyway

Ben Brandt (Apr 09 2025 at 15:27):

Ok what we are catching is actually the generated world Pre struct wrapper around an InstancePre that bindgen generates. We cache multiple different components this way that will have multiple invocations each.
But looking at the resident memory of the service for each pre we add to the cache, it goes up 1gb for the first and 130mb for each additional.

They are all python components with very similar makeups, but not identical

Ben Brandt (Apr 09 2025 at 15:28):

Maybe we are doing something wrong here but it is indeed very surprising behavior for us as well

Ben Brandt (Apr 09 2025 at 15:28):

We know the python ones will be bigger than rust ones, but it seems extreme

Alex Crichton (Apr 09 2025 at 15:29):

130M per instance could be the python interpreter just allocating that much memory on startup for each instance, and proving/disproving that would require tracking, e.g. via ResourceLimiter, the size of resources in a store in wasmtime

Alex Crichton (Apr 09 2025 at 15:29):

what I don't understand is the 1G for the initial component. Perhaps it might be useful to use a heap profiler of sorts to try to see where this allocation is going?

Ben Brandt (Apr 09 2025 at 15:30):

Yes we can give that a try and report back

Ben Brandt (Apr 09 2025 at 15:31):

130mb each I also would be ok with if we can figure out what that first one is about

Joel Dice (Apr 09 2025 at 15:38):

Not sure if it will make a significant difference, but you could also try using wasm-tools strip -a to strip the component and see if that affects memory usage.

Ben Brandt (Apr 10 2025 at 08:26):

FYI strip does shrink the module by 20mb, but doesn't change the resident memory significantly. Still working on a way to profile on my work laptop, but will get more details as soon as I can

Ben Brandt (Apr 10 2025 at 09:45):

Ok, I am in over my head with this profiling... But it seems like the biggest allocation paths are in the cranelift compiler code.

Maybe it is possible that keeping one of these components alive is keeping some cranelift things alive as well. Or maybe it is just that the resident memory goes up, but we aren't actually using it all. I may have to spend some time making a minimal reproducible engine + component to make it easier to debug

Ralph (Apr 10 2025 at 09:56):

Ben, please continue this exploration if you can; it's very interesting and may well prove to have tons of good information no matter how it resolves. That 1GB is stunning to me as well.

Joel Dice (Apr 10 2025 at 13:46):

You can take cranelift out of the picture by pre-compiling to a .cwasm using e.g. wasmtime compile and then loading the result at runtime.

Joel Dice (Apr 10 2025 at 15:41):

We'll be meeting today at 2025-04-10T13:00:00-06:00
Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Ben Brandt (Apr 15 2025 at 10:02):

Joel Dice said:

You can take cranelift out of the picture by pre-compiling to a .cwasm using e.g. wasmtime compile and then loading the result at runtime.

FYI, this does seem to make a difference. So it is something with the compilation.
It also seems possible that compilation is intensive enough that there is a lot of memory attached to the process, but it isnt' all being used once the compilation is done

Ben Brandt (Apr 15 2025 at 10:02):

I will try and finish up the experiments, and push them up to a repo for sharing

Ben Brandt (Apr 15 2025 at 15:10):

Ok I am highly suspecting that compilation just requests a lot of memory that doesn't all get paged in after use and then it doesn't get reused for the next component or something.

Joel Dice (Apr 15 2025 at 15:17):

Yeah, it's not surprising to me that compilation of a rather large component would be memory-intensive, and even supposing that Cranelift frees all the memory it uses, that doesn't mean it will necessarily all be returned to the OS promptly. If you haven't tried it yet, I'm a fan of heaptrack for profiling memory usage within a process; that should give you a more precise profile than e.g. watching the resident set size.

Ben Brandt (Apr 15 2025 at 15:19):

I'm currently using dhat-rs and seeing that behavior. So I think we just need to make a decision of how we want to handle this for our long running server to handle this behavior

Joel Dice (Apr 15 2025 at 16:11):

One option would be to run compilation in a separate process.

Ben Brandt (Apr 15 2025 at 16:40):

Yeah thought about that. We do gain a lot with incremental compilation since most of our components are pretty similar (mostly all python with the same wit world) so we'll have to see. Maybe we just plan on the memory footprint

Ben Brandt (Apr 23 2025 at 09:05):

@Joel Dice @Alex Crichton as a follow-up: as a stop-gap until we can play more with precompilation, we switched to jemalloc and are seeing much better memory usage within the container, so it seems to be helping with the memory fragmentation
(since rustc also uses jemalloc, I guess it makes sense if it works better for one compiler it might work well for another)

Dawn Gibson Wages (Apr 30 2025 at 20:32):

Can I get an invite to the Python subgroup re-occurring meeting?

Joel Dice (Apr 30 2025 at 20:33):

Sure! Please DM me your email address and I'll send you an invite.

Ben Brandt (May 08 2025 at 10:20):

Are we meeting today?

Joel Dice (May 08 2025 at 13:55):

Yup; usual details:
Time: 2025-05-08T13:00:00-06:00
Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Joel Dice (Jun 12 2025 at 15:16):

Anyone have agenda items for today's meeting? I don't have anything to discuss, myself.

Ben Brandt (Jun 12 2025 at 15:28):

I don't, and I won't be able to make it either. No updates on my end since last month, getting up to speed at the new job is keeping me busy

Brett Cannon (Jun 12 2025 at 17:13):

I was just going to bring up the stack protection stuff, but it isn't important enough right now just to have a meeting about it.

Joel Dice (Jun 12 2025 at 17:15):

Yeah, might want to keep that discussion on Zulip for now.

Joel Dice (Jun 12 2025 at 17:15):

I'll go ahead and cancel; we can always schedule a one-off meeting if something urgent comes up.

Joel Dice (Jul 10 2025 at 16:01):

For those who are available, we're planning to meet today:
Time: 2025-07-10T13:00:00-06:00
Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

I've put one item on the agenda -- feel free to add more.

Joel Dice (Aug 13 2025 at 21:31):

Anyone have agenda items for tomorrow's meeting? I don't have anything to report or discuss, myself.

Ramon Klass (Aug 13 2025 at 21:35):

I wanted to ask Ben about python3.14 wheels and I wanted to ask if you want to release componentize-py with 3.13 once or skip to 3.14? Anyway nothing that requires a synchronous meeting

Joel Dice (Aug 13 2025 at 21:38):

Regarding componentize-py and 3.13 vs. 3.14: I'm fine with either, but skipping to 3.14 sounds easiest if nobody needs 3.13 support specifically.

Ramon Klass (Aug 13 2025 at 21:45):

agreed, I'll rebase on the rc1 later to see how it goes, and it would be desirable to have wheels close to release, at least the code I need to componentize uses pydantic

Ben Brandt (Aug 14 2025 at 04:44):

Since 3.14 has the best wasi support it should be pretty straightforward to add a new version to the enum

Ralph (Aug 14 2025 at 12:02):

noted: https://bsky.app/profile/marianoguerra.org/post/3lwc4cwbtf22w

Joel Dice (Aug 14 2025 at 17:02):

I'm going to cancel today's meeting since Zulip seems sufficient for what we want to discuss.

Joel Dice (Aug 14 2025 at 17:04):

Well the series expired last month anyway, so nothing to cancel :grinning: . I'll update the series to resume next month.

Joel Dice (Sep 11 2025 at 16:57):

We have a meeting scheduled for 2025-09-11T13:00:00-06:00 today. Does anyone have agenda items?

@Brett Cannon I haven't had a chance to look at the WASI-SDK 26 pthread issue, sorry.

Joel Dice (Sep 11 2025 at 19:02):

We're meeting!
Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Joel Dice (Sep 11 2025 at 19:28):

It was just @Erik Rose and I today. @Ramon Klass Erik is interested in updating componentize-py to CPython 3.13.x or 3.14.x, so maybe you two can team up on that.

Erik Rose (Sep 11 2025 at 19:36):

Hi there! My motivation is to provide real implementations of things like zlib, which are expected by a lot of web frameworks we may be interested in supporting.

Ramon Klass (Sep 11 2025 at 21:18):

oh I wasn't aware it's today and was not at home, sorry
I think adding zlib should be relatively painless, compiling zlib statically with wasi-sdk then providing the right linking parameters
but I'm not sure how far that gets you, most(TM) code that imports zlib will a few lines later either try to send an http request or run an http server which both will fail, but getting further is worth it either way, or else we never find out what might be missing behind the current blocker

Erik Rose (Sep 11 2025 at 21:20):

We solved the HTTP part, at least for our current purposes: since our current project is to see if we can get Python running on a Fastly worker, we wrote a WSGI façade around Fastly's HTTP routines, so WSGI-using frameworks Just Work, at least as far as HTTP goes.

Erik Rose (Sep 11 2025 at 21:21):

Someday it'd be lovely to ditch the proprietary calls and express it in terms of WASI-HTTP. But baby steps. :-)

Ramon Klass (Sep 11 2025 at 21:21):

sounds great :) also it would be great if you could at least import httpx without crashing, even if it didn't work, that would help greatly with porting code already

Erik Rose (Sep 11 2025 at 21:21):

:+1:

Erik Rose (Sep 11 2025 at 21:22):

I imagined that upgrading componentize-py to Python 3.13 or 3.14 would take care of zlib, assuming the CPython folks got around to porting that one. Do I imagine wrong?

Milan (rajsite) (Sep 11 2025 at 21:35):

Kind of related, do we have friends who could poke the urllib3 folks for interest / commentary on adopting wasi:http support in this PR?
https://github.com/urllib3/urllib3/pull/3593
https://github.com/urllib3/urllib3?tab=readme-ov-file#maintainers

Ramon Klass (Sep 11 2025 at 21:38):

67 files changed, +6,038 −63

that's tough to get reviewed indeed, I don't know anyone to poke

Ramon Klass (Sep 11 2025 at 21:42):

@Erik Rose I'm not aware of changes regarding linking of zlib, and the new build script at https://github.com/python/cpython/blob/main/Tools/wasm/wasi/__main__.py doesn't seem to have anything special for it, if there was a change Brett would know (or have done it)

Milan (rajsite) (Sep 11 2025 at 21:46):

Ramon Klass said:

67 files changed, +6,038 −63

that's tough to get reviewed indeed, I don't know anyone to poke

Yea it's mostly wit and generated python bindings. I've had challenges with that in the JS community too. Would be nice if bytecodealliance had packages released against the wasi wit bindings to hide all that away for the different ecosystems instead of each library integration having to figure it out (either adding a bunch of custom tooling and build process or checking in tons of generated stuff).

Ramon Klass (Sep 11 2025 at 21:52):

I'm thinking about how you would do that in a way that "pip install httpx; import httpx; httpx.get()" works as expected, I mean, technically you could have wasi-specific wheels that contain different python code than the wheels for other platforms but that sounds a bit malicious, is there a cleaner way?

Milan (rajsite) (Sep 11 2025 at 22:01):

I don't know enough about python / python ecosystem to say myself. In the npm world I'd imagine having conditional exports that are stubbed for "normal" exports (node / browser) and for a new condition like wasi resolve to the jco generated js that has it's special imports to i.e. import { Fields } from 'wasi:http/types@0.2.3';

Ramon Klass (Sep 11 2025 at 22:04):

ah I thought your goal was that upstream urllib3 and httpx don't have to change anything
I think I would try an approach similar to what rust does, as in, bytecodealliance releases a wasi-http-sys package and nicely asks urllib3 to use that if platform==wasi

Ramon Klass (Sep 11 2025 at 22:04):

that way the 60k lines of generated code belong to those people who know why it is so many lines ;)

Ramon Klass (Sep 28 2025 at 14:23):

some notes regarding py3.14: I got componentize-py with 3.14.0rc3 in a usable state except for the socket support, which had stopped working with py3.13 already, I'm not sure if fixing the current version or working on a wasip3 version would be more fruitful

since py3.14, there is an additional cpython patch required, the stack overflow protection triggers instantly, this is the minimal fix possible, what do you think @Brett Cannon ? https://github.com/Gentle/cpython/commit/499d3ac1be27f42333355509f465033c1119d383

also cpython: 3.14 stopped generating the build-details.json build artifact which is required for make install (used in wasi-wheels repo)

also I've looked into building the non-vendored libraries so we can get past the zlib import issues, I have static libraries for zlib, zstd, bz2 and lzma, I just need to get the tiny build script into a github workflow

Brett Cannon (Oct 02 2025 at 17:46):

Sorry for the late reply; I'm back from the core dev sprints and vacation and trying to get through the backlog.

Ramon Klass said:

I'm not aware of changes regarding linking of zlib

I have plans to introduce an externals subcommand to my wasi command which will support zlib first, but it's a matter of time.

Ramon Klass said:

since py3.14, there is an additional cpython patch required, the stack overflow protection triggers instantly, this is the minimal fix possible, what do you think @Brett Cannon ?

I'm fine with it if tests continue to pass. Me, Hood from Emscripten, and Mark Shannon who authored the stack protection got together at EuroPython and agreed WASM-based platforms are going to opt out of the protection because we can't get it to work and we frankly don't need it for security.

Can you submit a PR to CPython?

Ramon Klass said:

I'm not sure if fixing the current version or working on a wasip3 version would be more fruitful

I'm personally waiting on WASI 0.3 as @Luke Wagner had suggested I just wait since threads should be coming which makes testing socket support possible.

Ramon Klass said:

also cpython: 3.14 stopped generating the build-details.json build artifact which is required for make install (used in wasi-wheels repo)

That's being tracked in https://github.com/python/cpython/issues/137878 .

Ramon Klass said:

also I've looked into building the non-vendored libraries so we can get past the zlib import issues, I have static libraries for zlib, zstd, bz2 and lzma, I just need to get the tiny build script into a github workflow

As I said I hope to get this working upstream so it's all built from source using https://github.com/python/cpython-source-deps . I've been waiting on https://discuss.python.org/t/pre-pep-redesigning-cpython-source-binary-dependencies/101824 to be resolved first as it might influence things.

Ramon Klass (Oct 02 2025 at 17:55):

I'm building zlib, bzip2, xz and zstd here https://github.com/Gentle/cpython-wasi-custom/blob/main/build.sh
output seems to be working

Ramon Klass (Oct 02 2025 at 17:58):

did I understand correctly that we have to also supply libmpdec sooner or later?

Brett Cannon (Oct 06 2025 at 17:05):

Ramon Klass said:

did I understand correctly that we have to also supply libmpdec sooner or later?

I don't think so (we have the libmpdec source in the CPython source), but I have not explicitly checked if it's being used.

Ramon Klass (Oct 07 2025 at 01:53):

@Brett Cannon https://github.com/python/cpython/blob/331158065b7426a791217157585e565157bb851c/Doc/deprecations/c-api-pending-removal-in-3.16.rst#L3 found where I read about libmpdec again

Ramon Klass (Oct 07 2025 at 01:55):

anyway 3.16 is not urgent and if the bundled libmpdec compiles then the tarball version should too

Brett Cannon (Oct 07 2025 at 18:51):

Ramon Klass said:

anyway 3.16 is not urgent and if the bundled libmpdec compiles then the tarball version should too

3.16 is a May 2026 problem :sweat_smile:.

Joel Dice (Oct 09 2025 at 16:29):

We're meeting at 2025-10-09T13:00:00-06:00 today!
Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Joel Dice (Nov 07 2025 at 18:37):

FYI, I just released componentize-py 0.19.0, which includes WASIp3 support. See this example of how to target wasi:http@0.3.0-rc-2025-10-16.

Joel Dice (Nov 12 2025 at 15:55):

We're planning to meet tomorrow at 2025-11-13T13:00:00-07:00 . Feel free to add anything to the agenda.
Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Last updated: Dec 06 2025 at 06:05 UTC