Stream: SIG-Guest-Languages

Topic: Python subgroup

Joel Dice (Aug 31 2023 at 15:32):

The Python subgroup will be meeting at 2023-08-31T13:00:00-06:00 today at https://meet.jit.si/PythonComponentTooling. Agenda is here: https://hackmd.io/hazcX6vkT0e-90nqKOvEeg. It's pretty open-ended; my main goal is to decide what projects to tackle next now that componentize-py is starting to mature.

Joel Dice (Sep 14 2023 at 18:15):

The Python subgroup will be meeting at 2023-09-14T13:00:00-06:00 at https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09. Agenda (sparse as it is) is here: https://hackmd.io/mLLo1CduQLeGk9h05WGFAw. Feel free to add items.

Joel Dice (Sep 27 2023 at 22:34):

@Hood Chatham apparently all SciPy tests are passing on LFortran: https://bytecodealliance.zulipchat.com/#narrow/stream/206238-general/topic/Can.20WASM.20be.20compiled.20to.20a.20standalone.20binary.3F/near/393576378

Hood Chatham (Sep 28 2023 at 00:32):

I think if I read that comment correctly, he means that he can "fully compile several of the Fortran modules" of scipy "and all tests pass" for the modules that can be compiled. But it does sound like it's coming along very well =)

Hood Chatham (Sep 28 2023 at 00:34):

As far as I know, LFORTRAN still can't quite generate position independent code but I'm pretty sure it's due to some trivial mistake

Hood Chatham (Sep 28 2023 at 00:34):

I was able to produce PIC binaries by first asking it to emit llvm IR and then compiling that

Hood Chatham (Sep 28 2023 at 00:35):

In other news, it looks like pmatos maybe fixed the problems with wasm table access builtins and PIC:
https://github.com/llvm/llvm-project/issues/65191#issuecomment-1737244995

Joel Dice (Sep 28 2023 at 17:02):

We'll be meeting at 2023-09-28T13:00:00-06:00 today. Agenda and notes here: https://hackmd.io/uhmQG41TS0mRtOTP9ikbUA. https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Brett Cannon (Sep 28 2023 at 17:35):

I'll probably miss today's meeting. My update is I'm working on a script to make building CPython easier.

Brett Cannon (Oct 04 2023 at 21:37):

CPython 3.12.0 WASI build https://github.com/brettcannon/cpython-wasi-build/releases/tag/v3.12.0

Scott Waye (Oct 05 2023 at 14:56):

How is Python generating binding for WIT ?

Joel Dice (Oct 05 2023 at 15:07):

CPython currently targets WASI Preview 1 (via wasi-sdk), which is not based on WIT. If you want to target WASI Preview 2 (which is based on WIT), you can use https://github.com/bytecodealliance/componentize-py (which uses CPython but also supports targeting arbitrary WIT worlds)

Scott Waye (Oct 05 2023 at 17:14):

I see, so you have a code generator in Rust but you decided not to create a backend to wit-bindgen, is that right?

Joel Dice (Oct 05 2023 at 17:21):

Correct. The main reason for that is most of the code componentize-py generates is Wasm, not Python. The intention is that part could be reused for other high-level languages.

Joel Dice (Oct 05 2023 at 17:22):

(i.e. all the lifting and lowering code is Wasm, with calls out to CPython to construct and deconstruct Python objects)

Scott Waye (Oct 05 2023 at 17:51):

Interesting, thanks. Do you know if javascript it the same by any chance?

Joel Dice (Oct 05 2023 at 17:55):

No, I think componentize-js currently generates JS code for lifting and lowering. If @Guy Bedford agrees, we could split the Wasm lift/lower generator out of componentize-py and reuse it in componentize-js. It would be a pretty big refactor, though.

Joel Dice (Oct 05 2023 at 17:58):

componentize-js is similar to componentize-py in that it has its own binding generator which is not part of wit-bindgen, if that's what you meant to ask.

Joel Dice (Oct 05 2023 at 18:00):

I think the trend is that each guest language tool for producing components will generate bindings its own way rather than have wit-bindgen do it. They'll all presumably use wit-parser and maybe share other code, but I don't think wit-bindgen will necessarily be a central repository for all bindings generators.

Scott Waye (Oct 05 2023 at 18:20):

Ok, there is a preference from the dotnet runtime team to follow a protobuf/protoc approach, like wit-bindgen. The c# approach is currently at this crossroads so I'm trying to gather opinions so thanks for the input. If anyone wants to chip in directly: https://github.com/dotnet/runtimelab/issues/2409

Joel Dice (Oct 12 2023 at 13:43):

Does anyone have any agenda items for today's Python subgroup meeting? If not, I was thinking of canceling it.

Joel Dice (Oct 12 2023 at 17:58):

I went ahead and cancelled it. My only update is that componentize-py now supports resources. Haven't been doing anything Python related since then.

Joel Dice (Oct 26 2023 at 16:26):

We'll be meeting at 2023-10-26T13:00:00-06:00 today. Agenda here: https://hackmd.io/Wu5-BHBfTdq_gri90BdRiw. Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Joel Dice (Oct 26 2023 at 19:01):

...er, maybe. Having technical issues logging into Zoom under the BA account. The credentials are no longer showing up in my BA 1password account.

Joel Dice (Oct 26 2023 at 19:07):

Sorry, it might not happen after all. I'm not able to log in to the host account, and having trouble finding anyone who can help on short notice.

Brett Cannon (Oct 26 2023 at 19:15):

My update was going to be wasi.py requires a quick validation by me on macOS and then adding support to specify the command for the WASI runtime used at Python build time (wasmtime 14 moved that from a nice-to-have to a requirement).

Joel Dice (Oct 26 2023 at 19:15):

We're in now!

Joel Dice (Nov 09 2023 at 00:17):

If I'm reading this right (I'll admit I've only skimmed it), SciPy can now be built by the LLVM 17 release of Flang for Windows and pass 100% of the test suite: https://labs.quansight.org/blog/building-scipy-with-flang. Might be worth using Flang to target Wasm and see what happens.

Joel Dice (Nov 09 2023 at 16:32):

We'll meet at 2023-11-09T13:00:00-07:00 today. Could be a short one if there's not a lot to discuss. Feel free to add to the agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Brett Cannon (Nov 22 2023 at 17:36):

I'm assuming there's no meeting tomorrow due to US Thanksgiving?

Joel Dice (Nov 22 2023 at 17:37):

Correct; I'll cancel it.

Joel Dice (Dec 07 2023 at 16:17):

We'll be meeting at 2023-12-07T13:00:00-07:00 today. Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09 Agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA

Joel Dice (Jan 04 2024 at 16:04):

Does anyone have agenda items for today's Python meeting? If not, I'll cancel it.

Joel Dice (Jan 18 2024 at 16:13):

Anyone have agenda items for today's Python meeting?

Ralph (Jan 18 2024 at 17:40):

how great you all are?

Ralph (Jan 18 2024 at 17:41):

prolly too late now, but.....

Joel Dice (Jan 18 2024 at 19:12):

I'm going to go ahead and cancel the meeting since there are no agenda items and I enjoy cancelling meetings.

Relatedly: In the past few meetings we _didn't_ cancel, we mostly discussed the various things we _should_ do but which nobody currently has bandwidth to work on:

• Build and publish WASI wheels for popular packages to a public index so devs can pip install them easily
  - Porting those packages can range from pretty trivial (e.g. a package with a simple, already very portable C native extention) to extraordinarily difficult (e.g. SciPy and anything else involving Fortran).
  - This is highly parallelizable work, but also an enormous amount of effort if we want to support e.g. all the packages Pyodide supports
• Improve the componentize-py dev experience (e.g. automatically create a virtual environment, install all required packages to it, and build the component)

In short, we're stalled right now -- lots of work to do, but no bandwidth to do it. Until we move out of that state, I don't expect we'll have much to discuss.

Ralph (Jan 18 2024 at 19:34):

I have a sneaking suspicion that there will be a lull having finished the project work... while people breathe and test things out.

Ralph (Jan 18 2024 at 19:35):

I built some samples and passed them around and had some real success on jco and wasmtime and spin for example, and the more that happens the more there will be some volunteers for the next tasks, porting wheels and so forth....

Ralph (Jan 18 2024 at 19:36):

I think the ms python peeps are going to be motivated, so I hope to help enable them to realize that they can really help push things forward by helping.

Ralph (Jan 18 2024 at 19:36):

<crossed fingers here />

Ralph (Jan 18 2024 at 19:36):

you have issues for steps marked out?

Ralph (Jan 18 2024 at 19:36):

I'll try to find people!

Joel Dice (Jan 18 2024 at 20:00):

componentize-py has an up-to-date issue list, and I'm always happy to mentor new devs: https://github.com/bytecodealliance/componentize-py/issues

Otherwise, the big effort is porting packages to WASI, and I'm not sure how to mark that out; i.e. which packages to prioritize. I'm still quite new to the whole Python ecosystem, so I don't even know what the most popular packages are. I guess we could make a checklist out of the list of packages Pyodide supports (https://github.com/pyodide/pyodide/tree/main/packages), but again I'm not sure how to order it by priority.

Ralph (Jan 19 2024 at 09:57):

good chance Brett will know, and the pythonistas here. I'll ask around

Brett Cannon (Jan 22 2024 at 20:02):

Ralph said:

good chance Brett will know, and the pythonistas here. I'll ask around

I do know and there's no need to ask around unless you're after conda because otherwise all roads lead to me. :sweat_smile: There aren't issues because I'm still focusing towards tier 2 support. After that is seeing if WASI-SDK 21 has what's necessary for dlopen() and figuring out how to tell the community to get the packages to work. Then we have to decide how the wheel platform tags will function (both what they specify and how to tell if a platform supports the tag), especially with wasi-libc having no forwards/backwards-compatibility. Once that's all done we can the ask to allow for uploads to PyPI and hopefully get various build backends to support WASI.

Joel Dice (Jan 22 2024 at 20:44):

wasi-sdk 21 does have shared library and dlopen support, so we can check that off the list. Even once we have all the decisions made and infrastructure in place, though, each porting each individual package will require effort. Some of them will just work with no code changes or WASI-specific hacks. As Pyodide has shown, though, many will require a lot more effort, and I expect that upstream maintainers won't want to take that on themselves.

Ralph (Jan 23 2024 at 10:16):

wasi-libc support resources will appear; it's becomign clear to a lot of people that this is a path that is needed.

Brett Cannon (Jan 23 2024 at 22:54):

Joel Dice said:

As Pyodide has shown, though, many will require a lot more effort, and I expect that upstream maintainers won't want to take that on themselves.

Two things on that. One, part of that will be due to things like no socket support, so we may not be entirely in the same spot as them depending on what WASI-libc covers. And two, I have some :carrot: ideas to entice projects into caring by solving some long-standing Python ecosystem issues that WebAssembly and WASI are uniquely positioned to help w/. :wink:

Brett Cannon (Jan 26 2024 at 21:01):

Just a quick note that wasmtime 17 (and earlier) don't pass Python's test suite under preview 2. I have filed the requisite bugs, but I wanted to save others time in case people rushed out to try preview 2 w/ CPython and noticed the test failures.

Joel Dice (Feb 01 2024 at 16:03):

I'm cancelling today's meeting (again) since many of us are attending the BA contributors' summit today.

Brett Cannon (Feb 07 2024 at 00:41):

https://github.com/python/devguide/pull/1273 is my PR documenting how to build CPython for WASI. Once this gets merged we will be down to just clean-up PRs to move WASI to tier 2 support!

Ramon Klass (Feb 07 2024 at 00:49):

very exciting :)

Brett Cannon (Feb 08 2024 at 21:40):

https://peps.python.org/pep-0011/#tier-2

Joel Dice (Feb 15 2024 at 15:06):

For the first time in 2 months, today's meeting is not cancelled.

• Time: 2024-02-15T13:00:00-07:00
• Agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
• Zoom link: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

See you then!

Joel Dice (Feb 16 2024 at 18:53):

@Brett Cannon I don't know if this is useful for you at all, but here are the terrible hacks I've applied to CPython 3.12.1 to enable .so and socket support: https://github.com/dicej/cpython/commit/118e9d8eeabb6345d57295f6cec1616694eacbc2

Ralph (Feb 18 2024 at 15:59):

love me some bad, bad hacks that work

Lann Martin (Feb 18 2024 at 18:21):

let terrible_dotnet_hack = detect_dotnet(&wasm_bytes);

This "temporary" fix (in some internal code) is over a year old now!

Joel Dice (Feb 28 2024 at 17:32):

Anybody have agenda items for tomorrow's meeting? If not I'll cancel it.

BTW, a quick update on my action item from the last meeting: I spoke with Dan Gohman about improving throughput on wasi-libc PRs, and our plan is essentially this: when a PR goes idle for a while (i.e. a week with no comments) without being merged after all the feedback has been addressed, I'll make a "final request for comments", and if there are none after several days, we'll merge it. This seems to be working well so far.

Regarding the other big question about maintaining binary compatibility across wasi-libc releases: no news there, i.e. nobody has stepped up to take (or fund) that responsibility yet.

Brett Cannon (Feb 28 2024 at 19:14):

My agenda item would be https://bytecodealliance.zulipchat.com/#narrow/stream/394175-SIG-Guest-Languages/topic/All.20known.20issues.20for.20CPython.20under.20wasmtime.2018.20.26.20WASI.20SDK.2021 , but we also don't need to have a meeting just for that.

Joel Dice (Feb 28 2024 at 21:07):

@Brett Cannon If we were to meet and discuss that (which I'd be happy to do), what would the goal(s) be? Increase visibility for those issues and invite feedback? Resolve open questions related to those issues?

Ralph (Feb 28 2024 at 21:20):

I might swing by to understand the area. I need to find some help here.

Joel Dice (Feb 28 2024 at 21:30):

Sounds good; I won't cancel it, then.

Joel Dice (Feb 29 2024 at 15:30):

Meeting info for today:

• Time: 2024-02-29T13:00:00-07:00
• Agenda: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA
• Zoom: https://zoom.us/j/95590111447?pwd=ekpERFNUZXNRcm9QVktLQlRHbXZYdz09

Brett Cannon (Feb 29 2024 at 16:53):

Joel Dice said:

Brett Cannon If we were to meet and discuss that (which I'd be happy to do), what would the goal(s) be? Increase visibility for those issues and invite feedback? Resolve open questions related to those issues?

Visibility, status update as to where CPython stands for WASI 0.2, and just in case someone knows I'm overlooking something.

Joel Dice (Mar 14 2024 at 14:02):

Any objections to me cancelling today's meeting? Looks like a few of our regular attendees can't make it, and we don't have anything on the agenda anyway.

Joel Dice (Mar 27 2024 at 21:57):

Anyone have agenda items for tomorrow's meeting? If not, we can just do an asynchronous status update here.

Joel Dice (Mar 28 2024 at 15:26):

I'm going to cancel the meeting since there are no agenda items (and since I _love_ cancelling meetings).
My update:

• Released componentize-py 0.13, which includes isyswasfa support (which I plan to replace with proper CM async support once matching support is available in wasm-tools, wasmtime, etc.), as well as a --stub-wasi option for eliminating any WASI imports (see example).
• Tried to help @Kyle Gray get Pandas working inside a component; failed due to trouble with Cython-generated C++ code, which we had to hand-edit to make compatible with -fno-exceptions but still had issues which might be cross-compiling related: https://bytecodealliance.zulipchat.com/#narrow/stream/219900-wasi/topic/Attempting.20to.20use.20pandas.20with.20componentize-py. Perhaps someone with more Cython experience can help? Might also be worth looking at what Pyodide does to support Pandas.

Joel Dice (Apr 11 2024 at 16:00):

No agenda again this week, so I'll cancel today's meeting if there are no objections.

Joel Dice (Apr 25 2024 at 13:20):

Anyone have agenda items for today's meeting?

Joel Dice (Apr 25 2024 at 18:28):

Crickets; I'll cancel it.

Joel Dice (Jul 11 2024 at 18:17):

For the first time in a while, we're having a meeting today at 2024-07-11T13:00:00-06:00 (i.e. about 45 minutes from now).

• Link: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09
• Agenda and notes: https://hackmd.io/3ZktfALiSZWn41ad7y3gnA

Brett Cannon (Jul 11 2024 at 20:35):

I think my most immediate question is if I use the new wasm32-wasi-preview2 target triple (although @Joel Dice made it sound like the triple was actually wasm23-wasip2), I get sockets, which is great! But compiling for a component instead of a module; I'm not sure how that affects me. I still want a python.wasm that I can pass to e.g. wasmtime to run the compiled interpreter. Does this change? And I assume it doesn't affect wanting to statically compile .o files for things like zlib so that I can statically link it?

Joel Dice (Jul 11 2024 at 20:39):

When you target wasm32-wasip2 (which is now the official name; wasm32-wasi-preview2 is no longer used anywhere, AFAIK), python.wasm will be a component, which should run fine using Wasmtime, node+jco, browser+jco, or any other WASIp2-capable runtime. It won't run on any runtime which only supports WASIp1, though.

And, yes, statically linking .o and .a files should continue to work like it always has.

Brett Cannon (Jul 11 2024 at 20:49):

FYI the wasm32-wasi-preview2 target is specifically mentioned at https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-22 because of you :wink:

Joel Dice (Jul 11 2024 at 20:51):

Yeah, that PR landed before the name change, which happened later in the WASI-SDK 22 development process.

Brett Cannon (Jul 11 2024 at 20:58):

https://github.com/python/cpython/issues/121634 is tracking trying out the new target triple (after I make sure WASI SDK 22 builds fine under wasm32-wasi).

Joel Dice (Jul 11 2024 at 21:00):

Note that you can always support both wasm32-wasip1 and wasm32-wasip2 in CPython if desired (i.e. as separate build targets). We decided not to do that for .NET because it compounded the maintenance burden, but maybe it wouldn't be so bad for CPython?

Brett Cannon (Jul 12 2024 at 18:39):

I assume I will support both (for now) by making the target triple an option for my build script

Victor Adossi (Jul 22 2024 at 15:47):

Hey all (and maybe @Joel Dice, though I assume :palm_tree: means vacation! ) -- I have some friends over SONY Midokura over here in :flag_japan: that are trying to do python WebAssembly things, and they're doing it the hard way -- compiling MicroPython w/ wasi-sdk and trying to work out all the hard parts themselves.

I know there was a TON of hard work in this area, but before I give them the links I have to sort of show the walk through/progression of shared everything linking and the road to componentize-py/work that's been done on it since (that I didn't do!), I wanted to ask if there's a good existing resource that describes everything concisely?

I'd like to do just a little bit better than pointing them to https://www.fermyon.com/blog/introducing-componentize-py and https://github.com/bytecodealliance/componentize-py and recent PRs to upstream python, etc.

Lann Martin (Jul 22 2024 at 16:02):

This is another resource: https://component-model.bytecodealliance.org/language-support/python.html

Victor Adossi (Jul 22 2024 at 16:06):

Thanks for that link, will definitely include it. They're quite technical as they've started and had some success with the MicroPython build so was hoping for just a few resources that also... throw them off the deep end, in a matter of speaking

Brett Cannon (Jul 22 2024 at 17:22):

@Victor Adossi how are they dealing with the long jump issue as my understanding is that the WASI SDK needs the exceptions proposal to land for that to work (i.e. I tried this last year :sweat_smile:)?

And yes, Joel is the best person to answer all of this. But if they are using MicroPython then how would linking come into play since I didn't think MicroPython had extension module support?

Joel Dice (Jul 22 2024 at 17:56):

Yes, I'm on vacation and typing this on my phone. Briefly: I spoke at length with a couple of people from Midokura at Wasmcon last year, and @YAMAMOTO Takashi is familiar with the work I've done to support shared libraries in wasi-sdk, so I think they already know a lot of the technical details. My understanding is that they're using Wamr, which does not yet support WASIp2 or the component model, which may help explain the approach they're currently taking.

Joel Dice (Jul 22 2024 at 17:56):

BTW, @Luke Wagner wrote this up about linking in the CM, for reference: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Linking.md

Victor Adossi (Jul 23 2024 at 04:58):

Hey thanks for the thoughtful responses!

@Brett Cannon I'm not sure exactly what which long jump issue this is referring to, but maybe it's the implementation done by Yamato-san here: https://github.com/WebAssembly/wasi-libc/pull/483 (this was mentioned & demo'd in a talk I saw last night!)

I was trying to discuss less linking specifically and more getting things working in python land in general. I know less about python and even less about micropython but was hoping for resources that sound like "X didn't work so then I Y'd but there was a problem with Z" kind of write-up that could be illuminating for people working with a python-flavored toolchain.

at Joel :palm_tree: (but so this doesn't ping)

Apologies! I actually saw a presentation on the work that Yamato-san did just last night :) And you're absolutely right about their approach, I need to ask more about what they're running into, maybe I'll check out the embedded WG.

I want to be clear that it wasn't quite about linking specifically so much as I wanted any more documentation on how it all sort of fits together (high level and low level), and possibly the messy kind -- I should have asked more specifically about the challenges they were running into so I could be more specific!

Brett Cannon (Jul 23 2024 at 18:03):

@Victor Adossi Thanks for the link! That's exactly the "long jump" issue I was referring to. Do you know if there are plans to upstream the changes to MicroPython (they have Emscripten support in the repo, so I suspect they would take WASI as well).

And https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md might be the low level dynamic linking docs you were looking for.

Victor Adossi (Jul 24 2024 at 03:57):

Hey @Brett Cannon no problem! I don't know if they plan on upstreaming, but I know they're working with MicroPython right now... I'll shoot over a question and certainly @YAMAMOTO Takashi will know way more than I do.

Thanks for that dynamic linking doc -- most of that (and how it integrates with non-emscripten LLVM WASM builds) is over my head but certainly useful for others. Reviewing the componentize-py code would probably help me understand more as well.

Joel Dice (Jul 29 2024 at 14:40):

Yeah, componentize-py needs more docs, including an ARCHITECTURE.md (hence this issue). Meanwhile, this presentation might be helpful.

Brett Cannon (Aug 08 2024 at 18:21):

I'm assuming there's no meeting today since nothing has been mentioned about it?

Joel Dice (Aug 08 2024 at 19:09):

Argh, sorry I forgot about this. I'm in Germany for an offsite this week and won't be able to host. I just cancelled it (yes, 9 minutes after it was supposed to start).

Brett Cannon (Aug 15 2024 at 18:31):

Joel Dice said:

Note that you can always support both wasm32-wasip1 and wasm32-wasip2 in CPython if desired (i.e. as separate build targets). We decided not to do that for .NET because it compounded the maintenance burden, but maybe it wouldn't be so bad for CPython?

So I got wasm32-wasip2 building, but I didn't try turning any new features like sockets on either (i.e. I got the wasi.py script that drives builds for CPython to allow specifying the host's target triple). But upon reflection, supporting both wasm32-wasi and wasm32-wasip2 as separate targets is a pain as Python's tier support is like Rust (which is my doing :sweat_smile:), so it would be a new platform requiring a buildbot test machine, sponsorship, etc.

I assume a wasm32-wasi and a wasm32-wasip1 build are not interchangeable if you stick to only the parts that are in both (I know the APIs are different, so I'm talking from a feature perspective)?

Ramon Klass (Aug 15 2024 at 18:36):

I'm sure I know less about the new targets than you but when they were announced, wasip1 was called a rename of wasi, has that changed?

ref: https://blog.rust-lang.org/2024/04/09/updates-to-rusts-wasi-targets.html

Joel Dice (Aug 15 2024 at 18:39):

AFAIK wasm32-wasi and wasm32-wasip1 are interchangeable in the sense they should both generate the exact same output.

Joel Dice (Aug 15 2024 at 18:40):

I.e. the latter is just a new name for the former.

Ramon Klass (Aug 15 2024 at 18:45):

while we're here, @Joel Dice would it be helpful if I try to update wasi-sdk and use Brett's build instead of maybe_make_cpython in componentize-py or do you want to do that yourself? I'd like to help

Joel Dice (Aug 15 2024 at 18:49):

@Ramon Klass This is my CPython patch -- it enables WASI sockets and shared library support. If there's a route to upstream either of those features in the near future, great; otherwise, I think we'll need to stick with my fork. The shared library support in particular is essential given how componentize-py works internally.

Ramon Klass (Aug 15 2024 at 18:50):

does the same apply to the wasi-wheels repo?

Joel Dice (Aug 15 2024 at 18:52):

Hmm, now I'm trying to remember what the cpython submodule in the wasi-wheels repo was for...

Joel Dice (Aug 15 2024 at 18:55):

looks like we use it to specify CROSS_PREFIX when building the various wheels. Maybe those builds require a libpython312.so to be present? Or maybe not? Feel free to switch it to a pre-built CPython for WASI -- maybe it will work?

Ramon Klass (Aug 15 2024 at 18:58):

ok then I'll start with wasi-wheels; are .a files from wasi-sdk23 compatible to wasi-sdk24 or should I always use the same wasi-sdk as build.py?

Joel Dice (Aug 15 2024 at 19:02):

Yes, AFAIK they should be compatible.

Ramon Klass (Aug 15 2024 at 19:14):

also @Brett Cannon I saw you want to include zlib into the build but what about other libraries? Specifically openssl, some libraries (in my case httpx) just assume that import _ssl will never fail, which is sad since you can't even import httpx because of that

Brett Cannon (Aug 15 2024 at 19:17):

Joel Dice said:

Ramon Klass This is my CPython patch -- it enables WASI sockets and shared library support. If there's a route to upstream either of those features in the near future, great; otherwise, I think we'll need to stick with my fork. The shared library support in particular is essential given how componentize-py works internally.

So that patch is half outdated, half doesn't work. :sweat_smile: The stuff to turn off the tests I already have upstream at Joel's request so that "dynamic" loading via imports work w/o patching, but the stuff that rips out the stack-related flags fails when Python is compiled under --with-pydebug which is necessary for CI testing.

I don't know about the sys/ioctl.h portion as every test in test_socket either errors out, gets skipped, or fails (lack of threading seems to be the main culprit for the failures, but I admittedly didn't dig into it).

If people want to help upstream stuff I'm happy to work w/ them.

Brett Cannon (Aug 15 2024 at 19:19):

Ramon Klass said:

also Brett Cannon I saw you want to include zlib into the build but what about other libraries? Specifically openssl, some libraries (in my case httpx) just assume that import _ssl will never fail, which is sad since you can't even import httpx because of that

I have been told OpenSSL won't really work as it requires constant time instructions for cryptography reasons and WebAssembly doesn't have that. For the rest I was hoping to get them working one by one.

Joel Dice (Aug 15 2024 at 19:19):

ssl is its own can of worms given the lack of constant time operations in Wasm and the risk of side channel attacks. I wouldn't recommend using OpenSSL on Wasm, even if it technically "works".

Ramon Klass (Aug 15 2024 at 19:25):

I see the point with "it wouldn't be secure either way", it's just a bit rough that just importing a package but never using it already kills the componentization, but I guess that's kind of the python ecosystem's fault, openssl was always optional, there should be guards around the import

Ramon Klass (Aug 15 2024 at 19:29):

just found that someone did in fact set up a build repo for openssl-wasi ;) https://github.com/jedisct1/openssl-wasm

Ramon Klass (Aug 15 2024 at 19:34):

(why yes, I totally trust that you patch rand_lib.c without explanation)

Ramon Klass (Aug 15 2024 at 19:35):

but thank you both, a definite answer on the ssl thing is good to have even if it's wontfix with good reason :)

Joel Dice (Aug 15 2024 at 20:20):

FWIW, this is one of the reasons for the wasi:sockets/tls interface I proposed a while ago. That will probably be more useful for languages with higher-level abstractions for TLS (e.g. .NET/System.Net.Security.SslStream, Rust/native-tls, JS/node:tls), but might eventually benefit Python as well.

Ramon Klass (Aug 15 2024 at 20:22):

that's a promising path forward :) python could use this to implement a different version of the _ssl module once it's accepted

Ralph (Aug 16 2024 at 12:22):

Ramon Klass said:

just found that someone did in fact set up a build repo for openssl-wasi ;) https://github.com/jedisct1/openssl-wasm

this particular spike didn't get too far, and I'm not sure it's the one to think about.

Ramon Klass (Aug 16 2024 at 12:29):

yes the wording in the README is rather unclear about the security implications their patches introduce, I think putting efforts toward a wit interface that allows using host tls is the best way forward

one of the main issues I see is, people right now don't just want to have TLS inside components, people probably would expect that they can compile their C code to wasi and run it, and that C code will probably #include openssl, so would that mean wasi-sdk needs a openssl-to-wasitls bridge that can be linked?

Ralph (Aug 16 2024 at 12:30):

both approaches will be needed, but the question now is how to specify and provide resources for such a wasi:tls/ssl

Ralph (Aug 16 2024 at 12:30):

questions abound yet

Bailey Hayes (Aug 19 2024 at 21:16):

Joel Dice said:

AFAIK wasm32-wasi and wasm32-wasip1 are interchangeable in the sense they should both generate the exact same output.

A lil late, but wanted to chime in that there is a hope that we could use wasm32-wasi as wasi 1.0 in the future.

Milan (Sep 07 2024 at 20:37):

Joel Dice said:

• Build and publish WASI wheels for popular packages to a public index so devs can pip install them easily
  - Porting those packages can range from pretty trivial (e.g. a package with a simple, already very portable C native extention) to extraordinarily difficult (e.g. SciPy and anything else involving Fortran).

Know where I can find the state of different packages that have wasi wheels published? Also curious if the recent LFortran wasm target improvements are helping for a path for scipy, etc: https://lfortran.org/blog/2024/05/fortran-on-web-using-lfortran/

Ramon Klass (Sep 07 2024 at 20:45):

there is a sandbox for building wasi wheels here https://github.com/dicej/wasi-wheels
it's a bit barebones and spits out zip files you need to extract into your site-pckages but all the needed parts are there, I'd also be interested in coordinated work on this :)

Ramon Klass (Sep 07 2024 at 20:47):

I have a proof of concept that turns the output into a NAME-wasi-X.Y.Z.whl that depends on NAME-X.Y.Z and only adds the additional .so files but I haven't converted all the build.shs to that yet

Joel Dice (Sep 10 2024 at 20:05):

We have a meeting scheduled for Thursday. I've still been focused on .NET stuff, so don't have much to report on the Python side. Feel free to add agenda items here.

Brett Cannon (Sep 11 2024 at 17:55):

My only update would be going over https://opensource.snarky.ca/Python/WASI which I will also be presenting at the plumber's summit, so I'm happy to talk about it, but also understand not having a meeting if it isn't enough.

Joel Dice (Sep 11 2024 at 18:39):

Let's go ahead and meet. I'll put that (i.e. Brent's WASI todo list) on the agenda, and I can also talk briefly about a couple of recent and in-progress contributions to componentize-py.

Brett Cannon (Sep 12 2024 at 19:10):

Is the meeting still happening? I'm in the lobby and Zoom is saying it's waiting for the host to start the meeting.

Joel Dice (Sep 12 2024 at 19:11):

oh crap, I had an alarm set, but then got caught up in a conversation

Joel Dice (Sep 12 2024 at 19:11):

will start it asap

Joel Dice (Sep 12 2024 at 19:11):

(need to dig up the password)

Joel Dice (Sep 12 2024 at 19:11):

sorry!

Joel Dice (Oct 10 2024 at 15:21):

Python meeting today at 2024-10-10T13:00:00-06:00 . Feel free to add items to the agenda. Zoom link: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Brett Cannon (Oct 10 2024 at 17:10):

If it's just us and you don't have anything to talk about, @Joel Dice , we can cancel as beyond the 3.13.0 release I haven't had WASI time lately.

Joel Dice (Oct 10 2024 at 17:12):

I was just going to point to https://github.com/WebAssembly/component-model/issues/401 in case you or anyone else hasn't seen it yet. Otherwise, I've got no news either, so I'm fine with canceling.

Joel Dice (Oct 10 2024 at 17:31):

I went ahead and cancelled it in favor of async discussion here as needed.

Joel Dice (Nov 13 2024 at 19:41):

We have agenda items for the meeting tomorrow (2024-11-14T13:00:00-07:00). If you're not otherwise occupied with e.g. KubeCon, please join us.

Brett Cannon (Nov 14 2024 at 17:06):

FYI I plan to be at the meeting but there's a slight chance I will be a little late

Joel Dice (Nov 14 2024 at 19:52):

Meeting starts in about 8 minutes: https://zoom.us/j/3385977601?pwd=UjJhYWh1MW5RTHg5eE9JTGV4UksrUT09

Last updated: Nov 22 2024 at 16:03 UTC