Stream: wasi

Topic: Attempting to use pandas with componentize-py

view this post on Zulip Kyle Gray (Mar 21 2024 at 20:13):

@Joel Dice I compiled pandas from wasi-wheels repository but can't use it with component-py. I get the following error: ```Error:

Caused by:
0: error while executing at wasm backtrace:
0: 0x5b8f58b - /0/pandas/_libs/window/aggregations.cpython-312-wasm32-wasi.so!<wasm function 157>
1: 0x7933972 - __init!<wasm function 1733>
1: wasm trap: wasm unreachable instruction executed```.

Is the expectation that it should work?

view this post on Zulip Kyle Gray (Mar 21 2024 at 20:13):

(wasn't sure where to post this as there isn't a componentize-py stream)

view this post on Zulip Joel Dice (Mar 21 2024 at 20:28):

Yeah, I had trouble with it, too. The instructions in https://github.com/dicej/wasi-wheels/pull/1 indicate to supply a trapping stub for __cxa_throw, which has the virtue of helping everything compile and link, but doesn't get us very far at runtime. I ended up using a different approach: passing -fno-exceptions to the compiler:

 $ git diff pandas |cat
diff --git a/pandas/build.sh b/pandas/build.sh
index 4074ef4..e5175c6 100755
--- a/pandas/build.sh
+++ b/pandas/build.sh
@@ -34,8 +34,8 @@ export CXX="${WASI_SDK_PATH}/bin/clang++"

 export PYTHONPATH=$CROSS_PREFIX/lib/python3.12

-export CFLAGS="-I${CROSS_PREFIX}/include/python3.12 -D__EMSCRIPTEN__=1"
-export CXXFLAGS="-I${CROSS_PREFIX}/include/python3.12"
+export CFLAGS="-I${CROSS_PREFIX}/include/python3.12 -D__EMSCRIPTEN__=1 -fno-exceptions"
+export CXXFLAGS="-I${CROSS_PREFIX}/include/python3.12 -fno-exceptions"
 export LDSHARED=${CC}
 export AR="${WASI_SDK_PATH}/bin/ar"
 export RANLIB=true
Upstreaming a number of deps by chrisdickinson · Pull Request #1 · dicej/wasi-wheels
Hi! Thanks for your great work on componentize-py. I've been working to componentize an existing Python application with a number of native dependencies. I've included my progress in this PR – I'm ...

view this post on Zulip Joel Dice (Mar 21 2024 at 20:29):

But that led to issues with the Cython generated code, which I had to edit by hand to remove the try/catch bits. I couldn't figure out how to tell Cython not to use exceptions, or if that's even possible.

Even after all that, it still failed at runtime when trying to load timezone data via dateutil (apparently due to missing gzip/bz2/lzma/tar support in the stdlib). At that point I gave up. Perhaps @Chris Dickinson has ideas.

view this post on Zulip Kyle Gray (Mar 21 2024 at 22:13):

I was able to get a bit further by following these steps. Instead of trying to figure out how to get gzip working, I manually gunzipped the zoneinfo file instead python-dateutil. the tarfile package works fine if the file isn't compressed.

view this post on Zulip Kyle Gray (Mar 21 2024 at 22:14):

Now I have a different error

Error: Traceback (most recent call last):
  File "/2/log.py", line 2, in <module>
    import pandas as pd
  File "/1/pandas/__init__.py", line 49, in <module>
    from pandas.core.api import (
  File "/1/pandas/core/api.py", line 1, in <module>
    from pandas._libs import (
  File "/1/pandas/_libs/__init__.py", line 18, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
  File "pandas/_libs/hashtable.pyx", line 1, in init pandas._libs.hashtable
  File "pandas/_libs/missing.pyx", line 1, in init pandas._libs.missing
  File "/1/pandas/_libs/tslibs/__init__.py", line 40, in <module>
    from pandas._libs.tslibs.conversion import localize_pydatetime
  File "pandas/_libs/tslibs/conversion.pyx", line 1, in init pandas._libs.tslibs.conversion
  File "pandas/_libs/tslibs/offsets.pyx", line 1, in init pandas._libs.tslibs.offsets
  File "pandas/_libs/tslibs/timestamps.pyx", line 1, in init pandas._libs.tslibs.timestamps
  File "pandas/_libs/tslibs/tzconversion.pyx", line 55, in init pandas._libs.tslibs.tzconversion
ValueError: Buffer dtype mismatch, expected 'const int64_t' but got 'long long'

view this post on Zulip Joel Dice (Mar 21 2024 at 22:15):

Yikes; looks like more Cython trouble. I have basically zero Cython experience, unfortunately. Maybe we need to tell Cython we're cross compiling for a 32-bit platform (wasm32-wasi) somehow?

view this post on Zulip Kyle Gray (Mar 21 2024 at 22:51):

I'm also unfamiliar with Cython. The ctypes package is imported in numpy and seems to work, but it's failing in pandas. Are you familiar with that at all?

view this post on Zulip Joel Dice (Mar 21 2024 at 22:57):

Yeah, my guess is that ctypes works just fine on wasm32-wasi, and I assume numpy uses it in a pretty conventional way to interfaces with the parts written in C++. pandas, on the other hand, adds Cython to the mix, and Cython converts Python-ish code to C/C++. My guess is that Cython also assumes that the C/C++ code it's generating will be compiled for the native platform it's running on rather than be cross-compiled to wasm32-wasi, so any platform-specific parts of that generated code are for the wrong platform.

view this post on Zulip Joel Dice (Mar 21 2024 at 22:59):

I'm just guessing, though: I haven't dug into it.

view this post on Zulip Brett Cannon (Jun 17 2024 at 22:11):

ctypes can be imported so you can get at the constants in the file, but not all the functionality is available, e.g. dynamic loading and execution of other code.

Last updated: Oct 23 2024 at 20:03 UTC