@Joel Dice I compiled pandas from wasi-wheels repository but can't use it with component-py
. I get the following error: ```Error:
Caused by:
0: error while executing at wasm backtrace:
0: 0x5b8f58b - /0/pandas/_libs/window/aggregations.cpython-312-wasm32-wasi.so!<wasm function 157>
1: 0x7933972 - __init!<wasm function 1733>
1: wasm trap: wasm unreachable
instruction executed```.
Is the expectation that it should work?
(wasn't sure where to post this as there isn't a componentize-py stream)
Yeah, I had trouble with it, too. The instructions in https://github.com/dicej/wasi-wheels/pull/1 indicate to supply a trapping stub for __cxa_throw
, which has the virtue of helping everything compile and link, but doesn't get us very far at runtime. I ended up using a different approach: passing -fno-exceptions
to the compiler:
$ git diff pandas |cat
diff --git a/pandas/build.sh b/pandas/build.sh
index 4074ef4..e5175c6 100755
--- a/pandas/build.sh
+++ b/pandas/build.sh
@@ -34,8 +34,8 @@ export CXX="${WASI_SDK_PATH}/bin/clang++"
export PYTHONPATH=$CROSS_PREFIX/lib/python3.12
-export CFLAGS="-I${CROSS_PREFIX}/include/python3.12 -D__EMSCRIPTEN__=1"
-export CXXFLAGS="-I${CROSS_PREFIX}/include/python3.12"
+export CFLAGS="-I${CROSS_PREFIX}/include/python3.12 -D__EMSCRIPTEN__=1 -fno-exceptions"
+export CXXFLAGS="-I${CROSS_PREFIX}/include/python3.12 -fno-exceptions"
export LDSHARED=${CC}
export AR="${WASI_SDK_PATH}/bin/ar"
export RANLIB=true
But that led to issues with the Cython generated code, which I had to edit by hand to remove the try/catch
bits. I couldn't figure out how to tell Cython not to use exceptions, or if that's even possible.
Even after all that, it still failed at runtime when trying to load timezone data via dateutil
(apparently due to missing gzip/bz2/lzma/tar support in the stdlib). At that point I gave up. Perhaps @Chris Dickinson has ideas.
I was able to get a bit further by following these steps. Instead of trying to figure out how to get gzip
working, I manually gunzipped the zoneinfo file instead python-dateutil. the tarfile
package works fine if the file isn't compressed.
Now I have a different error
Error: Traceback (most recent call last):
File "/2/log.py", line 2, in <module>
import pandas as pd
File "/1/pandas/__init__.py", line 49, in <module>
from pandas.core.api import (
File "/1/pandas/core/api.py", line 1, in <module>
from pandas._libs import (
File "/1/pandas/_libs/__init__.py", line 18, in <module>
from pandas._libs.interval import Interval
File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
File "pandas/_libs/hashtable.pyx", line 1, in init pandas._libs.hashtable
File "pandas/_libs/missing.pyx", line 1, in init pandas._libs.missing
File "/1/pandas/_libs/tslibs/__init__.py", line 40, in <module>
from pandas._libs.tslibs.conversion import localize_pydatetime
File "pandas/_libs/tslibs/conversion.pyx", line 1, in init pandas._libs.tslibs.conversion
File "pandas/_libs/tslibs/offsets.pyx", line 1, in init pandas._libs.tslibs.offsets
File "pandas/_libs/tslibs/timestamps.pyx", line 1, in init pandas._libs.tslibs.timestamps
File "pandas/_libs/tslibs/tzconversion.pyx", line 55, in init pandas._libs.tslibs.tzconversion
ValueError: Buffer dtype mismatch, expected 'const int64_t' but got 'long long'
Yikes; looks like more Cython trouble. I have basically zero Cython experience, unfortunately. Maybe we need to tell Cython we're cross compiling for a 32-bit platform (wasm32-wasi
) somehow?
I'm also unfamiliar with Cython. The ctypes
package is imported in numpy and seems to work, but it's failing in pandas. Are you familiar with that at all?
Yeah, my guess is that ctypes
works just fine on wasm32-wasi
, and I assume numpy
uses it in a pretty conventional way to interfaces with the parts written in C++. pandas
, on the other hand, adds Cython to the mix, and Cython converts Python-ish code to C/C++. My guess is that Cython also assumes that the C/C++ code it's generating will be compiled for the native platform it's running on rather than be cross-compiled to wasm32-wasi
, so any platform-specific parts of that generated code are for the wrong platform.
I'm just guessing, though: I haven't dug into it.
ctypes can be imported so you can get at the constants in the file, but not all the functionality is available, e.g. dynamic loading and execution of other code.
Last updated: Dec 23 2024 at 12:05 UTC