Colin D Murphy (Aug 27 2024 at 12:37): Colin D Murphy (Aug 27 2024 at 12:37):I'm trying to get some simple PDF functionality going with WASI. With some very small alterations, I was able to build pdfio. Unfortunately, when I try to build my component:
/opt/wasi-sdk/bin/clang -target wasm32-wasip2 -mexec-model=reactor -D_WASI_EMULATED_MMAN -lwasi-emulated-mman pdf.c hello_world.c pdf_component_type.o libz.a -lpdfio -o hello_world.wasm
I get this error:
wasm-ld: error: /tmp/hello_world-ccb71f.o: undefined symbol: memfd_create
I'm using the latest wasi-sdk DEB. Does it need to be rebuilt so that -D_WASI_EMULATED_MMAN will work? The PDF library is built around having a file descriptor.
Pat Hickey (Aug 27 2024 at 14:56):memfd_create isn’t part of mman emulation, it’s a different sort of Linux interface - it creates an anonymous file that resides in ram as opposed to a filesystem, but behaves as a file. It has a lot of uses in Linux but typically it’s for creating memory shared between multiple processes, which won’t really be applicable in wasi.
Pat Hickey (Aug 27 2024 at 14:58):From your repo it looks like you’re porting this library to wasi so I’d start by examining the way configure is determining whether memfd is available or not and trying to turn it off that way - since this library doesn’t indicate its Linux only it probably has ways to build without memfd
Colin D Murphy (Aug 27 2024 at 15:41):The PDF library needs a filehandle and I have the PDF in memory. I haven't written anything in C since 2012, so I'm unsure how to give it a filehandle to the PDF.
This is true of basically all PDF libraries. They were written to conserve memory for large PDFs, so they allocate small buffers and free them as needed.
If memfd_create won't work, I'm open to anything else.
Colin D Murphy (Aug 27 2024 at 15:42):Thanks for responding BTW
Pat Hickey (Aug 27 2024 at 15:42):ok, understood. in that case you may have to add your own memfd emulation to wasi-libc
Colin D Murphy (Aug 27 2024 at 15:43):OK. I'll give it a shot. Thanks!
Pat Hickey (Aug 27 2024 at 15:43):memfd is a syscall because fds can be sent between linux processes, but since thats not the case in wasi it can be implemented in userland
Pat Hickey (Aug 27 2024 at 15:44):but before diving into that id try to understand what the library falls back on when memfd_create isnt available
Pat Hickey (Aug 27 2024 at 15:45):when you say you have the pdf in memory, is providing it via wasi filesystem an option?
Pat Hickey (Aug 27 2024 at 15:45):is it in the hosts memory or did you already move it to / create it in the guest linear memory
Pat Hickey (Aug 27 2024 at 15:46):if its in the hosts memory you could either write it to a real host filesystem tmp file, or swap out wasmtime-wasi's wasi-filesystem imports for a very limited in-memory fs. if the pdf was created/loaded in the guests memory, you could mount a tmpdir to wasi-filesystem, and write it to the file.
Colin D Murphy (Aug 27 2024 at 15:48):It's being sent via a POST request. It's coming through exports_wasi_http_incoming_handler_handle. So it's already in memory.
Colin D Murphy (Aug 27 2024 at 15:49):This is just a POC. At some point we could probably stream pages.
Pat Hickey (Aug 27 2024 at 15:49):ok, got it. well, your fastest path to a POC is to have a tmpdir in your preopens, write it to a file, and pass it to your pdf lib that way.
Colin D Murphy (Aug 27 2024 at 15:50):OK. So use wasi-filesystem. Does wasmtime provide that out of the box?
Pat Hickey (Aug 27 2024 at 15:50):yeah definitely
Pat Hickey (Aug 27 2024 at 15:51):if you're using the cli, --dir will allow you to provide a host fs dir as a preopen
Pat Hickey (Aug 27 2024 at 15:52):if you wrote your own rust embedding the equivalent is in WasiCtxBuilder
Colin D Murphy (Aug 27 2024 at 15:52):OK. Thanks again.
Last updated: Nov 22 2024 at 16:03 UTC