lum1n0us (Apr 02 2024 at 09:23): lum1n0us (Apr 02 2024 at 09:23):I met an error when generating a component with an executable wasm and a dynamic library wasm via wasm-tools component link, and I am wondering that I might make a mistake and hope some helps here. I am gona list all my steps below and please highlight all my mistakes no matter how minor it is.
it is a simple reproduction of the shared-everything-linking example. Since I am going to use wasi-libc, I just bypass the libc code (which is in the example)
Here is the code
// libzip.h
#include <stdlib.h>
__attribute__((import_module("libzip"), import_name("zip"))) void *
zip(void *in, size_t in_size, size_t *out_size);
// libzip.c
#include "libzip.h"
#include <stdio.h>
#include <string.h>
__attribute__((export_name("zip"))) void *zip(void *in, size_t in_size,
size_t *out_size) {
printf("--> in zip()\n");
void *p = malloc(in_size);
memcpy(p, in, in_size);
*out_size = in_size;
return p;
}
// zipper.c
#include "libzip.h"
#include <stdio.h>
int main() {
printf("-- in zipper main()\n");
void *in = malloc(1024);
size_t out_size = 0;
void *out = zip(in, 512, &out_size);
return out_size == 512 ? 0 : 1;
}
I use /opt/wasi-sdk-21.0/bin/clang -fPIC -shared libzip.so libzip.c to build a shared library wasm and /opt/wasi-sdk-21.0/bin/clang -fPIC -pie -Wl,--import-memory -o zipper.wasm zipper.c to build an executable wasm. (--import-memory is necessary here since all wasm modules in one component are sharing the memory, IIUC)
Then, I tried to use /opt/wasm-tools-1.201.0-x86_64-linux/wasm-tools component link -t -o zipper.component.wat libc.so=/opt/wasi-sdk-21.0/share/wasi-sysroot/lib/wasm32-wasi/libc.so libzipper.so zipper.wasm to create a component (in .wat).
And I got an error:
error: failed to encode a component from modules
Caused by:
0: failed to validate component output
1: missing module instantiation argument named `libzip` (at offset 0x6bc2)
Plus, adding --adapt wasi_snapshot_preview1=./wasi_snapshot_preview1.command.wasm won't fix the error.
Joel Dice (Apr 02 2024 at 16:44):Hi. Looks like you're trying to repro the example from https://github.com/WebAssembly/component-model/blob/main/design/mvp/examples/SharedEverythingDynamicLinking.md, right? That example was written before wasm-tools component link even existed, and the implementation in wit-component (which is what provides wasm-tools component link) works differently than that document describes. Specifically, the inputs are expected to all be dynamic libraries which adhere to https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md. They should all import a memory and function table, and none of them is the "main" module. Instead, wit-component will synthesize a main module which provides the memory and function table which the user-supplied modules import.
Joel Dice (Apr 02 2024 at 16:49):Hypothetically, we could add support to wit-component for allowing the user to provide the "main" module instead of synthesizing it, but that's not currently supported (and I'm not sure how useful it would be beyond the current mode of operation anyway).
Joel Dice (Apr 02 2024 at 16:51):BTW, I also notice that you didn't provide libzip.so as one of the parameters to wasm-tools component link. Was that a deliberate omission? Note that wasm-tools component link does not yet support generating components which import modules, although that would definitely be a useful feature and I plan to add it someday if nobody beats me to it.
lum1n0us (Apr 03 2024 at 02:59):Yes. I am trying to repro the example from https://github.com/WebAssembly/component-model/blob/main/design/mvp/examples/SharedEverythingDynamicLinking.md.
main module is not mandatory. It is just my guess and it's wrong. I have fixed it by building zipper.wasm as a shared library too.
Add libzipper.so to command which now is /opt/wasm-tools-1.201.0-x86_64-linux/wasm-tools component link -t -o zipper.component.wat libc.so=/opt/wasi-sdk-21.0/share/wasi-sysroot/lib/wasm32-wasi/libc.so libzipper.so=libzipper.so zipper.wasm. Run it and the error from wit-component has gone. But more questions are raised.
/opt/wasi-sdk-21.0/bin/clang -fPIC --shared -o zipper.wasm zipper.cand check zipper.wasm. In its @dylink.0 section, needed value is libc.so. But I guess it should be libzipper.so. or zipper. So, is there a way to tell clang how to fill in needed of @dylink.0 ==? (@dylink.0
(mem-info (memory 28 2))
(needed "libc.so") # <== should be "libzipper.so" or "zipper"
)
wit component link doesn't include anything from libzipper.so and zipper.so. It seems it only synthesize necessary main modules but forgot to link. Did I use the wrong command line optioins?
(component
(core module (;0;)
(table (;0;) 1 funcref)
(memory (;0;) 17)
(global (;0;) (mut i32) i32.const 1048576)
(global (;1;) (mut i32) i32.const 1048592)
(global (;2;) (mut i32) i32.const 1114112)
(export "__stack_pointer" (global 0))
(export "__heap_base" (global 1))
(export "__heap_end" (global 2))
(export "__indirect_function_table" (table 0))
(export "memory" (memory 0))
(@producers
(processed-by "wit-component" "0.201.0")
)
)
(core module (;1;)
(type (;0;) (func))
(type (;1;) (func (param i32)))
(import "env" "memory" (memory (;0;) 0))
(import "env" "__indirect_function_table" (table (;0;) 0 funcref))
(func (;0;) (type 0))
(start 0)
(elem (;0;) (i32.const 1) func)
(elem (;1;) (i32.const 1) func)
(data (;0;) (i32.const 1048576) "\00\00\00\00\00\00\10\00")
(@producers
(processed-by "wit-component" "0.201.0")
)
)
(core instance (;0;) (instantiate 0))
(alias core export 0 "memory" (core memory (;0;)))
(core instance (;1;) (instantiate 1
(with "env" (instance 0))
)
)
(@producers
(processed-by "wit-component" "0.201.0")
)
)
P.S.
Exactly same problem happens when I trying with WATs from https://github.com/bytecodealliance/wasm-tools/tree/main/crates/wit-component/tests/components/link.
$ cd wasm-tools/crates/wit-component/tests/components/link
$ wasm-tools parse -o lib-c.wasm lib-c.wat
$ wasm-tools parse -o lib-foo.wasm lib-foo.wat
$ wasm-tools parse -o lib-bar.wasm lib-bar.wat
$ wasm-tools component link -t -o component.trial1.wat c=lib-c.wasm foo=lib-foo.wasm bar=lib-bar.wasm
$ cat component.trail1.wat
(component
(core module (;0;)
(table (;0;) 1 funcref)
(memory (;0;) 17)
(global (;0;) (mut i32) i32.const 1048576)
(global (;1;) (mut i32) i32.const 1048592)
(global (;2;) (mut i32) i32.const 1114112)
(export "__stack_pointer" (global 0))
(export "__heap_base" (global 1))
(export "__heap_end" (global 2))
(export "__indirect_function_table" (table 0))
(export "memory" (memory 0))
(@producers
(processed-by "wit-component" "0.202.0")
)
)
(core module (;1;)
(type (;0;) (func))
(type (;1;) (func (param i32)))
(import "env" "memory" (memory (;0;) 0))
(import "env" "__indirect_function_table" (table (;0;) 0 funcref))
(func (;0;) (type 0))
(start 0)
(elem (;0;) (i32.const 1) func)
(elem (;1;) (i32.const 1) func)
(data (;0;) (i32.const 1048576) "\00\00\00\00\00\00\10\00")
(@producers
(processed-by "wit-component" "0.202.0")
)
)
(core instance (;0;) (instantiate 0))
(alias core export 0 "memory" (core memory (;0;)))
(core instance (;1;) (instantiate 1
(with "env" (instance 0))
)
)
(@producers
(processed-by "wit-component" "0.202.0")
)
)
When generating a component, wasm-tools component link will omit any libraries which it can prove are unreachable (and indeed omit all of them if they're all unreachable, although it should probably report an error in that case). "Unreachable", in this case, means that a library neither exports any functions at the component level, nor does it provide imports to at least one other library which exports such functions, nor are any of its functions reachable via dlopen/dlsym (see the --dl-openable option).
Based on your previous message, I take it the component-level export should be wasi:cli/run@0.2.0 (i.e. you're making a CLI app), thus the main function in zipper.c. And zipper.wasm should be importing functions from libzip.so and/or libc.so, correct? In that case, using the header files for those libraries, building zipper.wasm as a shared library, and using the wasi_snapshot_preview1.command.wasm adapter to convert the WASIp1-style _start export into a WASIp2-style wasi:cli/run@0.2.0 export should be sufficient. No need for __attribute__((import_module(...), import_name(...))) or anything. When you compile and link zipper.wasm, you'll need to add e.g. -lzip and possibly point the linker to where libzip.so is using a -L flag. That will add libzip.so to the @dylink.0 needed entry.
Would you mind pushing what you have to a public repo? I can help debug that, if you like. Also, for reference, there are a bunch of tests which serve as small examples in https://github.com/bytecodealliance/wasm-tools/tree/main/crates/wit-component/tests/components -- the ones whose names start with "link" are the relevant ones.
lum1n0us (Apr 04 2024 at 08:36):Thanks a lot.
I have uploaded them to https://github.com/lum1n0us/about_shared_everything_linking.
The reproduction of the example of SharedEverythingLinking in example is success now. I am wondering:
__attribute__((import_module(""), import_name("zip")))? The demo in link is still failed. Its error are something like:
error: failed to encode a component from modules
Caused by:
0: failed to validate component output
1: missing module instantiation argument named `test:test/test` (at offset 0xaee)
I just opened a PR to fix the link demo: https://github.com/lum1n0us/about_shared_everything_linking/pull/1
Regarding your other question: I think __attribute__((import_module("env"), import_name("zip"))) should work.
lum1n0us (Apr 06 2024 at 08:27):thanks. it helps a lot.
Notification Bot (Apr 06 2024 at 08:27):lum1n0us has marked this topic as resolved.
Last updated: Oct 23 2024 at 20:03 UTC