Stream: git-wasmtime

Topic: wasmtime / issue #10286 Investigate removing symbol table...

Wasmtime GitHub notifications bot (Feb 24 2025 at 21:44):

alexcrichton opened issue #10286:

In https://github.com/bytecodealliance/wasmtime/pull/10285 it was shown that sections such as .symtab, .strtab, and .shstrtab account for a non-trivial portion of the size of an object file:

$ objdump --section-headers target/wasm32-wasip1/release/hello-wasm-world-0x00.cwasm

target/wasm32-wasip1/release/hello-wasm-world-0x00.cwasm:       file format elf64-littleriscv

Sections:
Idx Name              Size     VMA              Type
...
  3 .text             00011bdc 0000000000000000
...
  9 .symtab           00001788 0000000000000000
 10 .strtab           000040f0 0000000000000000
 11 .shstrtab         00000089 0000000000000000

They're almost 25% of the size of the .text section itself, wow!

These sections are occasionally useful to have in a *.cwasm when profiling. Native profilers/debuggers can often use these sections and think that a dynamic library is loaded here when the profiler doesn't otherwise have native jit integration (e.g. via perfmap, jitdump, etc). In that sense I think it's useful to still emit these by default, but we should have a Config::* option for removing them.

In theory it should be possible to just directly remove them entirely and nothing should break in Wasmtime itself. Famous last words though. I'm not 100% sure how we'd do this but I suspect we'd use the object crate a bit differently and maybe assign the same empty string to all symbol names or somehow not assign symbols at all (I'm not sure if that's possible?).

If something in Wasmtime does break from the removal of these symbols we should fix that to not rely on symbol names and instead use something in the compiled metadata here and there instead. For example function indices, offsets, etc. Exactly what the solution might look like depends on what's using a symbol (if any) and in what context it's using it.

Wasmtime GitHub notifications bot (Feb 24 2025 at 21:45):

alexcrichton commented on issue #10286:

cc @posborne

Wasmtime GitHub notifications bot (Feb 27 2025 at 17:58):

alexcrichton added the wasmtime:code-size label to Issue #10286.

Wasmtime GitHub notifications bot (Mar 13 2025 at 19:20):

posborne commented on issue #10286:

I did some quick experiments to prove out that for a basic wasm32-wasip1 rust hello world app, things still work fine with a few extra sections stripped. This is against the current main which has the previous round optimizations from #10285 in tree.

Compile for pulley 64 (with opt-level=s):

$ cargo run --release -- compile --target pulley64 -O opt-level=s -O memory-init-cow=n -o hello-wasm-world.cwasm hello-wasm-world.wasm
    Finished `release` profile [optimized] target(s) in 0.43s
     Running `target/release/wasmtime compile --target pulley64 -O opt-level=s -O memory-init-cow=n -o hello-wasm-world.cwasm hello-wasm-world.wasm`

Strip using llvm-strip; the GNU toolchain strip does not appear to like our ELF files, though I didn't dig into it:

$ llvm-strip hello-wasm-world.cwasm \
    --no-strip-all \
    -R .symtab -R .strtab -R .wasmtime.addrmap \
    -o hello-wasm-world-stripped.cwasm

Comparison of sections in base cwasm with stripped:

$ llvm-readelf -S hello-wasm-world.cwasm
There are 12 section headers, starting at offset 0x2e6f8:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .wasmtime.engine  PROGBITS        0000000000000000 000040 00035c 00   A  0   0  1
  [ 2] .wasmtime.bti     PROGBITS        0000000000000000 00039c 000001 00   A  0   0  1
  [ 3] .text             PROGBITS        0000000000000000 00039d 01108e 00   W  0   0  1
  [ 4] .wasmtime.addrmap PROGBITS        0000000000000000 01142b 011c5c 00   A  0   0  1
  [ 5] .wasmtime.traps   PROGBITS        0000000000000000 023087 000ae0 00   A  0   0  1
  [ 6] .rodata.wasm      PROGBITS        0000000000000000 023b67 0019e8 00   A  0   0  1
  [ 7] .name.wasm        PROGBITS        0000000000000000 02554f 0027a6 00   A  0   0  1
  [ 8] .wasmtime.info    PROGBITS        0000000000000000 027cf5 0010f5 00   A  0   0  1
  [ 9] .symtab           SYMTAB          0000000000000000 028df0 001788 18     10 251  8
  [10] .strtab           STRTAB          0000000000000000 02a578 0040f0 00      0   0  1
  [11] .shstrtab         STRTAB          0000000000000000 02e668 000089 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  R (retain), p (processor specific)

$ llvm-readelf -S hello-wasm-world-stripped.cwasm
There are 9 section headers, starting at offset 0x171f8:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .wasmtime.engine  PROGBITS        0000000000000000 000040 00035c 00   A  0   0  1
  [ 2] .wasmtime.bti     PROGBITS        0000000000000000 00039c 000001 00   A  0   0  1
  [ 3] .text             PROGBITS        0000000000000000 00039d 01108e 00   W  0   0  1
  [ 4] .wasmtime.traps   PROGBITS        0000000000000000 01142b 000ae0 00   A  0   0  1
  [ 5] .rodata.wasm      PROGBITS        0000000000000000 011f0b 0019e8 00   A  0   0  1
  [ 6] .name.wasm        PROGBITS        0000000000000000 0138f3 0027a6 00   A  0   0  1
  [ 7] .wasmtime.info    PROGBITS        0000000000000000 016099 0010f5 00   A  0   0  1
  [ 8] .shstrtab         STRTAB          0000000000000000 01718e 000067 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  R (retain), p (processor specific)

The compiled wasm module still functions (for this trivial example). Debugging would not be possible in the same ways, of course. It should be possible to use an approach like is done for generating split debuginfo using objcopy to retain the stripped information.

$ cargo run -- --target pulley64 --allow-precompiled -O memory-init-cow=n hello-wasm-world-stripped.cwasm
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.27s
     Running `target/debug/wasmtime --target pulley64 --allow-precompiled -O memory-init-cow=n hello-wasm-world-stripped.cwasm`
Hello, world!

And comparison of base wasm module size, compiled, and then stripped sizes.

$ ls -alh hello-wasm-world*
-rw-r--r--@ 1 paul.osborne  staff    93K Mar 13 14:12 hello-wasm-world-stripped.cwasm
-rw-r--r--@ 1 paul.osborne  staff   186K Mar 13 14:12 hello-wasm-world.cwasm
-rwxr-xr-x@ 1 paul.osborne  staff    62K Mar 13 12:17 hello-wasm-world.wasm

Wasmtime GitHub notifications bot (Mar 13 2025 at 19:25):

posborne commented on issue #10286:

.wasmtime.addrmap should already be dropped without the strip if using Config::generate_address_map to false but it isn't made configurable by the CLI at present in my quick look. That could be one useful change here.

For stripping the other bits, I do lean toward having this be something that is done by tooling outside of wasmtime itself; without changes to object I think we can't easily completely remove the sections. For delivering compiled modules to constrained environments, I think most projects will already have some tooling for doing post-processing in place.

Wasmtime GitHub notifications bot (Mar 13 2025 at 19:40):

alexcrichton commented on issue #10286:

Nice thanks for testing! This could all perhaps be a -C option for the CLI and a Config::* flag to remove the sections? We could then even hook that up into fuzzing to ensure that all various bits and knobs of functionality work when the sections are stripped.

For generate_address_map I think that's hooked up as -Daddress-map=n, and we should probably do a better job of making CLI options discoverable...

Personally I'd prefer to go the option-to-compilation route rather than post-processing. I wouldn't want to rely on llvm-strip for example since that would constrain us to always producing an ELF (or at least one that llvm-strip understands). While we're unlikely to change from ELF I also like being able to document how to do it with wasmtime itself vs needing external tools.

In the interim though documenting this and maybe filing a feature request with object upstream could make sense? In that we could document llvm-strip should work, ideally test this in CI, and then also document that we'll probably bake in functionality in the future.

Last updated: Apr 18 2025 at 08:04 UTC