wasmtime / issue #8697 New WASI thread doesn't start runn... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / issue #8697 New WASI thread doesn't start runn...

Wasmtime GitHub notifications bot (May 28 2024 at 16:12):

MrSmith33 opened issue #8697:

I'm trying to figure out how to start multiple threads and syncronize them with a mutex that uses atomics and wait/notify.

I made a minimal example that does not work:
Main thread spawns a new thread, then waits for u32 at offset 0 to change from 0 to something else.
New thread executes unreachable, to see if it runs at all.
Currently, the wasmtime process runs infinitely, and doesn't reach unreachable. Its memory consumption is always increasing.
;; wasmtime --wasi threads=y wasi_threads_small.wat
(module $wasi_threads_small
  (type (;0;) (func (param i32) (result i32)))
  (type (;1;) (func))
  (type (;2;) (func (param i32 i32)))
  (import "wasi" "thread-spawn" (func $thread_spawn (type 0)))
  (func $_start (type 1)
    i32.const 0
    call $thread_spawn
    drop

    i32.const 0
    i32.const 0
    i64.const -1
    memory.atomic.wait32

    unreachable)
  (func $wasi_thread_start (type 2) (param i32 i32)
    unreachable ;; this is not getting triggered
  )
  (memory (;0;) 17 1024 shared)
  (export "_start" (func $_start))
  (export "wasi_thread_start" (func $wasi_thread_start))
  (start $_start))
I'm testing on Windows 10, with latest version wasmtime-cli 21.0.1 (cedf9aa0f 2024-05-22)

Wasmtime GitHub notifications bot (May 28 2024 at 16:37):

bjorn3 commented on issue #8697:

The issue is the (start $_start) you wrote. This will cause the _start function to be called every time the module is initialized before anything else runs. wasi-threads initializes a completely new instance of the module for every spawned thread. Sharing nothing but the linear memory it imports. As such every thread will try to spawn a new thread and wait indefinitively before wasi_thread_start can be reached. Eventually this will cause you to run out of memory. The solution is to remove the (start $_start). Wasi will already call an exported function called _start on the main thread and the main thread only.

Wasmtime GitHub notifications bot (May 28 2024 at 19:51):

MrSmith33 commented on issue #8697:

Thanks, that was important bit of information.

The rest of my problems seem to come from lack of __stack_pointer setup in the new thread when using D compiler. I need to replicate this code in my executable somehow: https://github.com/WebAssembly/wasi-threads/blob/main/test/testsuite/wasi_thread_spawn.S. Is clang required to compile this .s file?

Am I correct that stack grows to lower addresses and that globals are instanced per thread?

Wasmtime GitHub notifications bot (May 28 2024 at 19:54):

bjorn3 commented on issue #8697:

Is clang required to compile this .s file?

Yes, this file uses uses a syntax which only clang uses.

Am I correct that stack grows to lower addresses

Yes, with LLVM the stack grows down. If you are writing your own language which compiles to wasm which doesn't need to interoperate with C, you are free to make the stack grow up if you want though.

and that globals are instanced per thread?

Yes, wasm globals are instanced per thread too.

Wasmtime GitHub notifications bot (May 29 2024 at 11:32):

MrSmith33 commented on issue #8697:

I have progress:

Installed fresh clang from https://github.com/llvm/llvm-project/releases/tag/llvmorg-18.1.6 via LLVM-18.1.6-win64.exe

Compiled the asm file with clang wasi_thread_spawn.S -c --target=wasm32-wasi -o wasi_thread_spawn.wasm

To link I pass wasi_thread_spawn.wasm as a linker argument via -Lwasi_thread_spawn.wasm

Now I can pass new stack pointer to thread-spawn, and the wasi_thread_start function sets the __stack_pointer

After that __wasi_thread_start_C is called, which is defined in the D code.

Here where the next problem is:

Initial memory size is 0x110000

I increase it by 1MiB + 1 page to 0x220000 in the main thread

New thread is spawned, stack pointer is set to 0x210000

Invalid memory acces in the new thread occurs:
Error: error while executing at wasm backtrace:
    0: 0x2ad0 - testsuite.wasm!__wasi_thread_start_C
    1: 0x2b6e - testsuite.wasm!wasi_thread_start
Caused by:
    0: memory fault at wasm address 0x20fff0 in linear memory of size 0x110000
    1: wasm trap: out of bounds memory access
Which makes me think that new thread is not seeing increased memory size for some reason.

Wasmtime GitHub notifications bot (May 29 2024 at 11:35):

MrSmith33 edited a comment on issue #8697:

I have progress:

Installed fresh clang from https://github.com/llvm/llvm-project/releases/tag/llvmorg-18.1.6 via LLVM-18.1.6-win64.exe

Compiled the asm file with clang wasi_thread_spawn.S -c --target=wasm32-wasi -o wasi_thread_spawn.wasm

To link I pass wasi_thread_spawn.wasm as a linker argument via -Lwasi_thread_spawn.wasm

Now I can pass new stack pointer to thread-spawn, and the wasi_thread_start function sets the __stack_pointer

After that __wasi_thread_start_C is called, which is defined in the D code.

Here where the next problem is:

Max memory size is set to 64MiB with -L--max-memory=67108864

Memory looks like (memory (;0;) 17 1024 shared)

Initial memory size is 0x110000

I increase it by 1MiB + 1 page to 0x220000 in the main thread

New thread is spawned, stack pointer is set to 0x210000

Invalid memory acces in the new thread occurs:
Error: error while executing at wasm backtrace:
    0: 0x2ad0 - testsuite.wasm!__wasi_thread_start_C
    1: 0x2b6e - testsuite.wasm!wasi_thread_start
Caused by:
    0: memory fault at wasm address 0x20fff0 in linear memory of size 0x110000
    1: wasm trap: out of bounds memory access
Which makes me think that new thread is not seeing increased memory size for some reason.

Wasmtime GitHub notifications bot (May 29 2024 at 17:55):

MrSmith33 commented on issue #8697:

I figured it out:

First I tried an example program once more

I found that the difference was in the memory definition, memory is both imported and exported at the same time in the example:
wasm (memory (export "memory") (import "foo" "bar") 1 1 shared)

My code only had export, no import:
wasm (memory (;0;) 34 1024 shared) (export "memory" (memory 0))

So I passed --import-memory wasm-ld linker flag to import memory.

Passing --import-memory disables exporting, so I had to pass --export-memory explicitly too.

Now my program contains:
wasm (import "env" "memory" (memory (;0;) 34 1024 shared)) (export "memory" (memory 0))

Now everything works as expected. I didn't find where documentation mentions the neccesity of marking memory as imported in addition to exporting and sharing.

Wasmtime GitHub notifications bot (May 29 2024 at 18:10):

bjorn3 commented on issue #8697:

I didn't find where documentation mentions the neccesity of marking memory as imported in addition to exporting and sharing.

It is documented that imported objects (including memories) will be shared between threads: https://github.com/WebAssembly/wasi-threads/blob/6b4e2e50a3929d9ebc3b1a54e36ab6f0c0ebc677/README.md?plain=1#L141-L142 It is not explicitly stated that you have to import the memory to behave the way most people would expect.

Wasmtime GitHub notifications bot (May 31 2024 at 16:02):

MrSmith33 closed issue #8697:

I'm trying to figure out how to start multiple threads and syncronize them with a mutex that uses atomics and wait/notify.

I made a minimal example that does not work:
Main thread spawns a new thread, then waits for u32 at offset 0 to change from 0 to something else.
New thread executes unreachable, to see if it runs at all.
Currently, the wasmtime process runs infinitely, and doesn't reach unreachable. Its memory consumption is always increasing.
;; wasmtime --wasi threads=y wasi_threads_small.wat
(module $wasi_threads_small
  (type (;0;) (func (param i32) (result i32)))
  (type (;1;) (func))
  (type (;2;) (func (param i32 i32)))
  (import "wasi" "thread-spawn" (func $thread_spawn (type 0)))
  (func $_start (type 1)
    i32.const 0
    call $thread_spawn
    drop

    i32.const 0
    i32.const 0
    i64.const -1
    memory.atomic.wait32

    unreachable)
  (func $wasi_thread_start (type 2) (param i32 i32)
    unreachable ;; this is not getting triggered
  )
  (memory (;0;) 17 1024 shared)
  (export "_start" (func $_start))
  (export "wasi_thread_start" (func $wasi_thread_start))
  (start $_start))
I'm testing on Windows 10, with latest version wasmtime-cli 21.0.1 (cedf9aa0f 2024-05-22)

Wasmtime GitHub notifications bot (May 31 2024 at 16:03):

MrSmith33 commented on issue #8697:

Was able to remove this extra .wasm file from compilation by using inline assembly:

// When wasi::thread-spawn is called, the WASM runtime calls wasi_thread_start in a new thread
// It sets __stack_pointer of a newly spawned thread and calls __wasi_thread_start_user
// It is important to mark this as naked, otherwise compiler may insert
// a read from __stack_pointer before it gets initialized
@naked export extern(C) void wasi_thread_start(i32 tid, Thread* thread) {
    import ldc.llvmasm;
    __asm("
        local.get 1                # Thread* thread
        i32.load  0#offset         # thread.stackPtr
        global.set __stack_pointer # __stack_pointer = thread.stackPtr
        local.get 0                # tid
        local.get 1                # start_arg
        call __wasi_thread_start_user
        return", "");
        // compiler adds unreachable after call
        // add return to avoid hitting it
}

Other than that piece of code I needed to add 5 extra flags in total to make threads work in D:
Compiler flags:
* -mattr=+bulk-memory,+atomics

Linker flags:
* --shared-memory
* --import-memory
* --export-memory
* --max-memory=67108864

I will close this issue, since everything works. Thanks for your help.

Last updated: Dec 23 2024 at 12:05 UTC