Stream: wasi

Topic: Thread Local Storage in WASM

Coulson Liang (Jun 03 2024 at 19:33):

So there are variants of wasi-libc supporting threading now, I'm curious how are Thread Local Storage (TLS) implemented for WebAssembly.
Traditionally, the TLS are managed in the user space by the libpthread, but TLS model like initial-exec won't work for WASM since absolute addressing is not supported in WASM. So I'm curious how currently wasi-libc and the runtime implements TLS, using a dynamic model?

Joel Dice (Jun 03 2024 at 20:44):

@Andrew Brown might know the answer to this.

Cheng Shao (Jun 03 2024 at 20:55):

you might want to take a look at https://github.com/WebAssembly/wasi-libc/blob/main/libc-top-half/musl/src/env/__init_tls.c

Coulson Liang (Jun 03 2024 at 21:38):

Thank you for pointing this out, I just skim through the code, it seems like the critical tls init functions __builtin_wasm_tls_base() and __wasm_init_tls are not present in this repo. Do you know where are these defined?

Cheng Shao (Jun 03 2024 at 21:47):

__builtin_wasm_tls_base() is just global.get __tls_base, see https://github.com/WebAssembly/tool-conventions/blob/main/Linking.md#thread-local-storage

Coulson Liang (Jun 03 2024 at 23:58):

Thank you! I think I have a basic notion of how the wasi-libc and llvm linker work in TLS scenario.
But I still have a confusion that, is the runtime also involved in this process? Is the runtime involved to be aware of the allocation / free of TLS segments, or these are managed transparent to the runtime?

Andrew Brown (Jun 04 2024 at 17:07):

In wasi-threads, all of the TLS machinery is handled by wasi-libc, not the runtime. That being said, TLS is probably one of the most experimental parts of the experimental wasi-threads target, so don't be surprised if things aren't fully fleshed out. There's a better story being developed now in the shared-everything-threads proposal.

Coulson Liang (Jun 04 2024 at 18:46):

Oh okay thank you, that sounds promising! Btw, in the current wasi-libc repo, is there examples of how tls variables are accessed and set?

Andrew Brown (Jun 04 2024 at 19:05):

Nah, but that's fixable: I think if you wanted to experiment with this you could write C code that uses __thread or _ThreadLocal or thread_local (etc.) and try to compile it to WebAssembly. Essentially you could write some tests to show what parts of this experimental support and what parts do not. I started this in #369 but I didn't like something about Makefile integration so I abandoned that approach. But any tests you want to upstream around this would be welcome!

Andrew Brown (Jun 04 2024 at 19:07):

(on second thought, looking at #369 again, it might be worthwhile to revive that to test the TLS side of things!)

Cheng Shao (Jun 04 2024 at 19:28):

here's one quick example of wasm tls on godbolt, hope it helps: https://godbolt.org/z/bf5jn6sdT

Coulson Liang (Jun 05 2024 at 16:28):

@Andrew Brown Hi so when I write code with __thread or thread_local, what compiler should I use? The clang in wasi-sdk wasi-sdk-22.0/bin/clang-18 --target=wasm32-unknown-wasi doesn't seem to understand any of these symbols

Joel Dice (Jun 05 2024 at 17:42):

I'm guessing you'll want to use --target=wasm32-wasi-threads at least; not sure if there are other flags you also need to specify.

Cheng Shao (Jun 05 2024 at 17:51):

you need to pass -pthread as well

Coulson Liang (Jun 06 2024 at 20:05):

__wasm_init_tls(mem);
    __asm__("local.get %0\n"
            "global.set __tls_base\n"
            :: "r"(tls_base));
    return mem;

This part of the wasi-libc __init_tls.c made me somehow confused. I understand what __wasm_init_tls and __tls_base do, but aren't we setting up the TLS for a new thread, and the __tls_base itself is also a thread-specific symbol. Then I'm confused by this "Chicken or the egg" paradox.

Coulson Liang (Jun 07 2024 at 02:39):

Or __tls_base isn't in the TLS block?

Pat Hickey (Jun 07 2024 at 03:20):

In that asm __tls_base is LLVM’s symbol for a wasm global. Globals do not reside in linear memory, and there is one created per module instance, and one module instance per thread (in wasi-threads, this story will change one day but it’s not important now). There is one common linear memory that is imported by each thread instance.

Pat Hickey (Jun 07 2024 at 03:22):

The llvm memory model is such that you can’t write Rust or C code that touches wasm globals, they can only manipulate linear memory, so to manipulate the global it needs that inline assembly. That assembly code is related to wat, but it’s different for historical reasons

Coulson Liang (Jun 08 2024 at 20:31):

Thank you, I see, the __tls_base reside outside the linear memory as module globals.

I still have some confusion with the wasi-libc codebase. I see the __tls_base is defined in the file
libc-top-half/musl/src/thread/wasm32/wasi_thread_start.s, along with a function wasi_thread_start. But I didn't see the pthread_create function call it, I didn't see anything calling it, which is weird.

Rather, the pthread_create is calling wasi-thread-spawn, which is an imported host function. But I didn't see the definition of this host function in wasmtime repo, where can I find it?

Last updated: Nov 22 2024 at 16:03 UTC