"cannot allocate memory in static TLS block" in dlopen() · wasmtime

Stream: wasmtime

Topic: "cannot allocate memory in static TLS block" in dlopen()

yonil (Mar 29 2023 at 21:05):

Hi,

I have a basic shared library written in rust, that uses wasmtime and only creates the default Engine using let engine = Engine::default().

When I call dlopen("libmy_library.so", RTLD_LAZY); from c++ code, i get the following error from dlerror(): cannot allocate memory in static TLS block

When I remove the call to Engine::default() (and thus, have no remaining dependencies on wasmtime), the library loads successfully.

The same behavior repeats with versions 5.0.0 -> 7.0.0 (haven't tried earlier ones).

Any thoughts on what could be the culprit, or how I should debug/solve this?

Alex Crichton (Mar 29 2023 at 21:08):

I think you may be running into this issue https://github.com/bytecodealliance/wasmtime/issues/3654

Cranelift: timing module for `cranelift-codegen` uses too much TLS space · Issue #3654 · bytecodealliance/wasmtime

.clif Test Case I think this is not relevant because it's an issue about thread-local usage. wasmtime/cranelift/codegen/src/timing.rs Lines 178 to 181 in 8043c1f thread_local! { static CURRENT_PASS...

Alex Crichton (Mar 29 2023 at 21:10):

One possible fix is to shrink the size of the TLS variables in Cranelift, probably with a box, since you're the second one to run into this.

Alex Crichton (Mar 29 2023 at 21:10):

Although as bjorn3 mentions there may be deeper toolchain fixes to apply here too

bjorn3 (Mar 29 2023 at 21:21):

When compiling the library are you doing something that sets the tls model to initial exec? If so using dlopen on it is not allowed. It may happen to work if the library uses little tls space as in that case it is allocated in the static tls block, but this block is tiny and thus will result in an error if more tls space is used, like cranelift-codegen does.

bjorn3 (Mar 29 2023 at 21:22):

Are you using any kind of build tool that may pass extra compile args?

yonil (Mar 29 2023 at 22:22):

bjorn3 said:

Are you using any kind of build tool that may pass extra compile args?

thanks for replying.
i'm using dotnet build with no special arguments

dotnet build command - .NET CLI

The dotnet build command builds a project and all of its dependencies.

yonil (Mar 29 2023 at 22:45):

which, in turn, invokes cargo build --release on the library that references wasmtime

bjorn3 (Mar 30 2023 at 09:14):

Is your .NET code linked into libmy_library.so or is the .NET code loading libmy_library.so? It looks like .NET itself enables -ftls-model=initial-exec for compiling the runtime: https://github.com/dotnet/runtime/blob/57ae91cf6fc5672e34705b1a272cf268761d505a/src/mono/mono/utils/mono-tls.h#L183

runtime/mono-tls.h at 57ae91cf6fc5672e34705b1a272cf268761d505a · dotnet/runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps. - runtime/mono-tls.h at 57ae91cf6fc5672e34705b1a272cf268761d505a · dotnet/runtime

yonil (Mar 31 2023 at 15:41):

thank you for the tip.
AFAICS, that flag doesn't get propagated to the build script that invokes cargo build --release --target-dir <target_dir> --package <package_name>

if this is a known issue that is expected to be fixed in the underlying libraries at some point (hinted by Alex's comments), that's fine, it's not urgent from my perspective. otherwise, if it's by design / isn't planned to be addressed - i would recommend there be some documentation/recommendation on how to overcome it.

Last updated: Oct 23 2024 at 20:03 UTC