Stream: git-wasmtime

Topic: wasmtime / issue #4980 Cranelift JITModule in multithreaded

Wasmtime GitHub notifications bot (Sep 29 2022 at 15:43):

mokhaled2992 opened issue #4980:

Hi everyone,

This is asking for help/advice. I have JIT struct very similar to the one in the toy demo. I only generate code that builds a wrapper function that only calls a rust function. Mainly following those examples (https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/jit/examples/jit-minimal.rs) and (https://github.com/bytecodealliance/cranelift-jit-demo/pull/60/files). Now I instantiate one single instance for each thread from that JIT struct and use each only in their own thread to call the function through the pointer I get from that respective JITModule instance. The problem is that I'm getting segmentation faults randomly (as in sometimes they happen sometimes not) when I try to call the functions through the returned pointers and this happens only in multi threaded testing, I never had this issue a single time when I tested my code in single threaded environment. Are there any limitations to using the JITModule in multithreaded environments or any common pitfalls that I should watch for? If needed, I can try to write a minimal example (although wouldn't be very different from the samples above).

Wasmtime GitHub notifications bot (Sep 29 2022 at 15:44):

mokhaled2992 edited issue #4980:

Hi everyone,

This is asking for help/advice. I have JIT struct very similar to the one in the toy demo. I only generate code that builds a wrapper function that only calls a rust function. Mainly following those examples (jit-minimal) and (cranelift-call rust function). Now I instantiate one single instance for each thread from that JIT struct and use each only in their own thread to call the function through the pointer I get from that respective JITModule instance. The problem is that I'm getting segmentation faults randomly (as in sometimes they happen sometimes not) when I try to call the functions through the returned pointers and this happens only in multi threaded testing, I never had this issue a single time when I tested my code in single threaded environment. Are there any limitations to using the JITModule in multithreaded environments or any common pitfalls that I should watch for? If needed, I can try to write a minimal example (although wouldn't be very different from the samples above).

Wasmtime GitHub notifications bot (Sep 29 2022 at 15:45):

mokhaled2992 edited issue #4980:

Hi everyone,

This is asking for help/advice. I have JIT struct very similar to the one in the toy demo. I only generate code that builds a wrapper function that only calls a rust function. Mainly following those examples (jit-minimal) and (cranelift-call rust function). Now I instantiate one single instance for each thread from that JIT struct and use each independently only in their own thread to call the function through the pointer I get from that respective JITModule instance. The problem is that I'm getting segmentation faults randomly (as in sometimes they happen sometimes not) when I try to call the functions through the returned pointers and this happens only in multi threaded testing, I never had this issue a single time when I tested my code in single threaded environment. Are there any limitations to using the JITModule in multithreaded environments or any common pitfalls that I should watch for? If needed, I can try to write a minimal example (although wouldn't be very different from the samples above).

Wasmtime GitHub notifications bot (Sep 29 2022 at 15:54):

mokhaled2992 edited issue #4980:

Hi everyone,

This is asking for help/advice. I have JIT struct very similar to the one in the toy demo. I only generate code that builds a wrapper function that only calls a rust function. Mainly following those examples (jit-minimal) and (cranelift-call rust function). Now I instantiate one single instance for each thread from that JIT struct and use each independently only in their own thread to call the function through the pointer I get from that respective JITModule instance. The problem is that I'm getting segmentation faults randomly (as in sometimes they happen sometimes not) when I try to call the functions through the returned pointers and this happens only in multi threaded testing, I never had this issue a single time when I tested my code in single threaded environment. Are there any limitations to using the JITModule in multithreaded environments or any common pitfalls that I should watch for? If needed, I can try to write a minimal example (although wouldn't be very different from the samples above).

I even tried without calling free_memory at all and still the same problem (only when running in multiple threads as mentioned)

Wasmtime GitHub notifications bot (Sep 29 2022 at 16:10):

mokhaled2992 edited issue #4980:

Hi everyone,

This is asking for help/advice. I have JIT struct very similar to the one in the toy demo. I only generate code that builds a wrapper function that only calls a rust function. Mainly following those examples (jit-minimal) and (cranelift-call rust function). Now I instantiate one single instance for each thread from that JIT struct and use each independently only in their own thread to call the function through the pointer I get from that respective JITModule instance. The problem is that I'm getting segmentation faults randomly (as in sometimes they happen sometimes not) when I try to call the functions through the returned pointers and this happens only in multi threaded testing, I never had this issue a single time when I tested my code in single threaded environment. Are there any limitations to using the JITModule in multithreaded environments or any common pitfalls that I should watch for? If needed, I can try to write a minimal example (although wouldn't be very different from the samples above).

I even tried without calling free_memory at all and still the same problem (only when running in multiple threads as mentioned)

Random guessing, Is there anything shared across JITModules? Perhaps something related to thos relocation tables?

Wasmtime GitHub notifications bot (Sep 29 2022 at 16:10):

mokhaled2992 edited issue #4980:

Hi everyone,

This is asking for help/advice. I have JIT struct very similar to the one in the toy demo. I only generate code that builds a wrapper function that only calls a rust function. Mainly following those examples (jit-minimal) and (cranelift-call rust function). Now I instantiate one single instance for each thread from that JIT struct and use each independently only in their own thread to call the function through the pointer I get from that respective JITModule instance. The problem is that I'm getting segmentation faults randomly (as in sometimes they happen sometimes not) when I try to call the functions through the returned pointers and this happens only in multi threaded testing, I never had this issue a single time when I tested my code in single threaded environment. Are there any limitations to using the JITModule in multithreaded environments or any common pitfalls that I should watch for? If needed, I can try to write a minimal example (although wouldn't be very different from the samples above).

I even tried without calling free_memory at all and still the same problem (only when running in multiple threads as mentioned)

Random guessing, Is there anything shared across JITModules? Perhaps something related to those relocation tables?

Wasmtime GitHub notifications bot (Sep 29 2022 at 16:57):

bjorn3 commented on issue #4980:

Nothing is shared between JITModule's. Instantiating a single JITModule per thread should be just fine. Have you tried getting a backtrace and referenced address for the crashes? Two things I could think of is that either somehow two memory mappings got more than 2GB apart from each without causing an immediate panic, or the memory allocator can't handle the fact that we change memory protection for allocated memory in multi threaded scenarios. Could you try enabling the selinux-fix feature to see if that fixes it? This switches from malloc to mmap.

Wasmtime GitHub notifications bot (Sep 29 2022 at 18:11):

mokhaled2992 commented on issue #4980:

Thanks for getting back so quickly! I will try this feature. I thought about panics but shouldn't I see this before invoking the function pointer? I mean at the steps of compiling/loading? Regarding the addresses, here is what I did already, I print the address I get from get_finalized_function, attach the debugger, get bt from the thread that received the signal. The last address in the bt is the same as the printed one.

Wasmtime GitHub notifications bot (Sep 29 2022 at 18:49):

bjorn3 commented on issue #4980:

The panic I was talking about should happen during finalize_definitions. If it didn't happen at that point, it won't happen at all.

Can you check if the memory at the the function address is mapped as r-x and not as rw-? You can check the /proc/<pid of crashing process>/maps file while the debugger is attached.

Wasmtime GitHub notifications bot (Sep 29 2022 at 18:56):

mokhaled2992 commented on issue #4980:

Yea verifying the permissions sounds like something worth checking. So are you suggesting that somehow setting the proper permissions fails silently? That would be really strange given that the segfault happens randomly, sometimes everything just works out in all threads. But I will check that, I already started a build with selinux-fix.

Last updated: Nov 22 2024 at 16:03 UTC