Stream: wamr

Topic: threadsafe api to terminate WAMR VM from another thread?

Georgii Rylov (Sep 18 2024 at 14:28):

Hi, we use WAMR and I call wasm_runtime_set_exception to stop the vm but I call it from a thread different to the vm thread. With thread sanitiser I detected bunch of data races between wasm_func_call and wasm_runtime_set_exception

Found this example but it seems it's also calling roughly the same non thread safe function https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/samples/terminate/src/main.c#L177

p.s. I asked the same thing in github discussions https://github.com/bytecodealliance/wasm-micro-runtime/discussions/3800#discussion-7202551

Georgii Rylov (Sep 18 2024 at 14:33):

My current idea is:

1. To add an API that can set an atomic flag
2. Insert check for this atomic flag inside HANDLE_OP_END
   and if the flag is set call wasm_runtime_set_exception

But maybe there's already an api that I missed or somebody dealt with the problem in another way?

Notification Bot (Sep 18 2024 at 14:56):

This topic was moved here from #general > threadsafe api to terminate WAMR VM from another thread? by fitzgen (he/him).

Georgii Rylov (Oct 02 2024 at 09:39):

here's the API that works for WASM https://github.com/bytecodealliance/wasm-micro-runtime/compare/main...g0djan:wasm-micro-runtime:godjan/terminate_from_another_thread?expand=1

Georgii Rylov (Oct 02 2024 at 09:47):

but when I try setting exception from a different thread to the VM running AOT it seems like AOT code continues to execute inside of this line after the exception is set. Inside it clears the stack with repetitive aot_free_frame calls before invoke_native is finished

As a result created call stack appears to be empty

@Wenyong Huang do you know why invoke_native continues to execute when set_exception is called from another thread?

Georgii Rylov (Oct 02 2024 at 09:50):

I made a workaround by inserting these lines inside of aot_free_frame

if (aot_copy_exception(module_inst, NULL)) {
  aot_create_call_stack(exec_env);
  aot_dump_call_stack(exec_env, true, NULL, 0);
}

but I believe the root problem is that there's an attempt to clear stack after exception is set and I don't get why it is like that

Wenyong Huang (Oct 08 2024 at 04:39):

@Georgii Rylov thanks for the testing, I think the invoke_native you mentioned is to execute the AOT function, and since the AOT code only checks whether the exception was thrown and whether current thread's terminated flag is set in several places (see the callings to check_exception_thrown in aot_emit_function.c and the callings to check_suspend_flags in core/iwasm/compilation), normally it will continue to execute until these places are met (for check_suspend_flags, we should also set the terminate flag in other thread).

How to safely terminate, suspend and resume wasm threads is really difficult to implement, normally the idea is to stop the world first - let all the threads (except the thread asking to stop) suspend and wait until other thread notify it to run. I ever opened an issue for the feature:
https://github.com/bytecodealliance/wasm-micro-runtime/issues/2319,
and had done some work in branch dev/thread_suspension:
https://github.com/bytecodealliance/wasm-micro-runtime/tree/dev/thread_suspension

But due to time bandwidth, I didn't test a lot, there may be some issues. If you are interested, maybe you can have a try, e.g. by calling the API wasm_runtime_terminate_all in thread_manager.h, and adding --enable-multi-thread flag for wamrc when generating AOT file.

Wenyong Huang (Oct 08 2024 at 04:44):

Please refer to https://github.com/bytecodealliance/wasm-micro-runtime/compare/dev/thread_suspension for the code changes.

Last updated: Nov 22 2024 at 17:03 UTC