Hi there, a fan of wasmtime here. We've been using wasmtime as part of our project for a plugin-like framework. I ran into the problem that a nested call_async
invocation causes the corruption of the stack on resume. That is, I started an instance whose wasm function was called (async), which in turn invokes a host function (async), which, as part of the logic follow, invokes the wasm function of another instance (but the same wasm code). The async call did return, but it looks like the wasm variables on stack are all corrupted... Is it because there is only one "WASM stack" during the async call (although there might be separated host stack created on the fly)? How does one implement such nested async call that a wasm function may end up calling another one through the host function logic? (Or is it possible) Thanks!
TLDR: I wrote a logic that does wasm->host->wasm kinda invocation (with async), while the call returns, the initial caller's stack seems to be corrupted. Is that an expected behavior, or I did something wrong?
Can you detail more what's being corrupted? The host stack? The guest stack? Also, to confirm, you aren't using unsafe rust are you? Or are you using the C API?
It's the guest stack. The local variables' values in wasm function are corrupted. No I'm not using unsafe Rust, just made some standard call_async, but what's interesting in this case is it triggers a host which again triggers call_async.
In this case it may not be the case that the guest module is prepared to be reentrant. Whether or not that works well depends on the guest module and how it was compiled. What language is the guest?
Rust is both for host and guest. Could you elaborate on "is prepared to be reentrant"?
I couldn't find such an info in the documentation, and wonder how wasmtime handles this...
Hm by reentrancy I basically mean that the guest is prepared to be called again when it calls out to the host. For Rust that in theory should be the case though.
Would you be able to share an example reproduction of this issue?
I think it may have something to do with the fact that I share the RuntimeMemory impl for both instances (caller * callee).
Could it be the case that the stack switching fiber mechanism is implemented by allocating in the guest memory?
So if both instances share the memory impl, the guest (wasm) stack may be corrupted?
No, the stacks for fibers are disjoint from guest memories
I'll note though that RuntimeMemory
is an internal implementation detail of Wasmtime, are you using the wasmtime-*
crates as opposed to the wasmtime
crate?
Yeah, basically I'm trying to implement a persisted (virtual) memory for my plugin...
It's kinda strange: after I tried to not share the memory, the corruption goes away.
Oh if you're using the wasmtime-*
internal crates like wasmtime-runtime
then all bets are off
those are intended to be exclusively used by the wasmtime
crate and are not vetted/reviewed for external consumption
(BTW, while the memory was shared, there should be no data race because the caller is suspended right? Also my case is single-threaded..)
I see...
With the wasmtime-runtime
crate, or other internal crates, there's quite a few things that could be going wrong so it's tough to say what the case might be here
So changes are, because I'm overriding the runtime memory, I may corrupt the fiber stack because that might be part of the runtime mem...
*chances
I see. Got it. Thanks for your patience! Now I have more clues of what's going on, and perhaps I should avoid doing that.
At least try not to derive my own RuntimeMemory impl...
https://docs.wasmtime.dev/api/wasmtime/struct.Config.html#method.with_host_stack <- would this be a way to have custom guest/wasm stack allocation?
Indeed!
if that doesn't work though feel free to file an issue
Thanks!
I realize the actual issue is not because of the stack, but the static global: when the call enters the inner one, the new instance is created, but that also triggers some initialization of those static globals in the same memory. Do you know if there is a way to prevent wasmtime runtime from initialize those statics?
It seems that with_host_stack only customize the host stack, not the wasm stack...With this configured, I still see wasm runtime stack is allocated right after my code+statics in the Linear Memory.
I guess the documentation could use some clarification for this.
Sorry but I'll reiterate again that crates below wasmtime
itself are not intended for use. We don't maintain documentation or try to make them usable at all, they're just for internalize organization. We also can't provide much help for them, so if you're using wasmtime-runtime
for example you'll be on your own
Im a bit confused: with_host_stack is a method of wasmtime::Config
Im not using wasmtime-runtime anymore.
oh oops sorry I misread, my apologies!
You're right that only the host stack is configured here, and wasmtime has no control over the guest shadow stack in use, that's entirely up to the guest to manage
I see! That’s consistent with my findings. Thanks!
On guest shadow stack: is there a way to let the stack pointer start with some manual location?
I found the corruption was caused by the fact that the inner callee instance doesn’t know the caller already has a stack allocated (they are the same code sharing the mem)…
Currently I'm not aware of a great way to do this, no
You'd need to read the wasm file and mutate it to inject your own intrinsics to manage the stack pointer
put another way, I'm not aware of any guest language using LLVM that has already solved the problem neatly of how to manage the shadow stack pointer
Got it, thanks for the help!
Till Schneidereit has marked this topic as resolved.
Last updated: Dec 23 2024 at 12:05 UTC