Ted Yin (Feb 15 2024 at 01:54): Ted Yin (Feb 15 2024 at 01:54):Hi there, a fan of wasmtime here. We've been using wasmtime as part of our project for a plugin-like framework. I ran into the problem that a nested call_async invocation causes the corruption of the stack on resume. That is, I started an instance whose wasm function was called (async), which in turn invokes a host function (async), which, as part of the logic follow, invokes the wasm function of another instance (but the same wasm code). The async call did return, but it looks like the wasm variables on stack are all corrupted... Is it because there is only one "WASM stack" during the async call (although there might be separated host stack created on the fly)? How does one implement such nested async call that a wasm function may end up calling another one through the host function logic? (Or is it possible) Thanks!
TLDR: I wrote a logic that does wasm->host->wasm kinda invocation (with async), while the call returns, the initial caller's stack seems to be corrupted. Is that an expected behavior, or I did something wrong?
Alex Crichton (Feb 15 2024 at 05:50):Can you detail more what's being corrupted? The host stack? The guest stack? Also, to confirm, you aren't using unsafe rust are you? Or are you using the C API?
Ted Yin (Feb 15 2024 at 06:27):It's the guest stack. The local variables' values in wasm function are corrupted. No I'm not using unsafe Rust, just made some standard call_async, but what's interesting in this case is it triggers a host which again triggers call_async.
Alex Crichton (Feb 15 2024 at 15:14):In this case it may not be the case that the guest module is prepared to be reentrant. Whether or not that works well depends on the guest module and how it was compiled. What language is the guest?
Ted Yin (Feb 15 2024 at 20:55):Rust is both for host and guest. Could you elaborate on "is prepared to be reentrant"?
Ted Yin (Feb 15 2024 at 20:55):I couldn't find such an info in the documentation, and wonder how wasmtime handles this...
Alex Crichton (Feb 15 2024 at 22:41):Hm by reentrancy I basically mean that the guest is prepared to be called again when it calls out to the host. For Rust that in theory should be the case though.
Would you be able to share an example reproduction of this issue?
Ted Yin (Feb 15 2024 at 23:01):I think it may have something to do with the fact that I share the RuntimeMemory impl for both instances (caller * callee).
Ted Yin (Feb 15 2024 at 23:02):Could it be the case that the stack switching fiber mechanism is implemented by allocating in the guest memory?
Ted Yin (Feb 15 2024 at 23:03):So if both instances share the memory impl, the guest (wasm) stack may be corrupted?
Alex Crichton (Feb 15 2024 at 23:05):No, the stacks for fibers are disjoint from guest memories
Alex Crichton (Feb 15 2024 at 23:06):I'll note though that RuntimeMemory is an internal implementation detail of Wasmtime, are you using the wasmtime-* crates as opposed to the wasmtime crate?
Ted Yin (Feb 15 2024 at 23:11):Yeah, basically I'm trying to implement a persisted (virtual) memory for my plugin...
Ted Yin (Feb 15 2024 at 23:11):It's kinda strange: after I tried to not share the memory, the corruption goes away.
Alex Crichton (Feb 15 2024 at 23:11):Oh if you're using the wasmtime-* internal crates like wasmtime-runtime then all bets are off
Alex Crichton (Feb 15 2024 at 23:12):those are intended to be exclusively used by the wasmtime crate and are not vetted/reviewed for external consumption
Ted Yin (Feb 15 2024 at 23:12):(BTW, while the memory was shared, there should be no data race because the caller is suspended right? Also my case is single-threaded..)
Ted Yin (Feb 15 2024 at 23:13):I see...
Alex Crichton (Feb 15 2024 at 23:13):With the wasmtime-runtime crate, or other internal crates, there's quite a few things that could be going wrong so it's tough to say what the case might be here
Ted Yin (Feb 15 2024 at 23:13):So changes are, because I'm overriding the runtime memory, I may corrupt the fiber stack because that might be part of the runtime mem...
Ted Yin (Feb 15 2024 at 23:13):*chances
Ted Yin (Feb 15 2024 at 23:14):I see. Got it. Thanks for your patience! Now I have more clues of what's going on, and perhaps I should avoid doing that.
Ted Yin (Feb 15 2024 at 23:14):At least try not to derive my own RuntimeMemory impl...
Ted Yin (Feb 15 2024 at 23:28):https://docs.wasmtime.dev/api/wasmtime/struct.Config.html#method.with_host_stack <- would this be a way to have custom guest/wasm stack allocation?
Alex Crichton (Feb 15 2024 at 23:29):Indeed!
Alex Crichton (Feb 15 2024 at 23:30):if that doesn't work though feel free to file an issue
Ted Yin (Feb 15 2024 at 23:51):Thanks!
Ted Yin (Feb 16 2024 at 06:59):I realize the actual issue is not because of the stack, but the static global: when the call enters the inner one, the new instance is created, but that also triggers some initialization of those static globals in the same memory. Do you know if there is a way to prevent wasmtime runtime from initialize those statics?
Ted Yin (Feb 16 2024 at 08:20):It seems that with_host_stack only customize the host stack, not the wasm stack...With this configured, I still see wasm runtime stack is allocated right after my code+statics in the Linear Memory.
Ted Yin (Feb 16 2024 at 08:20):I guess the documentation could use some clarification for this.
Alex Crichton (Feb 16 2024 at 15:55):Sorry but I'll reiterate again that crates below wasmtime itself are not intended for use. We don't maintain documentation or try to make them usable at all, they're just for internalize organization. We also can't provide much help for them, so if you're using wasmtime-runtime for example you'll be on your own
Ted Yin (Feb 16 2024 at 18:16):Im a bit confused: with_host_stack is a method of wasmtime::Config
Ted Yin (Feb 16 2024 at 18:18):Im not using wasmtime-runtime anymore.
Alex Crichton (Feb 16 2024 at 18:20):oh oops sorry I misread, my apologies!
Alex Crichton (Feb 16 2024 at 18:20):You're right that only the host stack is configured here, and wasmtime has no control over the guest shadow stack in use, that's entirely up to the guest to manage
Ted Yin (Feb 16 2024 at 18:22):I see! That’s consistent with my findings. Thanks!
Ted Yin (Feb 16 2024 at 18:23):On guest shadow stack: is there a way to let the stack pointer start with some manual location?
Ted Yin (Feb 16 2024 at 18:24):I found the corruption was caused by the fact that the inner callee instance doesn’t know the caller already has a stack allocated (they are the same code sharing the mem)…
Alex Crichton (Feb 16 2024 at 18:25):Currently I'm not aware of a great way to do this, no
Alex Crichton (Feb 16 2024 at 18:25):You'd need to read the wasm file and mutate it to inject your own intrinsics to manage the stack pointer
Alex Crichton (Feb 16 2024 at 18:26):put another way, I'm not aware of any guest language using LLVM that has already solved the problem neatly of how to manage the shadow stack pointer
Ted Yin (Feb 16 2024 at 18:27):Got it, thanks for the help!
Notification Bot (Sep 12 2024 at 15:38):Till Schneidereit has marked this topic as resolved.
Last updated: Nov 22 2024 at 16:03 UTC