wasmtime / issue #11869 Async libcalls "block" the store ... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / issue #11869 Async libcalls "block" the store ...

Wasmtime GitHub notifications bot (Oct 15 2025 at 21:45):

alexcrichton opened issue #11869:

At this time the implementation of async libcalls in Wasmtime all use the block_on helper internally in the implementation. This has the property, though, that the store is "locked" while the libcall is waiting on the result meaning that in a component-model-async world it's not possible to make progress on anything else in the store while this is happening. This notably affects Store::run_concurrent as well where any select-ed async computation won't make progress because the wasm is locking up everything.

To fix this it will require async libcalls to be refactored/reimplemented to not close over the store for the duration of their execution. Instead something like Accessor will be required where mutable access to a store can be temporarily granted but otherwise it's not held across await points. In implementing this it'll fix run_concurrent to correctly and actually run various computations in the provided closure concurrently. This will also enable other concurrent tasks within the store to make progress while a libcall is blocked.

Wasmtime GitHub notifications bot (Oct 15 2025 at 21:45):

alexcrichton added the wasm-proposal:component-model-async label to Issue #11869.

Wasmtime GitHub notifications bot (Feb 02 2026 at 05:29):

alexcrichton commented on issue #11869:

I talked with Luke and Joel about this a bit ago and wanted to write down some notes. Async libcalls right now are:

Things that trigger an async limiter

table growth (all kinds)

memory growth

Things that may trigger GC (async nature of GC, async limiter, etc)

table initialization

growing the gc heap

gc allocation

array new/init data/elem

Fuel running out

Epochs changing

All of these libcalls fall into the category of "the wasm is stuck between two plain/normal wasm instructions". It would be a violation of runtime semantics if Wasmtime were to allow something else to interleave between instructions. For example during an epoch yield it's not valid for Wasmtime to run some other wasm within the store as that could have visible side effects.

Put another way Wasmtime will need to enforce a "lock" where when these async situations are hit it prevents all wasm from continuing to execute. My rough idea for this is that the store, with concurrency enabled, will grow an async-recursive-mutex-of-sorts (probably not literally). This lock will be "bounced on" whenever wasm is entered via the host call or exited via a hostcall returning. The lock is then held across the async operation of an epoch/fuel/async limiters/gc/etc from above. The rough idea is that this is a Option<Vec<Waker>> in the store. If that's None, the lock isn't held. If it's Some, then the lock is held. When the lock is "dropped" then all the wakers are woken (if any).

This'll likely require refactoring some various points within Wasmtime to instrument more entries/exits with async in Rust. Ideally the lock acquisition/bounce/etc are all native async functions. This'll probably require some finesse.

The main learning from this is we can't simply start using run_concurrent (ish) within these libcalls. If we were to do that then we would accidentally allow the situation we don't want, which is executing wasm instructions between other instructions that don't allow interleaving. This means that to fully avoid blocking the store we'll have to add infrastructure, as opposed to just removing a limitation.

Wasmtime GitHub notifications bot (Feb 14 2026 at 22:31):

alexcrichton commented on issue #11869:

Another thought that occurs to me borne out of thoughts/discussion from https://github.com/bytecodealliance/wasmtime/pull/12587 -- preventing wasm execution during an async libcall is not enough, we have to prevent mutation of the entire store. This includes things like executing wasm, but it extends to host-initiated modifications of tables/globals as well. That's a needle I don't know how to thread...

Wasmtime GitHub notifications bot (May 21 2026 at 05:15):

alexcrichton commented on issue #11869:

A finding from https://github.com/bytecodealliance/wasmtime/pull/13424#discussion_r3277475089 -- In that PR it's refactoring table.grow to split up the "grow" and "fill" ops. This means that tables-of-non-nullable-reference-types are temporarily invalid. This isn't a problem to wasm, but can in theory be a problem to the host if we epoch yield during the fill operation and the host is allowed to actually inspect the table. That's not possible today, due to this issue, but is something we'll want to consider when resolving this.

Wasmtime GitHub notifications bot (Jul 24 2026 at 16:57):

fitzgen commented on issue #11869:

Another thought that occurs to me borne out of thoughts/discussion from #12587 -- preventing wasm execution during an async libcall is not enough, we have to prevent mutation of the entire store. This includes things like executing wasm, but it extends to host-initiated modifications of tables/globals as well. That's a needle I don't know how to thread...

Doesn't this just imply that the existing behavior, where the whole store is blocked by an async libcall, is the correct behavior?

Otherwise, when e.g. task A is paused because of an epoch interruption, we have to keep track of the transitive closure of state that task A can reach and keep all of that state "locked", only allowing unreachable-from-task-A state in the store to be accessed/modified. But, in practice, the reachable state is going to be basically the full store's state, because otherwise the embedder would just use multiple stores for disjoint guest state. Other tasks in the store are likely either also inside the same instance, or otherwise share some amount of state with task A, so we can't resume them anyways (that shared state will be locked). Additionally, tracking that reachable state is going to be somewhat ad-hoc, where as locking the whole store (which happens to be the current behavior!) is a principled and very easy to implement/maintain approach.

Wasmtime GitHub notifications bot (Jul 24 2026 at 23:04):

alexcrichton commented on issue #11869:

That's basically true yeah, but the one exception to that rule is the original genesis of this issue -- the AsyncFn passed to run_concurrent. Specifically the code inside of that AsyncFn appears like it's "blocking" meaning that if you were to apply a timeout within that AsyncFn, entirely independent of Wasmtime, it basically won't work. Once wasm does a blocking operation like an async-limit of memory growth the internals of the AsyncFn, timeouts and all, will never fire and are stuck until the blocking operation resolves. (I realize I'm using the term "blocking" here a bit inaccurately...)

Technically the Accessor almost provides the perfect API for handling all of this as well. (which is all that run_concurrent yields to the closure). If Accessor::with were an async function, then we could trivially handle all this because "store is locked" means that function temporarily blocks. Timeouts would still all progress as expected and everything would match embedder expectations. The problem though is that Accessor::with is a synchronous function that should not block, and that's what leaks the ability to modify everything in the store were we to run run_concurrent closures.

The criticality of run_concurrent is debatable IMO. We've got documentation referring to this issue's behavior and we also work around this issue in in-repo situations.

Wasmtime GitHub notifications bot (Jul 28 2026 at 17:36):

fitzgen commented on issue #11869:

Technically the Accessor _almost_ provides the perfect API for handling all of this as well. (which is all that run_concurrent yields to the closure). If Accessor::with were an async function, then we could trivially handle all this because "store is locked" means that function temporarily blocks. Timeouts would still all progress as expected and everything would match embedder expectations. The problem though is that Accessor::with is a synchronous function that should not block, and that's what leaks the ability to modify everything in the store were we to run run_concurrent closures.

Should we add Accessor::with_async in that case? This would let callers yield to the event loop so other futures can progress, while still closing over the store and therefore preventing re-entry into Wasm (i.e. the _async behavior instead of the _concurrent behavior).

Another way to implement that functionality, is to enter a nested event loop via e.g. tokio::runtime::Runtime::block_on. Embedders can actually do that today:
accessor.with(|store| {
    my_tokio_runtime.block_on(async {
        // lock `store` across async operations
        memory.grow_async(store).await?;
        // ...
        Ok(())
    });
});
But Wasmtime itself can't do that because it doesn't know the async runtime being used. We could add some kind of API to register a block_on function with Wasmtime to teach it how to do that though... Of course, I'm not sure this is any easier than implementing Accessor::with_async "correctly".

The criticality of run_concurrent is debatable IMO.

Are you saying you think we should maybe remove run_concurrent and Accessor completely? That seems pretty nuclear, and would remove our ability to multi-task between fibers within a store, no?

Wasmtime GitHub notifications bot (Jul 28 2026 at 17:48):

fitzgen commented on issue #11869:

I guess this would require adding

tls::get_async to hold the &mut dyn VMStore in an async closure

Adding a wait queue for Accessor::with/tls::get on a taken dyn VMStore, instead of panicking

Making Accessor::with/tls::get an async function, even when their closures are not, so that they can wait to run the closure until after it goes through the wait queue and gets the returned dyn VMStore

?

That is... quite a bit of churn, but I think it would handle everything we want?

Wasmtime GitHub notifications bot (Jul 28 2026 at 18:13):

alexcrichton commented on issue #11869:

If adding Accessor::with_async then that would require removing Accessor::with, which to me is the real problem. Async yield points lock down the store meaning with, if it executes, is unsound. An API Accessor::with which panics conditionally depending on a yield point I also think would be too surprising, meaning that admitting with_async to unblock the future in run_concurrent would necessitate removing with. In doing this, however, it'd break what I would imagine are most calls to with, many of which are done in a sync context.

For nested runtimes and block_on, regardless of whether that works that's definitely not the semantics that any runtime wants. The "blocking" behavior described in this issue is only blocking from the perspective of run_concurrent's async closure argument. No literal thread-level OS blocking happens, which is what block_on would introduce.

For the criticality aspect, I don't mean to propose removing run_concurrent or Accessor, instead just continuing to treat this issue as not-a-high-priority and hoping we have better ideas down the road.

That is... quite a bit of churn, but I think it would handle everything we want?

To clarify, you mean for adding Accessor::with_async and removing Accessor::with (morally at least)? If so the churn aspect is, in my opinion, the primary part to handle here. I'm not confident this is a case of "just update callers", and I think there will be quite a number of callers that just can't update.

Wasmtime GitHub notifications bot (Jul 28 2026 at 21:46):

fitzgen commented on issue #11869:

To clarify, you mean for adding Accessor::with_async and removing Accessor::with (morally at least)?

I would say that I am proposing what is morally an async lock around the dyn VMStore pointer we keep in TLS right now. Here is some rough pseudocode:
pub async fn with<R>(&self, f: impl FnOnce(...) -> R) -> R {
    let store = tls::lock().await;
    let result = f(self.token.as_context_mut(store));
    tls::unlock(store);
    result
}

pub async fn try_with<R>(&self, f: impl AsyncFnOnce(...) -> R) -> R {
    let store = tls::try_lock().await;
    let result = f(self.token.as_context_mut(store)).await;
    tls::unlock(store);
    result
}
where tls::lock and tls::unlock are probably actually a single async fn tls::with<R>(f: impl AsyncFnOnce(&mut dyn VMStore) -> R) -> R function where:

when locking if the VMStore has been taken we add ourselves to a wait queue (instead of panicking, which is what is done today) and

when unlocking we wake up the next future in the wait queue if any.

Wasmtime GitHub notifications bot (Jul 28 2026 at 21:47):

fitzgen edited a comment on issue #11869:

To clarify, you mean for adding Accessor::with_async and removing Accessor::with (morally at least)?

I would say that I am proposing what is morally an async lock around the dyn VMStore pointer we keep in TLS right now. Here is some rough pseudocode:
pub async fn with<R>(&self, f: impl FnOnce(...) -> R) -> R {
    let store = tls::lock().await;
    let result = f(self.token.as_context_mut(store));
    tls::unlock(store);
    result
}

pub async fn try_with<R>(&self, f: impl AsyncFnOnce(...) -> R) -> R {
    let store = tls::lock().await;
    let result = f(self.token.as_context_mut(store)).await;
    tls::unlock(store);
    result
}
where tls::lock and tls::unlock are probably actually a single async fn tls::with<R>(f: impl AsyncFnOnce(&mut dyn VMStore) -> R) -> R function where:

when locking if the VMStore has been taken we add ourselves to a wait queue (instead of panicking, which is what is done today) and

when unlocking we wake up the next future in the wait queue if any.

Wasmtime GitHub notifications bot (Jul 28 2026 at 21:47):

fitzgen edited a comment on issue #11869:

To clarify, you mean for adding Accessor::with_async and removing Accessor::with (morally at least)?

I would say that I am proposing what is morally an async lock around the dyn VMStore pointer we keep in TLS right now. Here is some rough pseudocode:
pub async fn with<R>(&self, f: impl FnOnce(...) -> R) -> R {
    let store = tls::lock().await;
    let result = f(self.token.as_context_mut(store));
    tls::unlock(store);
    result
}

pub async fn with_async<R>(&self, f: impl AsyncFnOnce(...) -> R) -> R {
    let store = tls::lock().await;
    let result = f(self.token.as_context_mut(store)).await;
    tls::unlock(store);
    result
}
where tls::lock and tls::unlock are probably actually a single async fn tls::with<R>(f: impl AsyncFnOnce(&mut dyn VMStore) -> R) -> R function where:

when locking if the VMStore has been taken we add ourselves to a wait queue (instead of panicking, which is what is done today) and

when unlocking we wake up the next future in the wait queue if any.

Wasmtime GitHub notifications bot (Jul 28 2026 at 22:05):

alexcrichton commented on issue #11869:

That makes sense, yeah, and implementation-wise that's more-or-less what I'd expect as well (maybe some more store integration, something like that). The problem to overcome though is updating all callers of Accessor::with to be async, and I think that'd be a pretty major change.

Last updated: Jul 29 2026 at 05:03 UTC