Stream: git-wasmtime

Topic: wasmtime / PR #4466 Out-of-band fuel metering

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:36):

pepyakin opened PR #4466 from pep-outband-fuel to main:

This is a prototype of the solution for https://github.com/bytecodealliance/wasmtime/issues/4109[^1]. This is very rough and is not intended to be landed as is. Rather, this PR here is to validate that this approach is sensible.

[^1]: I decided to change the name from slacked metering. First, I wanted to avoid mentioning async, so that's why it's not async fuel metering, since that async is orthogonal to the async currently used in wasmtime (as in Func::call_async). At the same time I don't know if slacked conveys the meaning (English is not my mother tongue). So I figured that that "out-of-band metering" is a better name. I contract it to outband in code, I assume it's fine since I saw people using it elsewhere. Please let me know if you have a better name!

Introduction

The regular fuel metering holds the fuel in the vmctx->rumtime_limits->fuel_consumed. At the beginning of each function, the value is loaded into a local variable. Roughly every basic block the value is increased with the cost of that basic block. The value is checked for overflow at function entries and loop headers. If fuel overflowed, then a certain libcall handles it. Before leaving a function (normally, through a call or before a trap), the fuel is dumped into the VMRuntimeLimits.

WIth the out-of-band fuel metering, the fuel is now promoted to a dedicated register tapping to the pinned_reg cranelift feature. The value is still increased every basic block, but the value does not leave the pinned register within wasm. Only at the wasm-host boundaries, i.e. trampolines & libcalls (not implemented as of this PR, coming later), is the fuel value loaded in or flushed from the pinned register into the VMRuntimeLimits.

Also, no checks are performed in the wasm. The checks are meant to be performed either when crossing the wasm-host boundary or asynchronously. Specifically, on Linux, the check is performed by sending a signal each, e.g., 1ms. The signal handler checks if the signal came from the wasmtime (on a best effort basis) and if the program counter points at some the JIT code. If it does, then that means the pinned register holds the currently consumed fuel value. If the fuel value is overflown, we bail out unwinding the wasm stack.

This kind of mechanism showed a great improvement in performance on our tests while still being deterministic as long as the in-wasm state is irrelevant in the case of the OOG.

Now, the prototype here right now targets x86_64 Linux. There is a plan to support aarch64 and macOS. Windows should also be possible to implement. The prototype does not support async. It would be great to support it, but additional work is required.

Implementation Notes and Rationale

Mutex

Right now, before entering we save the tid of the calling thread. This is because theoretically the store can be called from different threads. I also wanted to prepare for the async: potentially the future can be polled on any thread, with each fiber switch we can find ourselves on a new thread.

The problem is with Linux, it turns out that sending signals is a bit of a hassle. The signal's sender cannot know if the destination thread is dead or alive. Moreover, the tid can theoretically be reused and thus a signal could be sent to the wrong thread.

At first, I thought it might be a problem performance-wise, but now I don't think so. The reason is: that the mutex does not get too contended. The mutex is taken on wasm entry & exit and also during the out-of-band fuel check. The latter also uses try_lock. The interesting case is when the wasm tries to exit to host but the mutex is held: in that case, the exit will be delayed until the out-of-band check request is finished.

rt_tgsigqueueinfo

I resorted to using a raw syscall rt_tgsigqueueinfo on Linux to send the signal.

I thought about using pthread_sigqueue (in constrast to just pthread_kill) because it allows to send a sival. This is helpful to tell if the signal is coming from wasmtime or not. However, turns out that at least glibc does a bunch of syscalls that we probably don't want to have inside of the out-of-band fuel check request. So I decided to go straight for rt_tgsigqueueinfo. It takes the siginfo_t but it seems like the kernel does not use that and passes it as is, so I used this opportunity pass dummy values.

Another potential problem that I am not sure needs to be tackled: the pid is cached during the creation of the out-of-band check handle. This is not entirely correct since theoretically, it can change, but I figured it does not warrant worrying.

Future Work

If this gets a green light, then several things will need to be done in the future:

As I mentioned above it should work on other platforms, namely aarch64, macOS, and possibly Windows.

Then, make it work with async. That is actually a bit tricky. The main problem revolves around handling the yields. Specifically with Linux, if a signal handler interrupted the wasm code and figured it was OOG, it should yield the execution. Not sure how good is the idea to switch fibers from inside a signal handler. With macOS/Windows it's not any better: the check thread should manipulate the target thread so that it's possible to switch the fiber.

In case, that works, we can think of applying the same technique we use here for a high-performance substitution for the epoch interrupts.

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:36):

pepyakin edited PR #4466 from pep-outband-fuel to main:

This is a prototype of the solution for https://github.com/bytecodealliance/wasmtime/issues/4109 [^1]. This is very rough and is not intended to be landed as is. Rather, this PR here is to validate that this approach is sensible.

[^1]: I decided to change the name from slacked metering. First, I wanted to avoid mentioning async, so that's why it's not async fuel metering, since that async is orthogonal to the async currently used in wasmtime (as in Func::call_async). At the same time I don't know if slacked conveys the meaning (English is not my mother tongue). So I figured that that "out-of-band metering" is a better name. I contract it to outband in code, I assume it's fine since I saw people using it elsewhere. Please let me know if you have a better name!

Introduction

The regular fuel metering holds the fuel in the vmctx->rumtime_limits->fuel_consumed. At the beginning of each function, the value is loaded into a local variable. Roughly every basic block the value is increased with the cost of that basic block. The value is checked for overflow at function entries and loop headers. If fuel overflowed, then a certain libcall handles it. Before leaving a function (normally, through a call or before a trap), the fuel is dumped into the VMRuntimeLimits.

WIth the out-of-band fuel metering, the fuel is now promoted to a dedicated register tapping to the pinned_reg cranelift feature. The value is still increased every basic block, but the value does not leave the pinned register within wasm. Only at the wasm-host boundaries, i.e. trampolines & libcalls (not implemented as of this PR, coming later), is the fuel value loaded in or flushed from the pinned register into the VMRuntimeLimits.

Also, no checks are performed in the wasm. The checks are meant to be performed either when crossing the wasm-host boundary or asynchronously. Specifically, on Linux, the check is performed by sending a signal each, e.g., 1ms. The signal handler checks if the signal came from the wasmtime (on a best effort basis) and if the program counter points at some the JIT code. If it does, then that means the pinned register holds the currently consumed fuel value. If the fuel value is overflown, we bail out unwinding the wasm stack.

This kind of mechanism showed a great improvement in performance on our tests while still being deterministic as long as the in-wasm state is irrelevant in the case of the OOG.

Now, the prototype here right now targets x86_64 Linux. There is a plan to support aarch64 and macOS. Windows should also be possible to implement. The prototype does not support async. It would be great to support it, but additional work is required.

Implementation Notes and Rationale

Mutex

Right now, before entering we save the tid of the calling thread. This is because theoretically the store can be called from different threads. I also wanted to prepare for the async: potentially the future can be polled on any thread, with each fiber switch we can find ourselves on a new thread.

The problem is with Linux, it turns out that sending signals is a bit of a hassle. The signal's sender cannot know if the destination thread is dead or alive. Moreover, the tid can theoretically be reused and thus a signal could be sent to the wrong thread.

At first, I thought it might be a problem performance-wise, but now I don't think so. The reason is: that the mutex does not get too contended. The mutex is taken on wasm entry & exit and also during the out-of-band fuel check. The latter also uses try_lock. The interesting case is when the wasm tries to exit to host but the mutex is held: in that case, the exit will be delayed until the out-of-band check request is finished.

rt_tgsigqueueinfo

I resorted to using a raw syscall rt_tgsigqueueinfo on Linux to send the signal.

I thought about using pthread_sigqueue (in constrast to just pthread_kill) because it allows to send a sival. This is helpful to tell if the signal is coming from wasmtime or not. However, turns out that at least glibc does a bunch of syscalls that we probably don't want to have inside of the out-of-band fuel check request. So I decided to go straight for rt_tgsigqueueinfo. It takes the siginfo_t but it seems like the kernel does not use that and passes it as is, so I used this opportunity pass dummy values.

Another potential problem that I am not sure needs to be tackled: the pid is cached during the creation of the out-of-band check handle. This is not entirely correct since theoretically, it can change, but I figured it does not warrant worrying.

Future Work

If this gets a green light, then several things will need to be done in the future:

As I mentioned above it should work on other platforms, namely aarch64, macOS, and possibly Windows.

Then, make it work with async. That is actually a bit tricky. The main problem revolves around handling the yields. Specifically with Linux, if a signal handler interrupted the wasm code and figured it was OOG, it should yield the execution. Not sure how good is the idea to switch fibers from inside a signal handler. With macOS/Windows it's not any better: the check thread should manipulate the target thread so that it's possible to switch the fiber.

In case, that works, we can think of applying the same technique we use here for a high-performance substitution for the epoch interrupts.

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:37):

pepyakin edited PR #4466 from pep-outband-fuel to main:

This is a prototype of the solution for https://github.com/bytecodealliance/wasmtime/issues/4109 [^1]. This is very rough and is not intended to be landed as is. Rather, this PR here is to validate that this approach is sensible.

[^1]: I decided to change the name from slacked metering. First, I wanted to avoid mentioning async, so that's why it's not async fuel metering, since that async is orthogonal to the async currently used in wasmtime (as in Func::call_async). At the same time I don't know if slacked conveys the meaning (English is not my mother tongue). So I figured that that "out-of-band metering" is a better name. I contract it to outband in code, I assume it's fine since I saw people using it elsewhere. Please let me know if you have a better name!

Introduction

The regular fuel metering holds the fuel in the vmctx->rumtime_limits->fuel_consumed. At the beginning of each function, the value is loaded into a local variable. Roughly every basic block the value is increased with the cost of that basic block. The value is checked for overflow at function entries and loop headers. If fuel overflowed, then a certain libcall handles it. Before leaving a function (normally, through a call or before a trap), the fuel is dumped into the VMRuntimeLimits.

WIth the out-of-band fuel metering, the fuel is now promoted to a dedicated register tapping to the pinned_reg cranelift feature. The value is still increased every basic block, but the value does not leave the pinned register within wasm. Only at the wasm-host boundaries, i.e. trampolines & libcalls (not implemented as of this PR, coming later), is the fuel value loaded in or flushed from the pinned register into the VMRuntimeLimits.

Also, no checks are performed in the wasm. The checks are meant to be performed either when crossing the wasm-host boundary or asynchronously. Specifically, on Linux, the check is performed by sending a signal each, e.g., 1ms. The signal handler checks if the signal came from the wasmtime (on a best effort basis) and if the program counter points at some the JIT code. If it does, then that means the pinned register holds the currently consumed fuel value. If the fuel value is overflown, we bail out unwinding the wasm stack.

This kind of mechanism showed a great improvement in performance on our tests while still being deterministic as long as the in-wasm state is irrelevant in the case of the OOG.

Now, the prototype here right now targets x86_64 Linux. There is a plan to support aarch64 and macOS. Windows should also be possible to implement. The prototype does not support async. It would be great to support it, but additional work is required.

Implementation Notes and Rationale

Mutex

Right now, before entering we save the tid of the calling thread. This is because theoretically the store can be called from different threads. I also wanted to prepare for the async: potentially the future can be polled on any thread, with each fiber switch we can find ourselves on a new thread.

The problem is with Linux, it turns out that sending signals is a bit of a hassle. The signal's sender cannot know if the destination thread is dead or alive. Moreover, the tid can theoretically be reused and thus a signal could be sent to the wrong thread.

At first, I thought it might be a problem performance-wise, but now I don't think so. The reason is: that the mutex does not get too contended. The mutex is taken on wasm entry & exit and also during the out-of-band fuel check. The latter also uses try_lock. The interesting case is when the wasm tries to exit to host but the mutex is held: in that case, the exit will be delayed until the out-of-band check request is finished.

rt_tgsigqueueinfo

I resorted to using a raw syscall rt_tgsigqueueinfo on Linux to send the signal.

I thought about using pthread_sigqueue (in constrast to just pthread_kill) because it allows to send a sival. This is helpful to tell if the signal is coming from wasmtime or not. However, turns out that at least glibc does a bunch of syscalls that we probably don't want to have inside of the out-of-band fuel check request. So I decided to go straight for rt_tgsigqueueinfo. It takes the siginfo_t but it seems like the kernel does not use that and passes it as is, so I used this opportunity pass dummy values.

Another potential problem that I am not sure needs to be tackled: the pid is cached during the creation of the out-of-band check handle. This is not entirely correct since theoretically, it can change, but I figured it does not warrant worrying.

Future Work

If this gets a green light, then several things will need to be done in the future:

As I mentioned above it should work on other platforms, namely aarch64, macOS, and possibly Windows.

Then, make it work with async. That is actually a bit tricky. The main problem revolves around handling the yields. Specifically with Linux, if a signal handler interrupted the wasm code and figured it was OOG, it should yield the execution. Not sure how good is the idea to switch fibers from inside a signal handler. With macOS/Windows it's not any better: the check thread should manipulate the target thread so that it's possible to switch the fiber.

In case, that works, we can think of applying the same technique we use here for a high-performance substitution for the epoch interrupts.

Big thanks to @alexcrichton & @cfallin for their great support !

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:40):

pepyakin edited PR #4466 from pep-outband-fuel to main:

This is a prototype of the solution for https://github.com/bytecodealliance/wasmtime/issues/4109 [^1]. This is very rough and is not intended to be landed as is. Rather, this PR here is to validate that this approach is sensible and to source some preliminary feedback.

[^1]: I decided to change the name from slacked metering. First, I wanted to avoid mentioning async, so that's why it's not async fuel metering, since that async is orthogonal to the async currently used in wasmtime (as in Func::call_async). At the same time I don't know if slacked conveys the meaning (English is not my mother tongue). So I figured that that "out-of-band metering" is a better name. I contract it to outband in code, I assume it's fine since I saw people using it elsewhere. Please let me know if you have a better name!

Introduction

The regular fuel metering holds the fuel in the vmctx->rumtime_limits->fuel_consumed. At the beginning of each function, the value is loaded into a local variable. Roughly every basic block the value is increased with the cost of that basic block. The value is checked for overflow at function entries and loop headers. If fuel overflowed, then a certain libcall handles it. Before leaving a function (normally, through a call or before a trap), the fuel is dumped into the VMRuntimeLimits.

WIth the out-of-band fuel metering, the fuel is now promoted to a dedicated register tapping to the pinned_reg cranelift feature. The value is still increased every basic block, but the value does not leave the pinned register within wasm. Only at the wasm-host boundaries, i.e. trampolines & libcalls (not implemented as of this PR, coming later), is the fuel value loaded in or flushed from the pinned register into the VMRuntimeLimits.

Also, no checks are performed in the wasm. The checks are meant to be performed either when crossing the wasm-host boundary or asynchronously. Specifically, on Linux, the check is performed by sending a signal each, e.g., 1ms. The signal handler checks if the signal came from the wasmtime (on a best effort basis) and if the program counter points at some the JIT code. If it does, then that means the pinned register holds the currently consumed fuel value. If the fuel value is overflown, we bail out unwinding the wasm stack.

This kind of mechanism showed a great improvement in performance on our tests while still being deterministic as long as the in-wasm state is irrelevant in the case of the OOG.

Now, the prototype here right now targets x86_64 Linux. There is a plan to support aarch64 and macOS. Windows should also be possible to implement. The prototype does not support async. It would be great to support it, but additional work is required.

Implementation Notes and Rationale

Mutex

Right now, before entering we save the tid of the calling thread. This is because theoretically the store can be called from different threads. I also wanted to prepare for the async: potentially the future can be polled on any thread, with each fiber switch we can find ourselves on a new thread.

The problem is with Linux, it turns out that sending signals is a bit of a hassle. The signal's sender cannot know if the destination thread is dead or alive. Moreover, the tid can theoretically be reused and thus a signal could be sent to the wrong thread.

At first, I thought it might be a problem performance-wise, but now I don't think so. The reason is: that the mutex does not get too contended. The mutex is taken on wasm entry & exit and also during the out-of-band fuel check. The latter also uses try_lock. The interesting case is when the wasm tries to exit to host but the mutex is held: in that case, the exit will be delayed until the out-of-band check request is finished.

rt_tgsigqueueinfo

I resorted to using a raw syscall rt_tgsigqueueinfo on Linux to send the signal.

I thought about using pthread_sigqueue (in constrast to just pthread_kill) because it allows to send a sival. This is helpful to tell if the signal is coming from wasmtime or not. However, turns out that at least glibc does a bunch of syscalls that we probably don't want to have inside of the out-of-band fuel check request. So I decided to go straight for rt_tgsigqueueinfo. It takes the siginfo_t but it seems like the kernel does not use that and passes it as is, so I used this opportunity pass dummy values.

Another potential problem that I am not sure needs to be tackled: the pid is cached during the creation of the out-of-band check handle. This is not entirely correct since theoretically, it can change, but I figured it does not warrant worrying.

Future Work

If this gets a green light, then several things will need to be done in the future:

As I mentioned above it should work on other platforms, namely aarch64, macOS, and possibly Windows.

Then, make it work with async. That is actually a bit tricky. The main problem revolves around handling the yields. Specifically with Linux, if a signal handler interrupted the wasm code and figured it was OOG, it should yield the execution. Not sure how good is the idea to switch fibers from inside a signal handler. With macOS/Windows it's not any better: the check thread should manipulate the target thread so that it's possible to switch the fiber.

In case, that works, we can think of applying the same technique we use here for a high-performance substitution for the epoch interrupts.

Big thanks to @alexcrichton & @cfallin for their great support !

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:45):

pepyakin submitted PR review.

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:45):

pepyakin created PR review comment:

The platform-specific code here and elsewhere in this PR relies on quite some C&P.

I wondered if it would be a good idea to reshuffle the low-level code into a separate sys module. It will then contain the platform-specific parts for traphandlers, out-of-band fuel and whatnot.

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:46):

pepyakin submitted PR review.

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:46):

pepyakin created PR review comment:

Alternatively, we could use a magic constant.

Wasmtime GitHub notifications bot (Jul 19 2022 at 17:47):

pepyakin edited PR review comment.

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton submitted PR review.

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton submitted PR review.

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton created PR review comment:

Is this function ever executed with outband_fuel? This seems like a confusing implementation of this function since the purpose of the pinned register is that it's always valid and doesn't need reloading.

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton created PR review comment:

Similar to fuel_load_into_var I think this function may never be executed if outband_fuel is true?

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton created PR review comment:

Mind leaving a compile_error! for now?

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton created PR review comment:

Should this also check fuel during spilling? Otherwise I think nondeterminism might creep in where host functions could execute when the wasm module has no fuel left?

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton created PR review comment:

Similar to above I think that fuel_check is never executed with outband_fuel?

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton created PR review comment:

Oh I see that this is a slightly different query, but could the query be passed through a function parameter? Something like must_be_trap: bool but probably with an enum of some kind to be more descriptive.

Wasmtime GitHub notifications bot (Jul 22 2022 at 15:34):

alexcrichton created PR review comment:

Can this be folded into the preexisting trap initialization?

Last updated: Dec 23 2024 at 12:05 UTC