Stream: general

Topic: Benchmarking Runtimes

Aarav Dayal (Dec 01 2025 at 20:24):

Mission Accepted. Sounds to me like my first call of duty.

Thomas Trenner (Dec 03 2025 at 07:41):

Alex Crichton schrieb:

You can read up on LLVM's security policy here, notably:

For example, a maliciously crafted C, Rust or bitcode input file can cause arbitrary code to execute in LLVM

And this is NOT considered a security issue in LLVM since LLVM explicitly does not provide this guarantee

In consequence:
If I understand correctly, users of Emscripten (emcc), which leverages LLVM, are tasked with identifying potential CVEs from miscompilations. Is it accurate, then, to say that the fundamental difference between LLVM and Cranelift, in terms of miscompilation-induced CVE responsibility, is that LLVM places this burden on its users, while Cranelift's maintainers are the ones obligated to do so?

bjorn3 (Dec 03 2025 at 10:46):

Bugs in LLVM don't result in sandbox escapes with Emscripten given that Emscripten asks LLVM to output wasm that will run in the sandbox and thus no matter what LLVM outputs for (in)correct wasm, a sandbox escape would be impossible if the wasm engine doesn't have bugs. However if you were to compile malicious wasm to machine code with LLVM inside the wasm engine, that could potentially allow a sandbox escape if LLVM miscompiles something. In other words the difference between using LLVM to produce wasm and to compile was to machine code is whether LLVM is part of the TCB of the wasm engine or not.

Andy Wingo (Dec 03 2025 at 11:06):

Alex Crichton said:

An example of this is that a miscompile in LLVM is not a CVE.

I think this is not precisely right -- the page you link states:

However, a miscompilation where there are clear indications that it can result in the produced binary becoming significantly easier to exploit could be considered security sensitive, and should be reported to the security response group

Granted, it's a bit wishy-washy in the sense that "ease of exploit" depends on a threat model which is out of scope. But even if not deemed to be a CVE, AFAIU miscompilations are always taken seriously -- like we're not arguing that if presented with a miscompilation that LLVM maintainers would say "not my bug", right?

The argument here is not "LLVM is more free to miscompile than wasmtime" -- both are responsible for their output -- but rather "an untrusted Wasm input lowered to LLVM IR may cause the LLVM middle-end to DOS / blow the stack / cause a UAF" -- right?

bjorn3 (Dec 03 2025 at 11:35):

Historically LLVM has considered miscompilations to not be CVEs. Only recently they started having a security policy. And there have been multiple miscompilations reachable in safe rust (and probably also from wasm) that have caused memory safety issues (and thus for wasm almost certainly sandbox escapes) And there are multiple miscompilations that have been open for years like https://github.com/llvm/llvm-project/issues/89885 (when sse is disabled on x86, llvm const eval and runtime behavior mismatch for floats causing bound checks to be incorrectly optimized away as unreachable), https://github.com/rust-lang/rust/issues/70143 (stack probes are done after stack alignment, allowing stack smashing. not reachable for wasm), https://github.com/rust-lang/rust/issues/83060 (stack frames >2GB getting miscompiled. not reachable for wasm)

Alex Crichton (Dec 03 2025 at 15:33):

Thomas Trenner said:

If I understand correctly, users of Emscripten (emcc), which leverages LLVM, are tasked with identifying potential CVEs from miscompilations

I wouldn't personally phrase it as this, as generalizing slightly you're conclusing that all users of LLVM (e.g. C, C++, Rust, etc, developers too) all have to worry about CVEs and practically that's not the status quo. The general reason for that is that when you're compiling someting you're effectively never compiling possibly-malicios code. You're compiling your project + dependencies so any problems along the way are ICEs or "just bugs".

Alex Crichton (Dec 03 2025 at 15:34):

My comments about LLVM are about using it as a backend for compilation in a WebAssembly runtime. In such a scenario you are indeed compiling possibly-malicious code which is where this all comes up and is critically different from "I'm a normal user of a toolchain"

Alex Crichton (Dec 03 2025 at 15:38):

Andy Wingo said:

The argument here is not "LLVM is more free to miscompile than wasmtime" -- both are responsible for their output -- but rather "an untrusted Wasm input lowered to LLVM IR may cause the LLVM middle-end to DOS / blow the stack / cause a UAF" -- right?

I wouldn't phrase it myself like this personally. I'd echo bjorn3 as well and summarize it as:

• In Wasmtime we evaluate every miscompile we find as to whether it's a CVE
• In LLVM I don't believe that every miscompile is evaluated as to whether it could break someone's sandbox
• Both projects, however, fix miscompiles ASAP as far as I know

At least in the realm of miscompiles the main difference to me is how they're reviewed. I was also surprised to see that LLVM might consider some miscompiles as CVEs, that was definitely new to me. I would be pretty surprised though if every single bugfix going into LLVM, for every backend, was reviewed by some team somewhere as to whether it broke their sandbox and should be a CVE. Even if this did happen it would be problematic because that would mean that the bugfix would be public before it was determined to be a CVE, however.

This also doesn't touch on the aspect of compiling itself. We issue CVEs for arbitrary code execution at compile time, and LLVM does not. I believe that's pretty fundamental to LLVM's and Cranelift's design and can't be changed with some simple review.

Chris Fallin (Dec 03 2025 at 16:16):

Yeah, personally I would also focus more on the security of the compiler invocation itself (broadly construed, including DoS) -- LLVM "tries much harder" and as a consequence, is easier than Cranelift to drive into pathological compile times or resource usage with malicious input. Cranelift was born in the context of browser engines where loading a new web page should never cause CPU to spike uncontrollably because one is compiling the (untrusted) code on page

Chris Fallin (Dec 03 2025 at 16:16):

As an example, we're really careful about user input-controlled recursion (ie we avoid it) and in general are more suspicious of superlinear algorithms in Cranelift than LLVM is

Chris Fallin (Dec 03 2025 at 16:17):

That's not to say we're a strict provable linear-time compiler (that's Winch, if you need it) but we try to have a much less spiky compile-time profile, and spikes are viewed as problems to be fixed

Chris Woods (Dec 05 2025 at 22:50):

Aarav Dayal said:

wasmtime seems to be the best bet because of the strong backing by huge corporations.

as compared to the strong backing for LLVM?

Chris Fallin (Dec 05 2025 at 22:58):

Aarav is referring to Wasm engines specifically, I think, not compiler backends (though undoubtedly there are multiple Wasm engines with corporate backing as well)

Chris Woods (Dec 05 2025 at 23:00):

Aarav Dayal said:

Are there better code generators which give the best runtime performance as in my usecase startup times are a lesser priority over runtime performance.

Ok. This thread took a hard left turn at security and kept going.

Bringing it back to performance for a moment, you could compile your Wasm to a runtime and platform specific binary. This would be independent of the execution of the code. So in essence:
Offline / Prep steps

1. Receive Wasm
2. Compile Wasm -> AoT format (you can use LLVM here, in a container / vm etc)

Then... when you need to execute

3. Deploy AoT format to your blockchain VM.

The benefits of this approach are:

1. Smaller VM -> you only need the infrastructure to support the AoT based runtime (for WAMR, as an example, it's about 30-50kb)
2. Prep work is all done off line, as you allow your user to upload their code
3. The Wasm compiler (back to security) would execute in a VM or other framework, isolating it from your infrastructure, so the compiler can spike / die / go on a new years party and drink too much wine, etc... and you can catch it and kill it, rejecting user code.
4. When the code does execute in the VM - it's fast, very fast.

I imagine that this is probably what most of the large scale cloud / server hosts of Wasm do today anyway.

As Ralph said, everyone has a view on what is important. Think about your architecture, and what elements make the most sense for you to measure for performance.

Best of luck with your project!

Aarav Dayal (Dec 05 2025 at 23:01):

Chris Fallin said:

Aarav is referring to Wasm engines specifically, I think, not compiler backends (though undoubtedly there are multiple Wasm engines with corporate backing as well)

giphy.gif

Aarav Dayal (Dec 05 2025 at 23:01):

Okay I will read all of this and then reply

Chris Woods (Dec 05 2025 at 23:01):

Best of luck! :)

Aarav Dayal (Dec 05 2025 at 23:04):

Almost done. Just reading your message now.

Aarav Dayal (Dec 05 2025 at 23:11):

Chris Woods said:

Aarav Dayal said:

Are there better code generators which give the best runtime performance as in my usecase startup times are a lesser priority over runtime performance.

Ok. This thread took a hard left turn at security and kept going.

Bringing it back to performance for a moment, you could compile your Wasm to a runtime and platform specific binary. This would be independent of the execution of the code. So in essence:
Offline / Prep steps

1. Receive Wasm
2. Compile Wasm -> AoT format (you can use LLVM here, in a container / vm etc)

Then... when you need to execute

3. Deploy AoT format to your blockchain VM.

The benefits of this approach are:

1. Smaller VM -> you only need the infrastructure to support the AoT based runtime (for WAMR, as an example, it's about 30-50kb)
2. Prep work is all done off line, as you allow your user to upload their code
3. The Wasm compiler (back to security) would execute in a VM or other framework, isolating it from your infrastructure, so the compiler can spike / die / go on a new years party and drink too much wine, etc... and you can catch it and kill it, rejecting user code.
4. When the code does execute in the VM - it's fast, very fast.

I imagine that this is probably what most of the large scale cloud / server hosts of Wasm do today anyway.

As Ralph said, everyone has a view on what is important. Think about your architecture, and what elements make the most sense for you to measure for performance.

Best of luck with your project!

There are atleast a few obvious issues with this approach:

1. Since each node could be on a different architecture, you can't really just compile a single AoT binary.

2. Running a pre-compiled binary is literally an RCE and I am not sure if there is a good platform-independent way of implement fuel metering etc. especially if the program invokes syscalls, has static links, might even register signal handlers etc. This is literally a security vulnerability.

3. If you instead post the optimized LLVM IR, then we are back to having to use LLVM on each machine which can cause resource hogging or even cause LLVM to crash or result into an RCE.

None the less, thank you for suggesting the sandboxing, I might take a look into this as in theory I can try to get it to work especially if I "really" want to squeeze out as much performance as I can.

Here are a few questions I have:

1. WAMR still runs wasm binaries and I am not sure whether it is still safe to use especially if it uses LLVM for AoT compilation as previously discussed and clarified in this thread. (unless there is some level of sandboxing which if so please kindly let me know.)

Aarav Dayal (Dec 05 2025 at 23:12):

Also Chris thank you for the well wishes.

Chris Fallin (Dec 05 2025 at 23:56):

FWIW, several Wasm engines do implement an AOT workflow -- Wasmtime does, and that's how it's been used in several places in FaaS contexts. The idea is that you have a control plane that runs the compiler and a VM that executes the artifact that results; the key bit is that you control the control-plane, and take measures (e.g. code-signing at least) to protect the integrity of the artifact that comes out of the AOT compiler half of the Wasm engine. You do also need the ability for your control plane to build the artifact for each architecture and configuration you run (Wasmtime embeds the config used for compilation in the precompiled artifact and checks that it matches at load-time). That said, everyone makes different tradeoffs, and I also know of a few places that use Wasmtime and load the Wasm / compile locally, because they don't want to deal with all that

Last updated: Dec 06 2025 at 05:03 UTC