Stream: git-wasmtime

Topic: wasmtime / issue #11989 Support mmap flag MAP_JIT for MACOS

Wasmtime GitHub notifications bot (Nov 06 2025 at 09:18):

Ezyyz opened issue #11989:

Hello,

During the development of our application, we’ve introduced WebAssembly (WASM) as a plugin system. Our software is a security terminal, and we’re now encountering macOS memory permission-related issues.

We refer to Apple’s official documentation:
https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon#Enable-the-JIT-entitlements-for-the-Hardened-Runtime

In short, we need to allocate executable JIT memory using:

mmap(NULL, wasm_plugin.size(), PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANON | MAP_JIT, -1, 0)

and then write into it using the macOS-specific API pthread_jit_write_with_callback_np. Both MAP_JIT and pthread_jit_write_with_callback_np are macOS-specific APIs.

For example, we implemented a function like this:

struct jit_code {
  void* jit_region;
  void *dst;
  size_t instructions_length;
};

int jit_writing_callback(void *context) {
  struct jit_code *code = (struct jit_code *)context;
  memcpy(code->dst, code->jit_region, code->instructions_length);
  return 0;
}
void* mmap_from_vector(const std::vector<uint8_t>& plugin_content) {
    void* jit_mem = mmap(NULL, plugin_content.size(),
                         PROT_READ | PROT_WRITE | PROT_EXEC,
                         MAP_PRIVATE | MAP_ANON | MAP_JIT,
                         -1, 0);
    if (jit_mem == MAP_FAILED) {
        // "jit_mem mmap failed"
        return MAP_FAILED;
    }

    struct jit_code jtcode;
    jtcode.jit_region = (void*)plugin_content.data();
    jtcode.dst = jit_mem;
    jtcode.instructions_length = plugin_content.size();
    if (pthread_jit_write_with_callback_np(jit_writing_callback, &jtcode) != 0) {
        munmap(jit_mem, plugin_content.size());
        return MAP_FAILED;
    }
    return jit_mem;
}

The allocated jit_mem is thus marked as executable JIT memory.

Currently, Wasmtime does not provide native C/C++ APIs, so our approach is to build a dynamic library in Rust using Wasmtime and export C-compatible functions for WASM loading, destruction, and function invocation, which our application then calls.

However, after allocating memory using the above method and invoking the exported WASM functions through these C interfaces, the program crashes with:

Exception Type: EXC_BAD_ACCESS (SIGKILL - Code Signature Invalid)

This indicates that the memory permissions are still invalid at runtime.

We quickly realized the root cause: when we call Component::new and pass in the WASM binary, Wasmtime internally allocates new memory regions during instance creation—meaning the memory we pre-allocated at the upper layer is not actually used for code execution.

After reviewing Wasmtime's source code, we found system-level mmap wrappers in:
crates/wasmtime/src/runtime/vm/sys/unix/mmap.rs

Our questions are:

1. When Wasmtime allocates CodeMemory on macOS, does it use the mmap implementation in this file?
2. Is there any way to extend this implementation to support macOS’s special memory allocation requirements (i.e., using MAP_JIT and writing via pthread_jit_write_with_callback_np)?
3. If I modify this code directly, will it affect memory allocations other than CodeMemory? Because, as expected, only executable code needs to be allocated in this special way.

Thank you!

Wasmtime GitHub notifications bot (Nov 06 2025 at 09:20):

Ezyyz edited issue #11989:

Hello,

During the development of our application, we’ve introduced WebAssembly (WASM) as a plugin system. Our software is a security terminal, and we’re now encountering macOS memory permission-related issues.

We refer to Apple’s official documentation:
https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon#Enable-the-JIT-entitlements-for-the-Hardened-Runtime

In short, we need to allocate executable JIT memory using:

mmap(NULL, wasm_plugin.size(), PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANON | MAP_JIT, -1, 0)

and then write into it using the macOS-specific API pthread_jit_write_with_callback_np. Both MAP_JIT and pthread_jit_write_with_callback_np are macOS-specific APIs.

For example, we implemented a function like this:

struct jit_code {
  void* jit_region;
  void *dst;
  size_t instructions_length;
};

int jit_writing_callback(void *context) {
  struct jit_code *code = (struct jit_code *)context;
  memcpy(code->dst, code->jit_region, code->instructions_length);
  return 0;
}
void* mmap_from_vector(const std::vector<uint8_t>& plugin_content) {
    void* jit_mem = mmap(NULL, plugin_content.size(),
                         PROT_READ | PROT_WRITE | PROT_EXEC,
                         MAP_PRIVATE | MAP_ANON | MAP_JIT,
                         -1, 0);
    if (jit_mem == MAP_FAILED) {
        // "jit_mem mmap failed"
        return MAP_FAILED;
    }

    struct jit_code jtcode;
    jtcode.jit_region = (void*)plugin_content.data();
    jtcode.dst = jit_mem;
    jtcode.instructions_length = plugin_content.size();
    if (pthread_jit_write_with_callback_np(jit_writing_callback, &jtcode) != 0) {
        munmap(jit_mem, plugin_content.size());
        return MAP_FAILED;
    }
    return jit_mem;
}

The allocated jit_mem is thus marked as executable JIT memory.

At the time we began development, Wasmtime does not provide native C/C++ APIs, so our approach is to build a dynamic library in Rust using Wasmtime and export C-compatible functions for WASM loading, destruction, and function invocation, which our application then calls.

However, after allocating memory using the above method and invoking the exported WASM functions through these C interfaces, the program crashes with:

Exception Type: EXC_BAD_ACCESS (SIGKILL - Code Signature Invalid)

This indicates that the memory permissions are still invalid at runtime.

We quickly realized the root cause: when we call Component::new and pass in the WASM binary, Wasmtime internally allocates new memory regions during instance creation—meaning the memory we pre-allocated at the upper layer is not actually used for code execution.

After reviewing Wasmtime's source code, we found system-level mmap wrappers in:
crates/wasmtime/src/runtime/vm/sys/unix/mmap.rs

Our questions are:

1. When Wasmtime allocates CodeMemory on macOS, does it use the mmap implementation in this file?
2. Is there any way to extend this implementation to support macOS’s special memory allocation requirements (i.e., using MAP_JIT and writing via pthread_jit_write_with_callback_np)?
3. If I modify this code directly, will it affect memory allocations other than CodeMemory? Because, as expected, only executable code needs to be allocated in this special way.

Thank you!

Wasmtime GitHub notifications bot (Nov 06 2025 at 10:04):

bjorn3 commented on issue #11989:

If I modify this code directly, will it affect memory allocations other than CodeMemory?

CodeMemory is used to mmap the entire compiled blob. This includes both the executable code segment and read-only segments like linear memory initialization data and side tables used by the Wasmtime runtime. Currently when the mmap region is created it isn't yet known which parts will be mapped as executable. Instead CodeMemory reads the metadata containing this information directly from the mmaped region before it calls mprotect in the publish method.

Wasmtime GitHub notifications bot (Nov 06 2025 at 15:50):

alexcrichton commented on issue #11989:

Do you have a way that this can be reproduced locally? None of the current maintainers have, for example, shipped iOS applications or published macOS apps so we don't have prior experience with entitlements/MAP_JIT/etc. If you have a way to reproduce though many of us have macs ourselves so we can poke around and try to see what's going on.

Wasmtime GitHub notifications bot (Nov 06 2025 at 16:05):

bjorn3 commented on issue #11989:

Based on https://wiki.freepascal.org/Hardened_runtime_for_macOS, I think creating an entitlements.xml file with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
</dict>
</plist>

and then signing the executable with codesign --force --options runtime --timestamp --entitlements entitlements.plist --sign - /path/to/wasmtime would work. Where --sign - indicates an ad-hoc signature so you don't need a code signing ceritificate.

Wasmtime GitHub notifications bot (Nov 06 2025 at 16:29):

alexcrichton commented on issue #11989:

Oh nice! Ok so with that I can see wasmtime dying quickly without MAP_JIT. I can't attach a debugger and see exactly why since that seems to need some other entitlement which I don't know how to find. With this patch though:

diff --git a/crates/wasmtime/src/runtime/vm/sys/unix/mmap.rs b/crates/wasmtime/src/runtime/vm/sys/unix/mmap.rs
index 3463c3b1b6..75530e59bc 100644
--- a/crates/wasmtime/src/runtime/vm/sys/unix/mmap.rs
+++ b/crates/wasmtime/src/runtime/vm/sys/unix/mmap.rs
@@ -35,6 +35,15 @@ cfg_if::cfg_if! {
     }
 }

+cfg_if::cfg_if! {
+    if #[cfg(target_vendor = "apple")] {
+        const MAP_JIT: rustix::mm::MapFlags =
+            rustix::mm::MapFlags::from_bits_retain(libc::MAP_JIT as u32);
+    } else {
+        const MAP_JIT: rustix::mm::MapFlags = rustix::mm::MapFlags::empty();
+    }
+}
+
 impl Mmap {
     pub fn new_empty() -> Mmap {
         Mmap {
@@ -48,7 +57,7 @@ impl Mmap {
                 ptr::null_mut(),
                 size.byte_count(),
                 rustix::mm::ProtFlags::READ | rustix::mm::ProtFlags::WRITE,
-                rustix::mm::MapFlags::PRIVATE | MMAP_NORESERVE_FLAG,
+                rustix::mm::MapFlags::PRIVATE | MMAP_NORESERVE_FLAG | MAP_JIT,
             )?
         };
         let memory = std::ptr::slice_from_raw_parts_mut(ptr.cast(), size.byte_count());
@@ -73,7 +82,7 @@ impl Mmap {
                 //
                 // Virtual memory that cannot be accessed should not have a backing store reserved
                 // for it. Hence, passing in NORESERVE is correct here.
-                rustix::mm::MapFlags::PRIVATE | MMAP_NORESERVE_FLAG,
+                rustix::mm::MapFlags::PRIVATE | MMAP_NORESERVE_FLAG | MAP_JIT,
             )?
         };

@@ -94,7 +103,7 @@ impl Mmap {
                 ptr::null_mut(),
                 len,
                 rustix::mm::ProtFlags::READ | rustix::mm::ProtFlags::WRITE,
-                rustix::mm::MapFlags::PRIVATE,
+                rustix::mm::MapFlags::PRIVATE | MAP_JIT,
                 &file,
                 0,
             )

the crash goes away. @Ezyyz would you be able to test out that patch and see if it works for you?

Wasmtime GitHub notifications bot (Nov 06 2025 at 21:24):

bjorn3 commented on issue #11989:

Ideally you would only apply MAP_JIT to the memory range that will actually end up being executable. Having to specify MAP_JIT in the first place is a protection against exploits just remapping memory they control as executable using mprotect afaik.

Wasmtime GitHub notifications bot (Nov 07 2025 at 04:46):

Ezyyz commented on issue #11989:

I've applied this patch, and it is taking effect. However, from my perspective, it might be incorrect and could introduce issues:

First, the modification targets the lowest-level mmap implementation. If non-code memory allocations (e.g., data segments, heaps, etc.) also go through this path, they would incorrectly be allocated with MAP_JIT and executable permissions, which is both unnecessary and a security risk.

Second, this approach does not follow Apple’s recommended practice. According to Apple’s documentation, memory allocated with the MAP_JIT flag should not be directly writable. Instead, after mapping such memory as PROT_READ | PROT_EXEC, one should use the official API pthread_jit_write_with_callback_np() to safely write code into it in a controlled manner.

Therefore, the proper solution on macOS should be to handle code memory allocation specially:

Only use the MAP_JIT flag when allocating executable code memory (not for general-purpose allocations).
Use Apple’s recommended pthread_jit_write_with_callback_np() (or pthread_jit_write_protect_np()) to write machine code into that region.
For the correct implementation guidance, please refer to Apple’s official documentation:
https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon#Enable-the-JIT-entitlements-for-the-Hardened-Runtime

Additionally, note that the required entitlements differ based on usage:

com.apple.security.cs.allow-jit enables basic JIT support.
com.apple.security.cs.jit-write-allowlist (if used) may require different handling.

I found the relevant API in Rust documentation here:
https://docs.rs/libvips-rs/latest/x86_64-apple-darwin/libvips_rs/bindings/fn.pthread_jit_write_with_callback_np.html
https://docs.rs/libvips-rs/latest/x86_64-apple-darwin/libvips_rs/bindings/fn.pthread_jit_write_protect_np.html

The choice of entitlement—and thus the exact API usage—depends on the application’s specific security model and deployment scenario.

Thanks for your supporting!

Wasmtime GitHub notifications bot (Nov 08 2025 at 02:24):

alexcrichton commented on issue #11989:

Well, so sort of.

Ideally you would only apply MAP_JIT to the memory range that will actually end up being executable

I don't believe we're equipped to do this unfortunately. We create one mmap with the result of compilation and it's got the entire module image in it. We're not able to create separate mappings at this time. The downside of this seems relatively minor to me though in that while the area has MAP_JIT only the executable section has PROT_EXEC.

First, the modification targets the lowest-level mmap implementation. If non-code memory allocations (e.g., data segments, heaps, etc.) also go through this path, they would incorrectly be allocated with MAP_JIT and executable permissions, which is both unnecessary and a security risk.

True, yes. I meant this as a quick patch to see if something works. I don't want to rearchitect the codebase for a platform when we don't even known if it'll work (e.g. we can't test)

Second, this approach does not follow Apple’s recommended practice.

Apple's recommended practice is for a different style of JIT than Wasmtime. In Wasmtime once code is made executable we never make it non-executable ever again. There's no dynamic modification of code. My read of pthread_jit_write_with_callback_np is that it's not necessary for use in Wasmtime.

---

As an aside @Ezyyz it looks like you're using an LLM and/or chat gpt and/or claude to generate responses here. If not, that's a mistake on my part, but if you are I'd appreciate it if you didn't as the answers are pretty verbose and can be misleading around the exact specifics of what should be done here. I don't really trust an LLM to know how best to handle this part of Apple's platforms.

Wasmtime GitHub notifications bot (Nov 10 2025 at 02:20):

Ezyyz commented on issue #11989:

Yes, i use LLM as translator. I didn't noticed that it could mislead others.I just wanted to express more accurately.It took me a lot of time, whether searching Apple's security mechanisms or reading the Wasmtime code.

I think you're right. In my test cases, i used MAP_JIT and pthread_jit_write_with_callback_np and it worked well and it alse worked well when i don't use pthread_jit_write_with_callback_np to copy.
I'll feedback, if not using pthread_jit_write_with_callback_np causes other issues(eg. Failed to copy/exec with some entitlements on).

Wasmtime GitHub notifications bot (Nov 10 2025 at 16:21):

alexcrichton commented on issue #11989:

Sounds good yeah, and this patch is also one I'm happy to land in Wasmtime itself (or if you'd like to send a PR that would also be most welcome). We can have follow-up issues about how to make the hardening more precise but I'd ideally like to unblock you first and ensuer that something runs.

Wasmtime GitHub notifications bot (Dec 05 2025 at 00:23):

alexcrichton added the wasmtime:platform-support label to Issue #11989.

Wasmtime GitHub notifications bot (Dec 05 2025 at 00:34):

alexcrichton commented on issue #11989:

I spent some time investigating this today and here's my understanding of the situation as a non-expert in this field:

• Some reference documentation is at https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon. Notable confusing things to me are
  • This claims "When memory protection is enabled, a thread cannot write to a memory region and execute instructions in that region at the same time" and that memory protection is on-by-default for aarch64. This does not detail how this is enforced. For example I don't know if mmap fails, mprotect fails, or if a fault just happens at some point.
  • This claims Wasmtime "can only create one memory region with the MAP_JIT flag set" but testing locally seems to show otherwise. I ran the CLI with --preload which creates a region for the main module and the loaded module, and that seemed to work.
  • This makes mention of pthread_jit_write_protect_np which apparently disappears with the com.apple.security.cs.jit-write-allowlist entitlement. It says we should call pthread_jit_write_with_callback_np. The pthread_jit_write_with_callback_np seems to temporarily, on a per-thread basis, make some code only-writable and not-executable.
  • This says we should execute sys_icache_invalidate "before you execute the machine instructions on a recently updated memory page". Wasmtime does not do this right now.
• bjorn3's suggestion above is how this can be tested locally.
• My patch above gets Wasmtime working but is a bit coarse:
  • Ideally we'd re-mmap, with MAP_FIXED, only the text region with MAP_JIT and we'd leave off MAP_JIT from other pages. This would require refactoring since the .text section is discovered after a MmapVec is created.
• Seems like we may want to call sys_icache_invalidate in the jit-icache-coherence crate.
• I don't think we need to call pthread_jit_write_protect_np (maybe?)
• It seems like we probably don't need to call pthread_jit_write_with_callback_np. Maybe there's some shenanigans though where it would be applicable. I'm really not sure.
• Personally if we add this I'd want to ensure it's tested in CI. We'd just need a way to codesign all our executables. Maybe a custom "linker" which does the real linker thing and then signs the output as well? Maybe a custom "test runner" that signs everything?

Overall I'm on the cusp of feeling able to implement this but not quite. I still don't really understand what MAP_JIT is insofar as how we should manage it, initialize it, ensure caches are flushed/coherent, etc. I don't understand why the documentation says we can only have one MAP_JIT region, which would be a big problem, but I seem to be able to create two locally. I don't understand if we need to use pthread_jit_* APIs.

I suspect we'll need to refactor creation of executable memory in Wasmtime to appropriately adhere to these hardening rules. I don't think that would be too too hard to do, but I think it should be done by someone who knows more about what they're doing than I.

Last updated: Dec 06 2025 at 07:03 UTC