btwiuse added the bug label to Issue #7637.
btwiuse opened issue #7637:
This error was first discovered when I tried to compile and run the polkadot binary (https://github.com/paritytech/polkadot-sdk) on linux aarch64 native environment.
The binary won't be able to work unless I change
--wasmtime-instantiation-strategy
to a non-pooling option, i.e.:these don't work
- pooling-copy-on-write
- pooling
only these work
- recreate-instance-copy-on-write
- recreate-instance
- legacy-instance-reuse
Here is what the error looks like:
$ ./gear --tmp --wasmtime-instantiation-strategy pooling 2023-12-05 20:01:19 Gear Node 2023-12-05 20:01:19 ✌️ version 1.0.3-b46adb6363f 2023-12-05 20:01:19 ❤️ by Gear Technologies, 2021-2023 2023-12-05 20:01:19 📋 Chain specification: Vara Network Testnet 2023-12-05 20:01:19 🏷 Node name: ethereal-oranges-7690 2023-12-05 20:01:19 👤 Role: FULL 2023-12-05 20:01:19 💾 Database: RocksDb at /tmp/substratesSLmp4/chains/vara_network_testnet/db/full 2023-12-05 20:01:19 ⛓ Native runtime: vara-1030 (vara-1.tx1.au1) 2023-12-05 20:01:21 🔨 Initializing Genesis block/state (state: 0x174f…af08, header-hash: 0x5256…36d6) 2023-12-05 20:01:21 👴 Loading GRANDPA authority set from genesis on what appears to be first startup. 2023-12-05 20:01:21 Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)") Error: Service(Client(VersionInvalid("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")))
(Using a binary from https://get.gear.rs/gear-nightly-aarch64-unknown-linux-musl.tar.xz here cuz polkadot doesn't provide an official build for linux/aarch64, but the error is the same for polkadot)
The wasmtime version used by substrate is 8.0.1, and I'm able to reproduce it under the latest 15.0.1 release, so it's reasonable to assume any version in between are also affected
Test Case
wasmtime version: 15.0.1
use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy}; fn main() -> Result<(), Error> { let mut config = Config::new(); let _engine_good = Engine::new(dbg!(&config))?; // ok for both amd64 and aarch64 config.allocation_strategy(InstanceAllocationStrategy::pooling()); let _engine_linux_aarch64_bad = Engine::new(dbg!(&config))?; // failed for aarch64, but ok for amd64 Ok(()) }
Steps to Reproduce
- clone https://github.com/btwiuse/wasmtime-linux-aarch64-dbg
- cargo run under linux/amd64 environment, the program runs to completion
- cargo run under linux/aarch64 environment, the program fails on the second
Engine::new(...)?
Expected Results
the program runs to completion on both architectures
Actual Results
the program panics under linux/aarch64
Error: failed to create memory pool mapping Caused by: 0: mmap failed to reserve 0x5dc80000000 bytes 1: Cannot allocate memory (os error 12)
Versions and Environment
Wasmtime version or commit: 8.0.1 & 15.0.1
Operating system: linux
Architecture: aarch64
Extra Info
If you don't have a aarch64 linux server, you can try it in a qemu virtual machine. I tested it under a aarch64 crostini vm on chromeos, and termux proot environment on Android 14.
btwiuse edited issue #7637:
This error was first discovered when I tried to compile and run the polkadot binary (https://github.com/paritytech/polkadot-sdk) on linux aarch64 native environment.
The binary won't be able to work unless I change
--wasmtime-instantiation-strategy
to a non-pooling option, i.e.:these don't work
- pooling-copy-on-write
- pooling
only these work
- recreate-instance-copy-on-write
- recreate-instance
- legacy-instance-reuse
Here is what the error looks like:
$ ./gear --tmp --wasmtime-instantiation-strategy pooling 2023-12-05 20:01:19 Gear Node 2023-12-05 20:01:19 ✌️ version 1.0.3-b46adb6363f 2023-12-05 20:01:19 ❤️ by Gear Technologies, 2021-2023 2023-12-05 20:01:19 📋 Chain specification: Vara Network Testnet 2023-12-05 20:01:19 🏷 Node name: ethereal-oranges-7690 2023-12-05 20:01:19 👤 Role: FULL 2023-12-05 20:01:19 💾 Database: RocksDb at /tmp/substratesSLmp4/chains/vara_network_testnet/db/full 2023-12-05 20:01:19 ⛓ Native runtime: vara-1030 (vara-1.tx1.au1) 2023-12-05 20:01:21 🔨 Initializing Genesis block/state (state: 0x174f…af08, header-hash: 0x5256…36d6) 2023-12-05 20:01:21 👴 Loading GRANDPA authority set from genesis on what appears to be first startup. 2023-12-05 20:01:21 Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)") Error: Service(Client(VersionInvalid("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")))
(Using a binary from https://get.gear.rs/gear-nightly-aarch64-unknown-linux-musl.tar.xz here cuz polkadot doesn't provide an official build for linux/aarch64, but the error is the same for polkadot)
The wasmtime version used by substrate is 8.0.1, and I'm able to reproduce it under the latest 15.0.1 release, so it's reasonable to assume any version in between are also affected
Test Case
wasmtime version: 15.0.1
use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy}; fn main() -> Result<(), Error> { let mut config = Config::new(); let _engine_good = Engine::new(dbg!(&config))?; // ok for both amd64 and aarch64 config.allocation_strategy(InstanceAllocationStrategy::pooling()); let _engine_linux_aarch64_bad = Engine::new(dbg!(&config))?; // failed for aarch64, but ok for amd64 Ok(()) }
Steps to Reproduce
- clone https://github.com/btwiuse/wasmtime-linux-aarch64-dbg
- cargo run under linux/amd64 environment, the program runs to completion
- cargo run under linux/aarch64 environment, the program fails on the second
Engine::new(...)?
Expected Results
the program runs to completion on both architectures
Actual Results
the program panics under linux/aarch64
Error: failed to create memory pool mapping Caused by: 0: mmap failed to reserve 0x5dc80000000 bytes 1: Cannot allocate memory (os error 12)
Versions and Environment
Wasmtime version or commit: 8.0.1 & 15.0.1
Operating system: linux
Architecture: aarch64
Extra Info
If you don't have a aarch64 linux server, you can try it in a qemu virtual machine. I tested it under a aarch64 crostini vm on chromeos, and termux proot environment on a android 14 phone with 24 GB ram. I'm pretty sure the error isn't caused by insufficient memory.
alexcrichton commented on issue #7637:
Thanks for the detailed report! The error here though could mean a number of different things, however. The two primary reasons I can think of for this failing are:
- ENOMEM indicates that the virtual address space was exhausted. This is unlikely given your small reproduction program but could theoretically be the case for the original program (although still unlikely).
- ENOMEM can also indicate that the kernel ran out of memory allocating data structures tracking virtual memory usage. On Linux this can be affected by
sudo sysctl vm.max_map_count
. You can try raising that withsudo sysctl vm.max_map_count=N
and see if that helps?If
sysctl
fixes things then either that's the fix or Wasmtime's memory pool should be configured with fewer slots. That would likely be done through the embedding that's using Wasmtime as it seems like you're probably not configuring Wasmtime directly but rather indirectly.Also, FWIW, this isn't failing locally for me on arm64 Linux, so my best guess is that it's a difference in
sysctl vm.max_map_count
settings. That being said locally I had to turn that setting into the ~2k range to get failures which is well below the default 65k, so I'm not 100% sure about this.Can you paste the output of
sysctl vm.max_map_count
and also provide more kernel/OS information?
btwiuse commented on issue #7637:
On my crostini vm, it shows
$ sudo sysctl vm.max_map_count vm.max_map_count = 262144
Unfortunately I can't change it because I don't have root permission on the vm host
$ sudo sysctl vm.max_map_count=333333 sysctl: permission denied on key "vm.max_map_count"
Here is the system info:
btwiuse@penguin:~/wasmtime-dbg$ lscpu Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 1 Vendor ID: ARM Model: 4 Model name: Cortex-A53 Stepping: r0p4 BogoMIPS: 26.00 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Not affected Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; Branch predictor hardening, but not BHB Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid btwiuse@penguin:~/wasmtime-dbg$ uname -a Linux penguin 6.1.60-08585-g3c866855af53 #1 SMP PREEMPT Sun Nov 5 17:38:41 PST 2023 aarch64 GNU/Linux
![image](https://github.com/bytecodealliance/wasmtime/assets/54848194/6986bac8-9264-460b-b242-927ff837691c)
btwiuse commented on issue #7637:
My phone is not rooted, so can't get the max_map_count:
$ sysctl vm.max_map_count sysctl: cannot stat /proc/sys/vm/max_map_count: Permission denied
![image](https://github.com/bytecodealliance/wasmtime/assets/54848194/5f5355f5-25bb-48d5-b26e-0e5d6e65eaae)
[alarm@localhost wasmtime-dbg]$ uname -a Linux localhost 6.2.1-PRoot-Distro #1 SMP PREEMPT Wed Oct 4 10:42:48 UTC 2023 aarch64 GNU/Linux [alarm@localhost wasmtime-dbg]$ lscpu Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: ARM Model name: Cortex-A520 Model: 1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: r0p1 Frequency boost: enabled CPU(s) scaling MHz: 25% CPU max MHz: 2265.6001 CPU min MHz: 364.8000 BogoMIPS: 38.40 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt i ecv afp Model name: Cortex-A720 Model: 1 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 1 Stepping: r0p1 CPU(s) scaling MHz: 20% CPU max MHz: 3148.8000 CPU min MHz: 499.2000 BogoMIPS: 38.40 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt i ecv afp Model name: - Model: 1 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 CPU(s) scaling MHz: 20% CPU max MHz: 3302.3999 CPU min MHz: 480.0000 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Not affected Srbds: Not affected Tsx async abort: Not affected
btwiuse edited a comment on issue #7637:
My phone is not rooted, so can't get the max_map_count:
$ sudo sysctl vm.max_map_count sysctl: cannot stat /proc/sys/vm/max_map_count: Permission denied
![image](https://github.com/bytecodealliance/wasmtime/assets/54848194/5f5355f5-25bb-48d5-b26e-0e5d6e65eaae)
[alarm@localhost wasmtime-dbg]$ uname -a Linux localhost 6.2.1-PRoot-Distro #1 SMP PREEMPT Wed Oct 4 10:42:48 UTC 2023 aarch64 GNU/Linux [alarm@localhost wasmtime-dbg]$ lscpu Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: ARM Model name: Cortex-A520 Model: 1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: r0p1 Frequency boost: enabled CPU(s) scaling MHz: 25% CPU max MHz: 2265.6001 CPU min MHz: 364.8000 BogoMIPS: 38.40 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt i ecv afp Model name: Cortex-A720 Model: 1 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 1 Stepping: r0p1 CPU(s) scaling MHz: 20% CPU max MHz: 3148.8000 CPU min MHz: 499.2000 BogoMIPS: 38.40 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt i ecv afp Model name: - Model: 1 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 CPU(s) scaling MHz: 20% CPU max MHz: 3302.3999 CPU min MHz: 480.0000 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Not affected Srbds: Not affected Tsx async abort: Not affected
btwiuse commented on issue #7637:
Oh, I forgot the ChromeOS itself is based on Linux and I have developer mode turned on.
My device is Lenovo Chromebook Duet 10.1 inch with 4GB of ram.
I still get the same error if I execute the binary on chromeos
the default value is
vm.max_map_count = 65530
and the error persists even after I change it to 99999999
krane-rev4 /mnt/stateful_partition/dev_image # sysctl vm.max_map_count=99999999 vm.max_map_count = 99999999 krane-rev4 /mnt/stateful_partition/dev_image # ./w [main.rs:6] &config = Config { debug_info: false, parse_wasm_debuginfo: false, wasm_threads: true, wasm_reference_types: true, wasm_function_references: false, wasm_bulk_memory: true, wasm_simd: true, wasm_relaxed_simd: true, wasm_multi_value: true, static_memory_maximum_size: 4294967296, static_memory_guard_size: 2147483648, dynamic_memory_guard_size: 65536, guard_before_linear_memory: true, parallel_compilation: true, compiler_config: CompilerConfig { strategy: Auto, target: None, settings: { "enable_verifier": "false", "opt_level": "speed", }, flags: {}, cache_store: None, clif_dir: None, wmemcheck: false, }, } [main.rs:10] &config = Config { debug_info: false, parse_wasm_debuginfo: false, wasm_threads: true, wasm_reference_types: true, wasm_function_references: false, wasm_bulk_memory: true, wasm_simd: true, wasm_relaxed_simd: true, wasm_multi_value: true, static_memory_maximum_size: 4294967296, static_memory_guard_size: 2147483648, dynamic_memory_guard_size: 65536, guard_before_linear_memory: true, parallel_compilation: true, compiler_config: CompilerConfig { strategy: Auto, target: None, settings: { "enable_verifier": "false", "opt_level": "speed", }, flags: {}, cache_store: None, clif_dir: None, wmemcheck: false, }, } Error: failed to create memory pool mapping Caused by: 0: mmap failed to reserve 0x5dc80000000 bytes 1: Out of memory (os error 12)
system info:
krane-rev4 /mnt/stateful_partition/dev_image # uname -a Linux localhost 5.10.199-23580-g85923a9daa67 #1 SMP PREEMPT Mon Nov 6 18:56:13 PST 2023 aarch64 GNU/Linux krane-rev4 /mnt/stateful_partition/dev_image # lscpu Architecture: aarch64 Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: ARM Model name: Cortex-A53 Model: 4 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Stepping: r0p4 CPU(s) scaling MHz: 87% CPU max MHz: 1989.0000 CPU min MHz: 793.0000 BogoMIPS: 26.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid Model name: Cortex-A73 Model: 2 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Stepping: r0p2 CPU(s) scaling MHz: 40% CPU max MHz: 1989.0000 CPU min MHz: 793.0000 BogoMIPS: 26.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Not affected Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; Branch predictor hardening, but not BHB Srbds: Not affected Tsx async abort: Not affected
btwiuse commented on issue #7637:
I have included precompiled binaries in https://github.com/btwiuse/wasmtime-linux-aarch64-dbg
It's interesting when I execute it with
qemu-user-static
on a linux/amd64 host, it doesn't show the error:qemu-aarch64-static wasmtime-linux-aarch64-dbg
btwiuse edited a comment on issue #7637:
Oh, I forgot the ChromeOS itself is based on Linux and I have developer mode turned on.
My device is Lenovo Chromebook Duet 10.1 inch with 4GB of ram.
I still get the same error if I execute the binary directly on the chromeos host
the default value for
vm.max_map_count
on the host isvm.max_map_count = 65530
and the error persists even after I change it to 99999999
krane-rev4 /mnt/stateful_partition/dev_image # sysctl vm.max_map_count=99999999 vm.max_map_count = 99999999 krane-rev4 /mnt/stateful_partition/dev_image # ./w [main.rs:6] &config = Config { debug_info: false, parse_wasm_debuginfo: false, wasm_threads: true, wasm_reference_types: true, wasm_function_references: false, wasm_bulk_memory: true, wasm_simd: true, wasm_relaxed_simd: true, wasm_multi_value: true, static_memory_maximum_size: 4294967296, static_memory_guard_size: 2147483648, dynamic_memory_guard_size: 65536, guard_before_linear_memory: true, parallel_compilation: true, compiler_config: CompilerConfig { strategy: Auto, target: None, settings: { "enable_verifier": "false", "opt_level": "speed", }, flags: {}, cache_store: None, clif_dir: None, wmemcheck: false, }, } [main.rs:10] &config = Config { debug_info: false, parse_wasm_debuginfo: false, wasm_threads: true, wasm_reference_types: true, wasm_function_references: false, wasm_bulk_memory: true, wasm_simd: true, wasm_relaxed_simd: true, wasm_multi_value: true, static_memory_maximum_size: 4294967296, static_memory_guard_size: 2147483648, dynamic_memory_guard_size: 65536, guard_before_linear_memory: true, parallel_compilation: true, compiler_config: CompilerConfig { strategy: Auto, target: None, settings: { "enable_verifier": "false", "opt_level": "speed", }, flags: {}, cache_store: None, clif_dir: None, wmemcheck: false, }, } Error: failed to create memory pool mapping Caused by: 0: mmap failed to reserve 0x5dc80000000 bytes 1: Out of memory (os error 12)
system info:
krane-rev4 /mnt/stateful_partition/dev_image # uname -a Linux localhost 5.10.199-23580-g85923a9daa67 #1 SMP PREEMPT Mon Nov 6 18:56:13 PST 2023 aarch64 GNU/Linux krane-rev4 /mnt/stateful_partition/dev_image # lscpu Architecture: aarch64 Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: ARM Model name: Cortex-A53 Model: 4 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Stepping: r0p4 CPU(s) scaling MHz: 87% CPU max MHz: 1989.0000 CPU min MHz: 793.0000 BogoMIPS: 26.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid Model name: Cortex-A73 Model: 2 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Stepping: r0p2 CPU(s) scaling MHz: 40% CPU max MHz: 1989.0000 CPU min MHz: 793.0000 BogoMIPS: 26.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Not affected Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; Branch predictor hardening, but not BHB Srbds: Not affected Tsx async abort: Not affected
btwiuse commented on issue #7637:
I just applied the patch from https://github.com/bytecodealliance/wasmtime/issues/6003 and it fixed the error
$ git diff diff --git a/main.rs b/main.rs index 0e6b7dd..6e72a69 100644 --- a/main.rs +++ b/main.rs @@ -3,6 +3,11 @@ use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy}; fn main() -> Result<(), Error> { let mut config = Config::new(); + config.static_memory_maximum_size(0); + config.dynamic_memory_reserved_for_growth(0); + config.static_memory_guard_size(0); + config.dynamic_memory_guard_size(0); + let _engine_good = Engine::new(dbg!(&config))?; config.allocation_strategy(InstanceAllocationStrategy::pooling());
alexcrichton commented on issue #7637:
Thanks for the extra info! Given what you're seeing it seems unlikely you're running into
vm.max_map_count
as a limit. That being said my guess is you're running into some limit somewhere which is what's causing ENOMEM, and at this point your guess as to being able to pinpoint which limit is being hit is as good as mine.The configuration you applied I'll note has a large impact on the performance of WebAssembly, so it's not really a "fix" per se but is a set of knobs that can be turned in theory. For example with those knobs you've greatly reduced the virtual memory allocation so it succeeds.
Some other things you might want to try:
- I took a look at
sysctl -a | rg '^vm'
and the only other one that seemed relevant wasvm.overcommit_memory
. Is that nonzero on your systems? Alternatively, could you paste the output ofsysctl -a | rg '^vm'
on a system that fails to run this program?- Could you try bisecting the size necessary to trip the allocation failure? That can be done by tuning
PoolingAllocationConfig::total_memories
.Other than that though I'm running out of ideas myself :(
alexcrichton closed issue #7637:
This error was first discovered when I tried to compile and run the polkadot binary (https://github.com/paritytech/polkadot-sdk) on linux aarch64 native environment.
The binary won't be able to work unless I change
--wasmtime-instantiation-strategy
to a non-pooling option, i.e.:these don't work
- pooling-copy-on-write
- pooling
only these work
- recreate-instance-copy-on-write
- recreate-instance
- legacy-instance-reuse
Here is what the error looks like:
$ ./gear --tmp --wasmtime-instantiation-strategy pooling 2023-12-05 20:01:19 Gear Node 2023-12-05 20:01:19 ✌️ version 1.0.3-b46adb6363f 2023-12-05 20:01:19 ❤️ by Gear Technologies, 2021-2023 2023-12-05 20:01:19 📋 Chain specification: Vara Network Testnet 2023-12-05 20:01:19 🏷 Node name: ethereal-oranges-7690 2023-12-05 20:01:19 👤 Role: FULL 2023-12-05 20:01:19 💾 Database: RocksDb at /tmp/substratesSLmp4/chains/vara_network_testnet/db/full 2023-12-05 20:01:19 ⛓ Native runtime: vara-1030 (vara-1.tx1.au1) 2023-12-05 20:01:21 🔨 Initializing Genesis block/state (state: 0x174f…af08, header-hash: 0x5256…36d6) 2023-12-05 20:01:21 👴 Loading GRANDPA authority set from genesis on what appears to be first startup. 2023-12-05 20:01:21 Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)") Error: Service(Client(VersionInvalid("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")))
(Using a binary from https://get.gear.rs/gear-nightly-aarch64-unknown-linux-musl.tar.xz here cuz polkadot doesn't provide an official build for linux/aarch64, but the error is the same for polkadot)
The wasmtime version used by substrate is 8.0.1, and I'm able to reproduce it under the latest 15.0.1 release, so it's reasonable to assume any version in between are also affected
Test Case
wasmtime version: 15.0.1
use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy}; fn main() -> Result<(), Error> { let mut config = Config::new(); let _engine_good = Engine::new(dbg!(&config))?; // ok for both amd64 and aarch64 config.allocation_strategy(InstanceAllocationStrategy::pooling()); let _engine_linux_aarch64_bad = Engine::new(dbg!(&config))?; // failed for aarch64, but ok for amd64 Ok(()) }
Steps to Reproduce
- clone https://github.com/btwiuse/wasmtime-linux-aarch64-dbg
- cargo run under linux/amd64 environment, the program runs to completion
- cargo run under linux/aarch64 environment, the program fails on the second
Engine::new(...)?
Expected Results
the program runs to completion on both architectures
Actual Results
the program panics under linux/aarch64
Error: failed to create memory pool mapping Caused by: 0: mmap failed to reserve 0x5dc80000000 bytes 1: Cannot allocate memory (os error 12)
Versions and Environment
Wasmtime version or commit: 8.0.1 & 15.0.1
Operating system: linux
Architecture: aarch64
Extra Info
If you don't have a aarch64 linux server, you can try it in a qemu virtual machine. I tested it under a aarch64 crostini vm on chromeos, and termux proot environment on a android 14 phone with 24 GB ram. I'm pretty sure the error isn't caused by insufficient memory.
alexcrichton commented on issue #7637:
I'm going to close this issue since I don't think there's much in Wasmtime we can do about this. What I can recommend though is a block like this where
wasmtime serve
will dynamically detect it shouldn't use the pooling allocator on some aarch64 configurations. I suspect that's what was happening here where the virtual address space only has 39 bits instead of the expected 48 which resulted in these errors.
Last updated: Dec 23 2024 at 12:05 UTC