Stream: git-wasmtime

Topic: wasmtime / issue #7637 `Error: failed to create memory po...

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 20:21):

btwiuse added the bug label to Issue #7637.

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 20:21):

btwiuse opened issue #7637:

This error was first discovered when I tried to compile and run the polkadot binary (https://github.com/paritytech/polkadot-sdk) on linux aarch64 native environment.

The binary won't be able to work unless I change --wasmtime-instantiation-strategy to a non-pooling option, i.e.:

these don't work

only these work

Here is what the error looks like:

$ ./gear --tmp --wasmtime-instantiation-strategy pooling
2023-12-05 20:01:19 Gear Node
2023-12-05 20:01:19 ✌️  version 1.0.3-b46adb6363f
2023-12-05 20:01:19 ❤️  by Gear Technologies, 2021-2023
2023-12-05 20:01:19 📋 Chain specification: Vara Network Testnet
2023-12-05 20:01:19 🏷  Node name: ethereal-oranges-7690
2023-12-05 20:01:19 👤 Role: FULL
2023-12-05 20:01:19 💾 Database: RocksDb at /tmp/substratesSLmp4/chains/vara_network_testnet/db/full
2023-12-05 20:01:19 ⛓  Native runtime: vara-1030 (vara-1.tx1.au1)
2023-12-05 20:01:21 🔨 Initializing Genesis block/state (state: 0x174f…af08, header-hash: 0x5256…36d6)
2023-12-05 20:01:21 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2023-12-05 20:01:21 Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")
Error: Service(Client(VersionInvalid("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")))

(Using a binary from https://get.gear.rs/gear-nightly-aarch64-unknown-linux-musl.tar.xz here cuz polkadot doesn't provide an official build for linux/aarch64, but the error is the same for polkadot)

The wasmtime version used by substrate is 8.0.1, and I'm able to reproduce it under the latest 15.0.1 release, so it's reasonable to assume any version in between are also affected

Test Case

wasmtime version: 15.0.1

use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy};

fn main() -> Result<(), Error> {
    let mut config = Config::new();

    let _engine_good = Engine::new(dbg!(&config))?; // ok for both amd64 and aarch64

    config.allocation_strategy(InstanceAllocationStrategy::pooling());

    let _engine_linux_aarch64_bad = Engine::new(dbg!(&config))?; // failed for aarch64, but ok for amd64

    Ok(())
}

Steps to Reproduce

  1. clone https://github.com/btwiuse/wasmtime-linux-aarch64-dbg
  2. cargo run under linux/amd64 environment, the program runs to completion
  3. cargo run under linux/aarch64 environment, the program fails on the second Engine::new(...)?

Expected Results

the program runs to completion on both architectures

Actual Results

the program panics under linux/aarch64

Error: failed to create memory pool mapping

Caused by:
    0: mmap failed to reserve 0x5dc80000000 bytes
    1: Cannot allocate memory (os error 12)

Versions and Environment

Wasmtime version or commit: 8.0.1 & 15.0.1

Operating system: linux

Architecture: aarch64

Extra Info

If you don't have a aarch64 linux server, you can try it in a qemu virtual machine. I tested it under a aarch64 crostini vm on chromeos, and termux proot environment on Android 14.

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 20:25):

btwiuse edited issue #7637:

This error was first discovered when I tried to compile and run the polkadot binary (https://github.com/paritytech/polkadot-sdk) on linux aarch64 native environment.

The binary won't be able to work unless I change --wasmtime-instantiation-strategy to a non-pooling option, i.e.:

these don't work

only these work

Here is what the error looks like:

$ ./gear --tmp --wasmtime-instantiation-strategy pooling
2023-12-05 20:01:19 Gear Node
2023-12-05 20:01:19 ✌️  version 1.0.3-b46adb6363f
2023-12-05 20:01:19 ❤️  by Gear Technologies, 2021-2023
2023-12-05 20:01:19 📋 Chain specification: Vara Network Testnet
2023-12-05 20:01:19 🏷  Node name: ethereal-oranges-7690
2023-12-05 20:01:19 👤 Role: FULL
2023-12-05 20:01:19 💾 Database: RocksDb at /tmp/substratesSLmp4/chains/vara_network_testnet/db/full
2023-12-05 20:01:19 ⛓  Native runtime: vara-1030 (vara-1.tx1.au1)
2023-12-05 20:01:21 🔨 Initializing Genesis block/state (state: 0x174f…af08, header-hash: 0x5256…36d6)
2023-12-05 20:01:21 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2023-12-05 20:01:21 Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")
Error: Service(Client(VersionInvalid("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")))

(Using a binary from https://get.gear.rs/gear-nightly-aarch64-unknown-linux-musl.tar.xz here cuz polkadot doesn't provide an official build for linux/aarch64, but the error is the same for polkadot)

The wasmtime version used by substrate is 8.0.1, and I'm able to reproduce it under the latest 15.0.1 release, so it's reasonable to assume any version in between are also affected

Test Case

wasmtime version: 15.0.1

use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy};

fn main() -> Result<(), Error> {
    let mut config = Config::new();

    let _engine_good = Engine::new(dbg!(&config))?; // ok for both amd64 and aarch64

    config.allocation_strategy(InstanceAllocationStrategy::pooling());

    let _engine_linux_aarch64_bad = Engine::new(dbg!(&config))?; // failed for aarch64, but ok for amd64

    Ok(())
}

Steps to Reproduce

  1. clone https://github.com/btwiuse/wasmtime-linux-aarch64-dbg
  2. cargo run under linux/amd64 environment, the program runs to completion
  3. cargo run under linux/aarch64 environment, the program fails on the second Engine::new(...)?

Expected Results

the program runs to completion on both architectures

Actual Results

the program panics under linux/aarch64

Error: failed to create memory pool mapping

Caused by:
    0: mmap failed to reserve 0x5dc80000000 bytes
    1: Cannot allocate memory (os error 12)

Versions and Environment

Wasmtime version or commit: 8.0.1 & 15.0.1

Operating system: linux

Architecture: aarch64

Extra Info

If you don't have a aarch64 linux server, you can try it in a qemu virtual machine. I tested it under a aarch64 crostini vm on chromeos, and termux proot environment on a android 14 phone with 24 GB ram. I'm pretty sure the error isn't caused by insufficient memory.

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 20:35):

alexcrichton commented on issue #7637:

Thanks for the detailed report! The error here though could mean a number of different things, however. The two primary reasons I can think of for this failing are:

  1. ENOMEM indicates that the virtual address space was exhausted. This is unlikely given your small reproduction program but could theoretically be the case for the original program (although still unlikely).
  2. ENOMEM can also indicate that the kernel ran out of memory allocating data structures tracking virtual memory usage. On Linux this can be affected by sudo sysctl vm.max_map_count. You can try raising that with sudo sysctl vm.max_map_count=N and see if that helps?

If sysctl fixes things then either that's the fix or Wasmtime's memory pool should be configured with fewer slots. That would likely be done through the embedding that's using Wasmtime as it seems like you're probably not configuring Wasmtime directly but rather indirectly.

Also, FWIW, this isn't failing locally for me on arm64 Linux, so my best guess is that it's a difference in sysctl vm.max_map_count settings. That being said locally I had to turn that setting into the ~2k range to get failures which is well below the default 65k, so I'm not 100% sure about this.

Can you paste the output of sysctl vm.max_map_count and also provide more kernel/OS information?

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 21:06):

btwiuse commented on issue #7637:

On my crostini vm, it shows

$ sudo sysctl vm.max_map_count
vm.max_map_count = 262144

Unfortunately I can't change it because I don't have root permission on the vm host

$ sudo sysctl vm.max_map_count=333333
sysctl: permission denied on key "vm.max_map_count"

Here is the system info:

btwiuse@penguin:~/wasmtime-dbg$ lscpu
Architecture:                       aarch64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
CPU(s):                             8
On-line CPU(s) list:                0-7
Thread(s) per core:                 1
Core(s) per socket:                 8
Socket(s):                          1
Vendor ID:                          ARM
Model:                              4
Model name:                         Cortex-A53
Stepping:                           r0p4
BogoMIPS:                           26.00
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Not affected
Vulnerability Spectre v1:           Mitigation; __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Branch predictor hardening, but not BHB
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
btwiuse@penguin:~/wasmtime-dbg$ uname -a
Linux penguin 6.1.60-08585-g3c866855af53 #1 SMP PREEMPT Sun Nov  5 17:38:41 PST 2023 aarch64 GNU/Linux

![image](https://github.com/bytecodealliance/wasmtime/assets/54848194/6986bac8-9264-460b-b242-927ff837691c)

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 21:08):

btwiuse commented on issue #7637:

My phone is not rooted, so can't get the max_map_count:

$ sysctl vm.max_map_count
sysctl: cannot stat /proc/sys/vm/max_map_count: Permission denied

![image](https://github.com/bytecodealliance/wasmtime/assets/54848194/5f5355f5-25bb-48d5-b26e-0e5d6e65eaae)

[alarm@localhost wasmtime-dbg]$ uname -a
Linux localhost 6.2.1-PRoot-Distro #1 SMP PREEMPT Wed Oct  4 10:42:48 UTC 2023 aarch64 GNU/Linux
[alarm@localhost wasmtime-dbg]$ lscpu
Architecture:           aarch64
  CPU op-mode(s):       64-bit
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              ARM
  Model name:           Cortex-A520
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 2
    Socket(s):          1
    Stepping:           r0p1
    Frequency boost:    enabled
    CPU(s) scaling MHz: 25%
    CPU max MHz:        2265.6001
    CPU min MHz:        364.8000
    BogoMIPS:           38.40
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s
                        m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt
                        i ecv afp
  Model name:           Cortex-A720
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 5
    Socket(s):          1
    Stepping:           r0p1
    CPU(s) scaling MHz: 20%
    CPU max MHz:        3148.8000
    CPU min MHz:        499.2000
    BogoMIPS:           38.40
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s
                        m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt
                        i ecv afp
  Model name:           -
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 1
    Socket(s):          1
    CPU(s) scaling MHz: 20%
    CPU max MHz:        3302.3999
    CPU min MHz:        480.0000
Vulnerabilities:
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Not affected
  Srbds:                Not affected
  Tsx async abort:      Not affected

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 21:09):

btwiuse edited a comment on issue #7637:

My phone is not rooted, so can't get the max_map_count:

$ sudo sysctl vm.max_map_count
sysctl: cannot stat /proc/sys/vm/max_map_count: Permission denied

![image](https://github.com/bytecodealliance/wasmtime/assets/54848194/5f5355f5-25bb-48d5-b26e-0e5d6e65eaae)

[alarm@localhost wasmtime-dbg]$ uname -a
Linux localhost 6.2.1-PRoot-Distro #1 SMP PREEMPT Wed Oct  4 10:42:48 UTC 2023 aarch64 GNU/Linux
[alarm@localhost wasmtime-dbg]$ lscpu
Architecture:           aarch64
  CPU op-mode(s):       64-bit
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              ARM
  Model name:           Cortex-A520
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 2
    Socket(s):          1
    Stepping:           r0p1
    Frequency boost:    enabled
    CPU(s) scaling MHz: 25%
    CPU max MHz:        2265.6001
    CPU min MHz:        364.8000
    BogoMIPS:           38.40
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s
                        m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt
                        i ecv afp
  Model name:           Cortex-A720
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 5
    Socket(s):          1
    Stepping:           r0p1
    CPU(s) scaling MHz: 20%
    CPU max MHz:        3148.8000
    CPU min MHz:        499.2000
    BogoMIPS:           38.40
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 s
                        m3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 dgh bt
                        i ecv afp
  Model name:           -
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 1
    Socket(s):          1
    CPU(s) scaling MHz: 20%
    CPU max MHz:        3302.3999
    CPU min MHz:        480.0000
Vulnerabilities:
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Not affected
  Srbds:                Not affected
  Tsx async abort:      Not affected

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 21:34):

btwiuse commented on issue #7637:

Oh, I forgot the ChromeOS itself is based on Linux and I have developer mode turned on.

My device is Lenovo Chromebook Duet 10.1 inch with 4GB of ram.

I still get the same error if I execute the binary on chromeos

the default value is

vm.max_map_count = 65530

and the error persists even after I change it to 99999999

krane-rev4 /mnt/stateful_partition/dev_image # sysctl vm.max_map_count=99999999
vm.max_map_count = 99999999

krane-rev4 /mnt/stateful_partition/dev_image # ./w
[main.rs:6] &config = Config {
    debug_info: false,
    parse_wasm_debuginfo: false,
    wasm_threads: true,
    wasm_reference_types: true,
    wasm_function_references: false,
    wasm_bulk_memory: true,
    wasm_simd: true,
    wasm_relaxed_simd: true,
    wasm_multi_value: true,
    static_memory_maximum_size: 4294967296,
    static_memory_guard_size: 2147483648,
    dynamic_memory_guard_size: 65536,
    guard_before_linear_memory: true,
    parallel_compilation: true,
    compiler_config: CompilerConfig {
        strategy: Auto,
        target: None,
        settings: {
            "enable_verifier": "false",
            "opt_level": "speed",
        },
        flags: {},
        cache_store: None,
        clif_dir: None,
        wmemcheck: false,
    },
}
[main.rs:10] &config = Config {
    debug_info: false,
    parse_wasm_debuginfo: false,
    wasm_threads: true,
    wasm_reference_types: true,
    wasm_function_references: false,
    wasm_bulk_memory: true,
    wasm_simd: true,
    wasm_relaxed_simd: true,
    wasm_multi_value: true,
    static_memory_maximum_size: 4294967296,
    static_memory_guard_size: 2147483648,
    dynamic_memory_guard_size: 65536,
    guard_before_linear_memory: true,
    parallel_compilation: true,
    compiler_config: CompilerConfig {
        strategy: Auto,
        target: None,
        settings: {
            "enable_verifier": "false",
            "opt_level": "speed",
        },
        flags: {},
        cache_store: None,
        clif_dir: None,
        wmemcheck: false,
    },
}
Error: failed to create memory pool mapping

Caused by:
    0: mmap failed to reserve 0x5dc80000000 bytes
    1: Out of memory (os error 12)

system info:

krane-rev4 /mnt/stateful_partition/dev_image # uname -a
Linux localhost 5.10.199-23580-g85923a9daa67 #1 SMP PREEMPT Mon Nov 6 18:56:13 PST 2023 aarch64 GNU/Linux
krane-rev4 /mnt/stateful_partition/dev_image # lscpu
Architecture:           aarch64
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              ARM
  Model name:           Cortex-A53
    Model:              4
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           r0p4
    CPU(s) scaling MHz: 87%
    CPU max MHz:        1989.0000
    CPU min MHz:        793.0000
    BogoMIPS:           26.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
  Model name:           Cortex-A73
    Model:              2
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           r0p2
    CPU(s) scaling MHz: 40%
    CPU max MHz:        1989.0000
    CPU min MHz:        793.0000
    BogoMIPS:           26.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
Vulnerabilities:
  Gather data sampling: Not affected
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec rstack overflow: Not affected
  Spec store bypass:    Not affected
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Mitigation; Branch predictor hardening, but not BHB
  Srbds:                Not affected
  Tsx async abort:      Not affected

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 21:36):

btwiuse commented on issue #7637:

I have included precompiled binaries in https://github.com/btwiuse/wasmtime-linux-aarch64-dbg

It's interesting when I execute it with qemu-user-static on a linux/amd64 host, it doesn't show the error:

qemu-aarch64-static wasmtime-linux-aarch64-dbg

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 21:40):

btwiuse edited a comment on issue #7637:

Oh, I forgot the ChromeOS itself is based on Linux and I have developer mode turned on.

My device is Lenovo Chromebook Duet 10.1 inch with 4GB of ram.

I still get the same error if I execute the binary directly on the chromeos host

the default value for vm.max_map_count on the host is

vm.max_map_count = 65530

and the error persists even after I change it to 99999999

krane-rev4 /mnt/stateful_partition/dev_image # sysctl vm.max_map_count=99999999
vm.max_map_count = 99999999

krane-rev4 /mnt/stateful_partition/dev_image # ./w
[main.rs:6] &config = Config {
    debug_info: false,
    parse_wasm_debuginfo: false,
    wasm_threads: true,
    wasm_reference_types: true,
    wasm_function_references: false,
    wasm_bulk_memory: true,
    wasm_simd: true,
    wasm_relaxed_simd: true,
    wasm_multi_value: true,
    static_memory_maximum_size: 4294967296,
    static_memory_guard_size: 2147483648,
    dynamic_memory_guard_size: 65536,
    guard_before_linear_memory: true,
    parallel_compilation: true,
    compiler_config: CompilerConfig {
        strategy: Auto,
        target: None,
        settings: {
            "enable_verifier": "false",
            "opt_level": "speed",
        },
        flags: {},
        cache_store: None,
        clif_dir: None,
        wmemcheck: false,
    },
}
[main.rs:10] &config = Config {
    debug_info: false,
    parse_wasm_debuginfo: false,
    wasm_threads: true,
    wasm_reference_types: true,
    wasm_function_references: false,
    wasm_bulk_memory: true,
    wasm_simd: true,
    wasm_relaxed_simd: true,
    wasm_multi_value: true,
    static_memory_maximum_size: 4294967296,
    static_memory_guard_size: 2147483648,
    dynamic_memory_guard_size: 65536,
    guard_before_linear_memory: true,
    parallel_compilation: true,
    compiler_config: CompilerConfig {
        strategy: Auto,
        target: None,
        settings: {
            "enable_verifier": "false",
            "opt_level": "speed",
        },
        flags: {},
        cache_store: None,
        clif_dir: None,
        wmemcheck: false,
    },
}
Error: failed to create memory pool mapping

Caused by:
    0: mmap failed to reserve 0x5dc80000000 bytes
    1: Out of memory (os error 12)

system info:

krane-rev4 /mnt/stateful_partition/dev_image # uname -a
Linux localhost 5.10.199-23580-g85923a9daa67 #1 SMP PREEMPT Mon Nov 6 18:56:13 PST 2023 aarch64 GNU/Linux
krane-rev4 /mnt/stateful_partition/dev_image # lscpu
Architecture:           aarch64
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              ARM
  Model name:           Cortex-A53
    Model:              4
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           r0p4
    CPU(s) scaling MHz: 87%
    CPU max MHz:        1989.0000
    CPU min MHz:        793.0000
    BogoMIPS:           26.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
  Model name:           Cortex-A73
    Model:              2
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           r0p2
    CPU(s) scaling MHz: 40%
    CPU max MHz:        1989.0000
    CPU min MHz:        793.0000
    BogoMIPS:           26.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
Vulnerabilities:
  Gather data sampling: Not affected
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec rstack overflow: Not affected
  Spec store bypass:    Not affected
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Mitigation; Branch predictor hardening, but not BHB
  Srbds:                Not affected
  Tsx async abort:      Not affected

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 21:57):

btwiuse commented on issue #7637:

I just applied the patch from https://github.com/bytecodealliance/wasmtime/issues/6003 and it fixed the error

$ git diff
diff --git a/main.rs b/main.rs
index 0e6b7dd..6e72a69 100644
--- a/main.rs
+++ b/main.rs
@@ -3,6 +3,11 @@ use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy};
 fn main() -> Result<(), Error> {
     let mut config = Config::new();


+    config.static_memory_maximum_size(0);
+    config.dynamic_memory_reserved_for_growth(0);
+    config.static_memory_guard_size(0);
+    config.dynamic_memory_guard_size(0);
+
     let _engine_good = Engine::new(dbg!(&config))?;

     config.allocation_strategy(InstanceAllocationStrategy::pooling());

view this post on Zulip Wasmtime GitHub notifications bot (Dec 05 2023 at 22:23):

alexcrichton commented on issue #7637:

Thanks for the extra info! Given what you're seeing it seems unlikely you're running into vm.max_map_count as a limit. That being said my guess is you're running into some limit somewhere which is what's causing ENOMEM, and at this point your guess as to being able to pinpoint which limit is being hit is as good as mine.

The configuration you applied I'll note has a large impact on the performance of WebAssembly, so it's not really a "fix" per se but is a set of knobs that can be turned in theory. For example with those knobs you've greatly reduced the virtual memory allocation so it succeeds.

Some other things you might want to try:

Other than that though I'm running out of ideas myself :(

view this post on Zulip Wasmtime GitHub notifications bot (Nov 15 2024 at 18:09):

alexcrichton closed issue #7637:

This error was first discovered when I tried to compile and run the polkadot binary (https://github.com/paritytech/polkadot-sdk) on linux aarch64 native environment.

The binary won't be able to work unless I change --wasmtime-instantiation-strategy to a non-pooling option, i.e.:

these don't work

only these work

Here is what the error looks like:

$ ./gear --tmp --wasmtime-instantiation-strategy pooling
2023-12-05 20:01:19 Gear Node
2023-12-05 20:01:19 ✌️  version 1.0.3-b46adb6363f
2023-12-05 20:01:19 ❤️  by Gear Technologies, 2021-2023
2023-12-05 20:01:19 📋 Chain specification: Vara Network Testnet
2023-12-05 20:01:19 🏷  Node name: ethereal-oranges-7690
2023-12-05 20:01:19 👤 Role: FULL
2023-12-05 20:01:19 💾 Database: RocksDb at /tmp/substratesSLmp4/chains/vara_network_testnet/db/full
2023-12-05 20:01:19 ⛓  Native runtime: vara-1030 (vara-1.tx1.au1)
2023-12-05 20:01:21 🔨 Initializing Genesis block/state (state: 0x174f…af08, header-hash: 0x5256…36d6)
2023-12-05 20:01:21 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2023-12-05 20:01:21 Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")
Error: Service(Client(VersionInvalid("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Out of memory (os error 12)")))

(Using a binary from https://get.gear.rs/gear-nightly-aarch64-unknown-linux-musl.tar.xz here cuz polkadot doesn't provide an official build for linux/aarch64, but the error is the same for polkadot)

The wasmtime version used by substrate is 8.0.1, and I'm able to reproduce it under the latest 15.0.1 release, so it's reasonable to assume any version in between are also affected

Test Case

wasmtime version: 15.0.1

use wasmtime::{Config, Engine, Error, InstanceAllocationStrategy};

fn main() -> Result<(), Error> {
    let mut config = Config::new();

    let _engine_good = Engine::new(dbg!(&config))?; // ok for both amd64 and aarch64

    config.allocation_strategy(InstanceAllocationStrategy::pooling());

    let _engine_linux_aarch64_bad = Engine::new(dbg!(&config))?; // failed for aarch64, but ok for amd64

    Ok(())
}

Steps to Reproduce

  1. clone https://github.com/btwiuse/wasmtime-linux-aarch64-dbg
  2. cargo run under linux/amd64 environment, the program runs to completion
  3. cargo run under linux/aarch64 environment, the program fails on the second Engine::new(...)?

Expected Results

the program runs to completion on both architectures

Actual Results

the program panics under linux/aarch64

Error: failed to create memory pool mapping

Caused by:
    0: mmap failed to reserve 0x5dc80000000 bytes
    1: Cannot allocate memory (os error 12)

Versions and Environment

Wasmtime version or commit: 8.0.1 & 15.0.1

Operating system: linux

Architecture: aarch64

Extra Info

If you don't have a aarch64 linux server, you can try it in a qemu virtual machine. I tested it under a aarch64 crostini vm on chromeos, and termux proot environment on a android 14 phone with 24 GB ram. I'm pretty sure the error isn't caused by insufficient memory.

view this post on Zulip Wasmtime GitHub notifications bot (Nov 15 2024 at 18:09):

alexcrichton commented on issue #7637:

I'm going to close this issue since I don't think there's much in Wasmtime we can do about this. What I can recommend though is a block like this where wasmtime serve will dynamically detect it shouldn't use the pooling allocator on some aarch64 configurations. I suspect that's what was happening here where the virtual address space only has 39 bits instead of the expected 48 which resulted in these errors.

Last updated: Nov 22 2024 at 16:03 UTC