wasmtime / issue #3695 `static_memory_forced` doesn't alw... · git-wasmtime

Stream: git-wasmtime

Topic: wasmtime / issue #3695 `static_memory_forced` doesn't alw...

Wasmtime GitHub notifications bot (Jan 17 2022 at 17:09):

bkolobara opened issue #3695:

I'm running into some troubles with the Engine configuration settings around memories in wasmtime = "0.33". There may be a few different issues here, but because everything is entangled together I will post them all here.

I'm trying to spawn 100k wasm instances and am hitting a few problems that I assume are related around virtual memory exhaustion.

This is a minimal example that demonstrates the first issue:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    config.static_memory_maximum_size(15 * 65536);
    config.static_memory_guard_size(65536);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
My reasoning here is:

config.static_memory_forced(true) will force the engine to always use static memory.

config.static_memory_maximum_size(15 * 65536) will allocate up to 15 wasm pages of virtual memory.

config.static_memory_guard_size(65536) will add one more wasm page of virtual memory.

There is also going to be one extra OS page (4kb?) of virtual memory as a guard before the memory.

This works fine on 64bit MacOs, but fails to finish on 64bit linux with an error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value:
Insufficient resources: System call failed: Cannot allocate memory (os error 12)', src/main.rs:14:60
I have also unsuccessfully tried to use dynamic memories instead:
    config.static_memory_maximum_size(0);
    config.dynamic_memory_reserved_for_growth(65536);
I assume this should force all memories to be dynamic (static_memory_maximum_size(0)), but only allocate up to 15 wasm pages of virtual memory. However, this fails again with an Insufficient resources error on Linux, but works on MacOs.

What would be a correct approach here that lets me specify a maximum virtual memory size, but also works on Linux so I can spawn more than 16k memories?

While experimenting with this, I also noticed another inconsistency. If we change the config.static_memory_maximum_size to a lower value than the wasm module is requesting:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    // 5 Wasm pages
    config.static_memory_maximum_size(5 * 65536);
    config.static_memory_guard_size(4096);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        // 10 Wasm pages requested
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
The Engine is going to use a dynamic memory, even it was configured with static_memory_forced(true). This previous example will be failing also on MacOs too with an Insufficient resources error. Only if we add a config.dynamic_memory_reserved_for_growth(65536); it will work.

Personal thoughts

There is a lot of documentation around these settings, but I have read multiple times through it and somehow still can't figure it out. The most annoying issue is that the documentation only lists defaults for 32 bit and 64 bit machines, but the behaviour also seems to differ depending on the OS you are running.

With memory64 and dynamic_memory_reserved_for_growth the lines are even more blured between static and dynamic memory. I'm wondering if maybe we could get away with just one memory model that makes it clearer what is happening in the background?

It would be a dynamic memory with the following settings:
1. memory_initial_virtual_size(u64) - The dynamic memory can grow until this point without copying.
2. memory_virtual_resize(bool) - This would turn it into a static memory.
3. memory_guard_size(u64) - The guard page.

The defaults (64bit systems) would be:
memory_initial_virtual_size(4G);
memory_virtual_resize(false);
memory_guard_size(2G);
Resulting in the same properties the current defaults have.

This model is less feature rich, but I feel like the current system is so complicated that it's almost impossible to reason about what is going to be happening when you mix multiple settings.

Wasmtime GitHub notifications bot (Jan 17 2022 at 17:09):

bkolobara labeled issue #3695:

I'm running into some troubles with the Engine configuration settings around memories in wasmtime = "0.33". There may be a few different issues here, but because everything is entangled together I will post them all here.

I'm trying to spawn 100k wasm instances and am hitting a few problems that I assume are related around virtual memory exhaustion.

This is a minimal example that demonstrates the first issue:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    config.static_memory_maximum_size(15 * 65536);
    config.static_memory_guard_size(65536);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
My reasoning here is:

config.static_memory_forced(true) will force the engine to always use static memory.

config.static_memory_maximum_size(15 * 65536) will allocate up to 15 wasm pages of virtual memory.

config.static_memory_guard_size(65536) will add one more wasm page of virtual memory.

There is also going to be one extra OS page (4kb?) of virtual memory as a guard before the memory.

This works fine on 64bit MacOs, but fails to finish on 64bit linux with an error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value:
Insufficient resources: System call failed: Cannot allocate memory (os error 12)', src/main.rs:14:60
I have also unsuccessfully tried to use dynamic memories instead:
    config.static_memory_maximum_size(0);
    config.dynamic_memory_reserved_for_growth(65536);
I assume this should force all memories to be dynamic (static_memory_maximum_size(0)), but only allocate up to 15 wasm pages of virtual memory. However, this fails again with an Insufficient resources error on Linux, but works on MacOs.

What would be a correct approach here that lets me specify a maximum virtual memory size, but also works on Linux so I can spawn more than 16k memories?

While experimenting with this, I also noticed another inconsistency. If we change the config.static_memory_maximum_size to a lower value than the wasm module is requesting:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    // 5 Wasm pages
    config.static_memory_maximum_size(5 * 65536);
    config.static_memory_guard_size(4096);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        // 10 Wasm pages requested
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
The Engine is going to use a dynamic memory, even it was configured with static_memory_forced(true). This previous example will be failing also on MacOs too with an Insufficient resources error. Only if we add a config.dynamic_memory_reserved_for_growth(65536); it will work.

Personal thoughts

There is a lot of documentation around these settings, but I have read multiple times through it and somehow still can't figure it out. The most annoying issue is that the documentation only lists defaults for 32 bit and 64 bit machines, but the behaviour also seems to differ depending on the OS you are running.

With memory64 and dynamic_memory_reserved_for_growth the lines are even more blured between static and dynamic memory. I'm wondering if maybe we could get away with just one memory model that makes it clearer what is happening in the background?

It would be a dynamic memory with the following settings:
1. memory_initial_virtual_size(u64) - The dynamic memory can grow until this point without copying.
2. memory_virtual_resize(bool) - This would turn it into a static memory.
3. memory_guard_size(u64) - The guard page.

The defaults (64bit systems) would be:
memory_initial_virtual_size(4G);
memory_virtual_resize(false);
memory_guard_size(2G);
Resulting in the same properties the current defaults have.

This model is less feature rich, but I feel like the current system is so complicated that it's almost impossible to reason about what is going to be happening when you mix multiple settings.

Wasmtime GitHub notifications bot (Jan 18 2022 at 10:31):

bkolobara edited issue #3695:

I'm running into some troubles with the Engine configuration settings around memories in wasmtime = "0.33". There may be a few different issues here, but because everything is entangled together I will post them all here.

I'm trying to spawn 100k wasm instances and am hitting a few problems that I assume are related around virtual memory exhaustion.

This is a minimal example that demonstrates the first issue:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    config.static_memory_maximum_size(15 * 65536);
    config.static_memory_guard_size(65536);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
My reasoning here is:

config.static_memory_forced(true) will force the engine to always use static memory.

config.static_memory_maximum_size(15 * 65536) will allocate up to 15 wasm pages of virtual memory.

config.static_memory_guard_size(65536) will add one more wasm page of virtual memory.

There is also going to be one extra OS page (4kb?) of virtual memory as a guard before the memory.

This works fine on 64bit MacOs, but fails to finish on 64bit linux with an error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value:
Insufficient resources: System call failed: Cannot allocate memory (os error 12)', src/main.rs:14:60
I have also unsuccessfully tried to use dynamic memories instead:
    config.static_memory_maximum_size(0);
    config.dynamic_memory_reserved_for_growth(15 * 65536);
I assume this should force all memories to be dynamic (static_memory_maximum_size(0)), but only allocate up to 15 wasm pages of virtual memory. However, this fails again with an Insufficient resources error on Linux, but works on MacOs.

What would be a correct approach here that lets me specify a maximum virtual memory size, but also works on Linux so I can spawn more than 16k memories?

While experimenting with this, I also noticed another inconsistency. If we change the config.static_memory_maximum_size to a lower value than the wasm module is requesting:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    // 5 Wasm pages
    config.static_memory_maximum_size(5 * 65536);
    config.static_memory_guard_size(4096);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        // 10 Wasm pages requested
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
The Engine is going to use a dynamic memory, even it was configured with static_memory_forced(true). This previous example will be failing also on MacOs too with an Insufficient resources error. Only if we add a config.dynamic_memory_reserved_for_growth(65536); it will work.

Personal thoughts

There is a lot of documentation around these settings, but I have read multiple times through it and somehow still can't figure it out. The most annoying issue is that the documentation only lists defaults for 32 bit and 64 bit machines, but the behaviour also seems to differ depending on the OS you are running.

With memory64 and dynamic_memory_reserved_for_growth the lines are even more blured between static and dynamic memory. I'm wondering if maybe we could get away with just one memory model that makes it clearer what is happening in the background?

It would be a dynamic memory with the following settings:
1. memory_initial_virtual_size(u64) - The dynamic memory can grow until this point without copying.
2. memory_virtual_resize(bool) - This would turn it into a static memory.
3. memory_guard_size(u64) - The guard page.

The defaults (64bit systems) would be:
memory_initial_virtual_size(4G);
memory_virtual_resize(false);
memory_guard_size(2G);
Resulting in the same properties the current defaults have.

This model is less feature rich, but I feel like the current system is so complicated that it's almost impossible to reason about what is going to be happening when you mix multiple settings.

Wasmtime GitHub notifications bot (Jan 18 2022 at 10:31):

bkolobara edited issue #3695:

I'm running into some troubles with the Engine configuration settings around memories in wasmtime = "0.33". There may be a few different issues here, but because everything is entangled together I will post them all here.

I'm trying to spawn 100k wasm instances and am hitting a few problems that I assume are related around virtual memory exhaustion.

This is a minimal example that demonstrates the first issue:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    config.static_memory_maximum_size(15 * 65536);
    config.static_memory_guard_size(65536);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
My reasoning here is:

config.static_memory_forced(true) will force the engine to always use static memory.

config.static_memory_maximum_size(15 * 65536) will allocate up to 15 wasm pages of virtual memory.

config.static_memory_guard_size(65536) will add one more wasm page of virtual memory.

There is also going to be one extra OS page (4kb?) of virtual memory as a guard before the memory.

This works fine on 64bit MacOs, but fails to finish on 64bit linux with an error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value:
Insufficient resources: System call failed: Cannot allocate memory (os error 12)', src/main.rs:14:60
I have also unsuccessfully tried to use dynamic memories instead:
    config.static_memory_maximum_size(0);
    config.dynamic_memory_reserved_for_growth(15 * 65536);
I assume this should force all memories to be dynamic (static_memory_maximum_size(0)), but only allocate up to 15 wasm pages of virtual memory initially. However, this fails again with an Insufficient resources error on Linux, but works on MacOs.

What would be a correct approach here that lets me specify a maximum virtual memory size, but also works on Linux so I can spawn more than 16k memories?

While experimenting with this, I also noticed another inconsistency. If we change the config.static_memory_maximum_size to a lower value than the wasm module is requesting:
use wasmtime::{Config, Engine, Memory, MemoryType, Store};

fn main() {
    let mut config = Config::new();
    // 5 Wasm pages
    config.static_memory_maximum_size(5 * 65536);
    config.static_memory_guard_size(4096);
    config.static_memory_forced(true);

    let engine = Engine::new(&config).unwrap();

    let mut hold = Vec::new();
    for _ in 0..100_000 {
        let mut store = Store::new(&engine, ());
        // 10 Wasm pages requested
        Memory::new(&mut store, MemoryType::new(10, None)).unwrap();
        hold.push(store);
    }
}
The Engine is going to use a dynamic memory, even it was configured with static_memory_forced(true). This previous example will be failing also on MacOs too with an Insufficient resources error. Only if we add a config.dynamic_memory_reserved_for_growth(65536); it will work.

Personal thoughts

There is a lot of documentation around these settings, but I have read multiple times through it and somehow still can't figure it out. The most annoying issue is that the documentation only lists defaults for 32 bit and 64 bit machines, but the behaviour also seems to differ depending on the OS you are running.

With memory64 and dynamic_memory_reserved_for_growth the lines are even more blured between static and dynamic memory. I'm wondering if maybe we could get away with just one memory model that makes it clearer what is happening in the background?

It would be a dynamic memory with the following settings:
1. memory_initial_virtual_size(u64) - The dynamic memory can grow until this point without copying.
2. memory_virtual_resize(bool) - This would turn it into a static memory.
3. memory_guard_size(u64) - The guard page.

The defaults (64bit systems) would be:
memory_initial_virtual_size(4G);
memory_virtual_resize(false);
memory_guard_size(2G);
Resulting in the same properties the current defaults have.

This model is less feature rich, but I feel like the current system is so complicated that it's almost impossible to reason about what is going to be happening when you mix multiple settings.

Wasmtime GitHub notifications bot (Jan 18 2022 at 18:31):

alexcrichton commented on issue #3695:

Thanks for the report on this! I completely agree the settings are confusing, even as the one who tried to document them all originally I still struggle and reread the docs every time... I think what you've proposed actually makes a lot more sense and would be great to implement. I think we can probably keep wasmtime's internals the same they are now but it should be much easier to consume with the API you've proposed (or something similar). Would you be up for making a PR for this?

For the first example of yours, though, I think Wasmtime is performing as expected. AFAIK there's no OS-specific behavior of Wasmtime itself but I think you're running into OS-specific limitations and behavior of mmap itself. Using strace to follow the programs:

The first program (forced static, 15 page maximum) I see virtual memory reservations of 17 wasm pages, which corresponds to one page before, one page after, and 15 for the static memory. This is followed by an mprotect for 10 pages which makes the first 10 pages accessible.

The second program (forced dynamic, 15 pages growth) I see virtual memory reservations of 27 wasm pages which corresponds to 1 page guard at the front, 1 at the end, 10 for the initial memory, and 15 to grow into. There's then an mprotect for 10 pages to initialize the memory.

For the third program ("forced static", 5 page maximum) I see virtual memory reservations of 2GB + 12 wasm pages. You're right in that this is mistakenly selecting the dynamic strategy so the 2GB is the default amount of room to grow into and otherwise there's a guard in front, guard at the end, and 10 pages for the initial memory.

So for the first two programs I think that Wasmtime is actually behaving as expected, although perhaps the OS is not behaving as you expect? I am not personally very familiar with the nuances of mmap on OS-es and how it relates to overcomit and such. For Linux I see that the failing syscall is mprotect where we make the initial pages read/write. Otherwise though fixing the issue where a static memory is "forced" but we still pick dynamic I think would be fixed with an easier-to-understand configuration scheme like you're proposing. The specific bug here is that instantiation of a 10-page memory into a 5-page-maximum config should simply fail, not accidentally fall back to dynamic allocation.

Wasmtime GitHub notifications bot (Jan 19 2022 at 09:45):

bkolobara commented on issue #3695:

Thank you Alex for investigating this!

You are right, this is a linux specific setting and Wasmtime is behaving as expected. The setting is vm.max_map_count and can be increased (until the next reboot) with sysctl -w vm.max_map_count=262144.

I can try to contribute both PRs, one fixing the actual bug:

The specific bug here is that instantiation of a 10-page memory into a 5-page-maximum config should simply fail, not accidentally fall back to dynamic allocation.

And one for the memory config simplification.

Wasmtime GitHub notifications bot (Jan 19 2022 at 15:03):

alexcrichton commented on issue #3695:

Awesome, sounds great, and thanks!

Last updated: Dec 23 2024 at 12:05 UTC