Stream: git-wasmtime

Topic: wasmtime / Issue #2513 Test large multi threads got panic

Wasmtime GitHub notifications bot (Dec 16 2020 at 06:25):

leenux opened Issue #2513:

• What are the steps to reproduce the issue?
  Change N_THREADS from 10 to 200000 for mock 200000 users run wasm.
  cargo run --example threads
  If N_THREADS < 20000 its work fine and no panic.

• What do you expect to happen? What does actually happen? Does it panic, and
  if so, with which assertion?

thread 'mainCreating callback...
' panicked at 'thread 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }<unnamed>', ' panicked at '/home/leenux/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rsInstantiating module...
failed to set up alternative stack guard page:', thread '610library/std/src/sys/unix/stack_overflow.rs<unnamed>::' panicked at '29156failed to set up alternative stack guard page
:', note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
library/std/src/sys/unix/stack_overflow.rs13:
156:13
Instantiating module...
Extracting export...
fatal runtime error: failed to initiate panic, error 5
Creating callback...
Creating callback...
Aborted (core dumped)

• Which Wasmtime version / commit hash / branch are you using?
  0.21

• If relevant, can you include some extra information about your environment?
  (Rust version, operating system, architecture...)
  rust:1.48
  os:ubuntu 20.04
  arch:x86-64
  

Wasmtime GitHub notifications bot (Dec 16 2020 at 06:25):

leenux labeled Issue #2513:

• What are the steps to reproduce the issue?
  Change N_THREADS from 10 to 200000 for mock 200000 users run wasm.
  cargo run --example threads
  If N_THREADS < 20000 its work fine and no panic.

• What do you expect to happen? What does actually happen? Does it panic, and
  if so, with which assertion?

thread 'mainCreating callback...
' panicked at 'thread 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }<unnamed>', ' panicked at '/home/leenux/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rsInstantiating module...
failed to set up alternative stack guard page:', thread '610library/std/src/sys/unix/stack_overflow.rs<unnamed>::' panicked at '29156failed to set up alternative stack guard page
:', note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
library/std/src/sys/unix/stack_overflow.rs13:
156:13
Instantiating module...
Extracting export...
fatal runtime error: failed to initiate panic, error 5
Creating callback...
Creating callback...
Aborted (core dumped)

• Which Wasmtime version / commit hash / branch are you using?
  0.21

• If relevant, can you include some extra information about your environment?
  (Rust version, operating system, architecture...)
  rust:1.48
  os:ubuntu 20.04
  arch:x86-64
  

Wasmtime GitHub notifications bot (Dec 16 2020 at 06:37):

bjorn3 commented on Issue #2513:

This doesn't seem to be a wasmtime problem. Rust itself fails to setup a stack guard: https://github.com/rust-lang/rust/blob/95b4a4f0eee935f9e0c80b0ceef34866bcb72ca3/library/std/src/sys/unix/stack_overflow.rs#L156 This seems to simply be resource exhaustion.

Wasmtime GitHub notifications bot (Dec 16 2020 at 07:30):

leenux commented on Issue #2513:

@bjorn3 It doesn't look like resource exhaustion from top

Wasmtime GitHub notifications bot (Dec 16 2020 at 07:43):

cfallin commented on Issue #2513:

@leenux this looks like rust-lang/rust#78497. Basically, the Rust standard library's code that sets up a guard-page mapping for a thread's stack is creating a new virtual-memory mapping for that guard page; and the Linux kernel has a limit on the number of individual virtual-memory mappings in a single address space. By default this limit is 65530 (just under 2^16), and a thread requires two new mappings (the stack, and the guard page under the stack), so by default you'll hit the limit after creating a little over 32000 threads in one process.

It looks like the sysctl for this is adjustable, though performance isn't going to be great (on any OS) with that many entries in the VM map. I might suggest green threads/fibers or an event-driven async model if you need to handle that many simultaneous requests :-)

Wasmtime GitHub notifications bot (Dec 16 2020 at 08:19):

leenux commented on Issue #2513:

@cfallin Do wasmtime has any plans to add async feature?I intend to use wasmtime to execute their own workflow script.200000 concurrent is the minimum requirement.

Wasmtime GitHub notifications bot (Dec 16 2020 at 15:58):

alexcrichton commented on Issue #2513:

You can read up more on proposed async support in Wasmtime at https://github.com/bytecodealliance/rfcs/pull/2, although depending on your use case it may not escape the virtual memory limits yet.

Wasmtime GitHub notifications bot (Dec 21 2020 at 12:21):

leenux commented on Issue #2513:

I'm using async to run wasm bytecode, benchmark is ok for our service

Wasmtime GitHub notifications bot (Dec 21 2020 at 12:21):

leenux closed Issue #2513:

• What are the steps to reproduce the issue?
  Change N_THREADS from 10 to 200000 for mock 200000 users run wasm.
  cargo run --example threads
  If N_THREADS < 20000 its work fine and no panic.

• What do you expect to happen? What does actually happen? Does it panic, and
  if so, with which assertion?

thread 'mainCreating callback...
' panicked at 'thread 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }<unnamed>', ' panicked at '/home/leenux/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rsInstantiating module...
failed to set up alternative stack guard page:', thread '610library/std/src/sys/unix/stack_overflow.rs<unnamed>::' panicked at '29156failed to set up alternative stack guard page
:', note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
library/std/src/sys/unix/stack_overflow.rs13:
156:13
Instantiating module...
Extracting export...
fatal runtime error: failed to initiate panic, error 5
Creating callback...
Creating callback...
Aborted (core dumped)

• Which Wasmtime version / commit hash / branch are you using?
  0.21

• If relevant, can you include some extra information about your environment?
  (Rust version, operating system, architecture...)
  rust:1.48
  os:ubuntu 20.04
  arch:x86-64
  

Last updated: Dec 23 2024 at 12:05 UTC