wsta (Jan 20 2023 at 15:46): wsta (Jan 20 2023 at 15:46):Hi,
I am attempting to run the Wizer benchmarks, specifically the UAP one. But I am seeing an increase in execution time instead of a decrease.
% time wasmtime uap_bench.control.wasm
wasmtime uap_bench.control.wasm 0.03s user 0.01s system 90% cpu 0.038 total
% time wasmtime uap_bench.wizer.wasm
wasmtime uap_bench.wizer.wasm 0.15s user 0.03s system 98% cpu 0.182 total
It seems like the optimized module is 5 times slower than the unoptimized one. I am benchmarking the modules that are checked in to the repository without rebuilding them, but I have also tried to rebuild the modules using build.shwithout any noticible effect on the execution time. (Except for some reason the first time I run the modules they are slower) Another thing that confuses me is that I was able to run both modules without initializing the uap-core submodule, I thought the unoptimized module, at leaast, would need access to the regexes.yamlfile inside it. (I am new to Rust, so I might just be misunderstanding something) Initializing the submodule had no visible effect on the execution time.
wasmtime -V shows wasmtime-cli 4.0.0, cargo -V shows cargo 1.66.1 (ad779e08b 2023-01-10), and I am using MacOS Ventura 13.1 arm64. Is there something I am missing?
Alex Crichton (Jan 20 2023 at 17:13):Note that by timing the wasmtime cli you're also including the time it takes to compile the module or load a cached compilation from disk which means it may not be reflective of the runtime execution speed
fitzgen (he/him) (Jan 20 2023 at 18:59):Alex is correct. That is why the benchmarks measure instantiation and execution time separately from compilation time and process init time and all of that: https://github.com/bytecodealliance/wizer/blob/main/benches/uap.rs
You should be able to repro via cargo bench in the Wizer repo.
wsta (Jan 23 2023 at 08:24):Alex Crichton said:
Note that by timing the wasmtime cli you're also including the time it takes to compile the module or load a cached compilation from disk which means it may not be reflective of the runtime execution speed
Thank you for the clarification! I am new to WASM, so sorry if I ask any obvious questions. I thought that .wasm files were already precompiled, so why does wasmtimeneed to compile it again? Is there any way to run the WASM module without compiling it? Or rather, what would I need to do to reproduce the execution time results listed in the wizer README?
Ramon Klass (Jan 23 2023 at 09:37):there is no CPU that understands wasm bytecode natively, so you always have to translate the wasm bytecode into your architecture machine code in some way, that step can't be left out entirely. These benchmarks assume that you load a wasm module once and then run it a lot of times with different data
Ramon Klass (Jan 23 2023 at 09:37):in that scenario it is realistic to ignore the initial startup
wsta (Jan 24 2023 at 12:31):Ramon Klass said:
in that scenario it is realistic to ignore the initial startup
OK, but I thought the initial startup was exactly what Wizer was meant to optimize? Wouldn't I be able to, without Wizer, invoke my initialization code only once and then run the rest of the module lots of times? For example, say I have an export called "init" and an export called "run", wouldn't I be able to call only "run" lots of times after I have called "init"? In this case it seems to me like the initial setup, like downloading the module, compiling it, and running the initialization function, is an important part to take into consideration. Basically, if I should ignore the time spent compiling, why should I not also ignore the time that the initialization function takes?
Ramon Klass (Jan 24 2023 at 12:35):to be honest I lack the insight to answer that question, and I'd also like to know the answer if anyone can chime in
bjorn3 (Jan 24 2023 at 13:06):Wizer is meant to reduce the time it takes to recreate a wasm instance many times.
bjorn3 (Jan 24 2023 at 13:08):The reason not to reuse a wasm instance many times is isolation. Say you have a web server which handles requests through a wasm module. By recreating the instance on every request you can ensure that any corruption due to bugs (or maliciously) doesn't affect any other request.
bjorn3 (Jan 24 2023 at 13:10):You could also for example have the web server check that the wasm module only accesses things the logged in user is allowed to access. If you do this reusing wasm instances between requests may allow an attacker to leak information from prior requests of another user or to setup the instance such that it stores data of another user and on the next request by the attacker present it.
wsta (Jan 24 2023 at 13:39):That makes sense, thank you for your answer. In the Wizer README it is stated that it could negitively impact network transfer times. But if I understand you correctly then this is not really related to the problem that Wizer intends to solve, since the use case that Wizer targets is when both network transfer and compiling are only done once, but initializing (and maybe instantiating) the module is done multiple times?
Ramon Klass (Jan 24 2023 at 13:43):yes, you download the module once, you compile it once, you instantiate and run it often, and wizer speeds up the run it part, as far as I understand
Last updated: Oct 23 2024 at 20:03 UTC