Wasmtime GitHub notifications bot (Jun 04 2026 at 07:27): Wasmtime GitHub notifications bot (Jun 04 2026 at 07:27):bongjunj opened issue #13554:
Hi, I was a bit fond of the following rules and trying to understand how they work.
While I was reading the code, I ran some performance benchmark to see the performance impact of them.
When I was expecting it should speed up execution time, the result turned out that it can be less than desired.
For a spidermoney-regex, this slows the execution by 5.73%The table below shows the average execution time measured with sightglass-cli using the default config (processes=10, iterations=10).
Slowdown = 1 - Upstream / NoMagic
Program Upstream NoMagic Slowdown blake3-scalar/benchmark.wasm 416809.8 415652.7 0.28% blake3-simd/benchmark.wasm 408021.7 406928.1 0.27% bz2/benchmark.wasm 114914714.4 114829391.1 0.07% pulldown-cmark/benchmark.wasm 7975220.8 7916613.0 0.73% regex/benchmark.wasm 272510604.8 272406933.4 0.04% spidermonkey/spidermonkey-json.wasm 512516258.4 536087944.1 -4.60% spidermonkey/spidermonkey-markdown.wasm 296328581.8 292117657.4 1.42% spidermonkey/spidermonkey-regex.wasm 122409216.8 115393885.2 5.73% sqlite3/sqlite3.wasm 3611039582.9 3610499560.4 0.01% Steps to Reproduce
- Upstream commit: 1f0294d34f27def54f22981ab575af2f1c258184
- NoMagic: remove the rules from upstream
Expected Results
It should speed up.
Actual Results
It slowed down.
Versions and Environment
Cranelift version or commit: stated above
Operating system: Linux 6.8.0-110-generic #110-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 19 15:09:20 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Architecture:
<details>
<summary>
lscpu
</summary>Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 2 Stepping: 6 CPU(s) scaling MHz: 37% CPU max MHz: 3500.0000 CPU min MHz: 800.0000 BogoMIPS: 5800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 1.5 MiB (32 instances) L1i cache: 1 MiB (32 instances) L2 cache: 40 MiB (32 instances) L3 cache: 48 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-15,32-47 NUMA node1 CPU(s): 16-31,48-63 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Indirect target selection: Mitigation; Aligned branch/return thunks Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected</details>
Wasmtime GitHub notifications bot (Jun 04 2026 at 07:27):bongjunj added the bug label to Issue #13554.
Wasmtime GitHub notifications bot (Jun 04 2026 at 07:27):bongjunj added the cranelift label to Issue #13554.
Wasmtime GitHub notifications bot (Jun 04 2026 at 07:27):bongjunj edited issue #13554:
Hi, I was a bit fond of the following rules and trying to understand how they work.
While I was reading the code, I ran some performance benchmark to see the performance impact of them.
When I was expecting it should speed up execution time, the result turned out that it can be less than desired.
For a spidermoney-regex, this slows the execution by 5.73%The table below shows the average execution time measured with sightglass-cli using the default config (processes=10, iterations=10).
Slowdown = 1 - Upstream / NoMagic
Program Upstream NoMagic Slowdown blake3-scalar/benchmark.wasm 416809.8 415652.7 0.28% blake3-simd/benchmark.wasm 408021.7 406928.1 0.27% bz2/benchmark.wasm 114914714.4 114829391.1 0.07% pulldown-cmark/benchmark.wasm 7975220.8 7916613.0 0.73% regex/benchmark.wasm 272510604.8 272406933.4 0.04% spidermonkey/spidermonkey-json.wasm 512516258.4 536087944.1 -4.60% spidermonkey/spidermonkey-markdown.wasm 296328581.8 292117657.4 1.42% spidermonkey/spidermonkey-regex.wasm 122409216.8 115393885.2 5.73% sqlite3/sqlite3.wasm 3611039582.9 3610499560.4 0.01% Steps to Reproduce
- Upstream commit: 1f0294d34f27def54f22981ab575af2f1c258184
- NoMagic: remove the rules from upstream
Expected Results
It should speed up.
Actual Results
It slowed down.
Versions and Environment
Cranelift version or commit: stated above
Operating system: Linux 6.8.0-110-generic #110-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 19 15:09:20 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Architecture:
<details>
<summary>
lscpu
</summary>Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 2 Stepping: 6 CPU(s) scaling MHz: 37% CPU max MHz: 3500.0000 CPU min MHz: 800.0000 BogoMIPS: 5800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 1.5 MiB (32 instances) L1i cache: 1 MiB (32 instances) L2 cache: 40 MiB (32 instances) L3 cache: 48 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-15,32-47 NUMA node1 CPU(s): 16-31,48-63 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Indirect target selection: Mitigation; Aligned branch/return thunks Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected</details>
Wasmtime GitHub notifications bot (Jun 04 2026 at 17:22):cfallin commented on issue #13554:
@bongjunj thanks for filing this issue! It is definitely surprising: a multiply instruction (with the reciprocal in the mod-2^N multiplicative group) should always be faster than a divide instruction. So it's possible/likely that there are some other weird interactions going on here, perhaps with regalloc or code scheduling or something causing DCE or const-prop not to work or ...
Would you be willing to dig a bit deeper and isolate the particular hot machine code that changes? That would help us isolate this further. We can of course do this eventually to but as long as you're investigating, if able, it'd be helpful. Thanks!
Wasmtime GitHub notifications bot (Jun 05 2026 at 05:49):bongjunj commented on issue #13554:
Thanks for the reply! @cfallin I'm digging it further at the moment, and will get back as soon as I find something.
Wasmtime GitHub notifications bot (Jul 14 2026 at 07:07):bongjunj commented on issue #13554:
@cfallin
Back from a conference and a long vacation! Sorry for the delay putting these results together.
I experimented with Intel Xeon 6326 CPU, and
perf stat ...reported consistently thaticache_data.stallsincreases with the magic div/rem rules enabled. This could indicate the longer instruction sequence (resulted from expanding an integer division instruction) can interfere with instruction cache and fetch/decode pipeline.
no magichere means the data is from a cranelift compiler without the magic div/rem rules, andmagicindicates the data is from a cranelift compiler with those rules.- Only four cases (spidermonkey-{regex,markdown}, pulldown-cmark, sqlite3) are here because the slowdowns have been consistently observed.
Program Cycles (no magic) Cycles (magic) Instructions (no magic) Instructions (magic) CPI (no magic) CPI (magic) icache_data.stalls(no magic)icache_data.stalls(magic)spidermonkey-regex 31,344,498,031 31,444,136,679 58,783,481,950 58,798,325,453 0.533219 0.534779 2,132,223,420 2,154,794,511 spidermoney-markdown 35,104,553,656 35,155,308,680 67,218,071,025 67,234,948,959 0.522249 0.522873 2,255,115,231 2,273,552,987 pulldown-cmark 1,071,760,342 1,077,104,406 1,959,738,871 1,959,124,958 0.546889 0.549789 74,061,363 75,000,735 sqlite3 71,233,836,698 71,527,251,425 213,399,652,709 214,828,098,679 0.333805 0.332951 876,017,651 932,663,503 The magic div/rem rules certainly increased the code size of some functions. In sqlite3, 13 division (
idivl) instructions have been converted to 22 multiplies and 26 shifts. This might imply that the increased code size wasted the instruction cache capacity and thus degraded the processor's instruction fetch/decode performance.To the best of my knowledge, such hardware specific details are not fully modeled in Cranelift backends.
And balancing code size and (better) instruction sequence is known to be tricky.
I would like to hear thoughts on how can we mitigate this kind of issues.. perhaps toggling some rules based on the target hardware or some static analysis?
Wasmtime GitHub notifications bot (Jul 20 2026 at 16:47):cfallin commented on issue #13554:
Thanks for this data!
You're correct that we don't model the impact of code size/density on performance. It would be hard to have a specific model for that at our level of implementation (along with e.g. the fact that we don't model microarchitectures generally) -- so the best we can do is to minimize code size generally, balanced with other factors.
The aspect that really interests me here is that you're seeing such significant movement in the cycle count. I would expect more dynamic instruction count (the static count is higher per site, as you note), but divides should be very slow compared to multiplies and shifts (per Agner Fog's tables, on Ice Lake, a recent Intel microarchitecture, 32-bit integer divide (
idivl) is 12 cycles; 32-bit multiply is 4 cycles; a shift is 1 cycle). From first principles, we should absolutely take 1 or 2 multiplies + shifts in exchange for a divide -- both the critical path length (latency) and overall execution unit occupancy should improve. Would it be possible for you to dig in further and try to either find a minimal kernel and make a reproducer out of it, or profile at the instruction level and do an instruction-by-instruction performance diff?
Wasmtime GitHub notifications bot (Jul 27 2026 at 13:08):bongjunj commented on issue #13554:
I looked closer to the sqlite3 benchmark.
Interestingly, the division (in the no-magic version) and its magic instruction sequence (in the upstream version) were never executed!
So, in other words, it is true that the optimized instruction sequence should take less cycles (obviously)... but, in this case, those were never run.
This points to other possibilities..The hightest function is
wasm[0]::function[191]+0x14d60(this is probably thesqlite3VdbeExecinterpreter loop -- this fact is vibe-guessed by codex-gpt-5.6-sol).The magic-div/rem optimizations are confirmed to be working well:
;; no-magic version 7ffb7544cbfb: mov 0x70(%rsp),%rax 7ffb7544cc00: mov 0x78(%rsp),%rdx 7ffb7544cc05: sub %edx,%eax 7ffb7544cc07: cltd 7ffb7544cc08: mov $0x14,%r8d 7ffb7544cc0e: idiv %r8dturns into
;; upstream 76fc4e9b2bd6: mov 0x70(%rsp),%rax 76fc4e9b2bdb: mov 0x78(%rsp),%rdx 76fc4e9b2be0: sub %edx,%eax 76fc4e9b2be2: imull 0x12378(%rip) # 14de0 <wasm[0]::function[191]+0x14d60> 76fc4e9b2be8: sar $0x3,%edx 76fc4e9b2beb: mov %rdx,%rax 76fc4e9b2bee: shr $0x1f,%eax 76fc4e9b2bf1: add %edx,%eaxIt was interesting that stack frame size did not change unlike previous performance gaps.
Both producedsub $0x230,%rsp:;; upstream 0000000000000080 <wasm[0]::function[191]>: 80: 55 push %rbp 81: 48 89 e5 mov %rsp,%rbp 84: 4c 8b 57 08 mov 0x8(%rdi),%r10 88: 4d 8b 52 18 mov 0x18(%r10),%r10 8c: 49 81 c2 40 02 00 00 add $0x240,%r10 93: 4c 3b d4 cmp %rsp,%r10 96: 0f 87 09 4d 01 00 ja 14da5 <wasm[0]::function[191]+0x14d25> 9c: 48 81 ec 30 02 00 00 sub $0x230,%rsp;; no-magic Disassembly of section .text: 0000000000000080 <wasm[0]::function[191]>: 80: 55 push %rbp 81: 48 89 e5 mov %rsp,%rbp 84: 4c 8b 57 08 mov 0x8(%rdi),%r10 88: 4d 8b 52 18 mov 0x18(%r10),%r10 8c: 49 81 c2 40 02 00 00 add $0x240,%r10 93: 4c 3b d4 cmp %rsp,%r10 96: 0f 87 2c 42 01 00 ja 142c8 <wasm[0]::function[191]+0x14248> 9c: 48 81 ec 30 02 00 00 sub $0x230,%rspNotably, the number of data movements increased.
I uploaded the disasssembly files for verifying the result:
wasm-function-191.x86_64.nomagic.txt
wasm-function-191.x86_64.upstream.txt
%rsp-related moves: 6,035 -> 6,442 (+6.7%)- stack loads: 4,641 -> 5,044 (+8.7%)
- total code (.text) size: 82,752 byte -> 85,536 byte
The increase of %rsp-moves and stack (re-)loads looks suspicious, possibly indicating that this regression relates to register allocation.
I'm not aware of the regalloc2 implementation, so I'm not quite sure that this interpretation is valid at the moment.
Wasmtime GitHub notifications bot (Jul 27 2026 at 13:08):bongjunj edited a comment on issue #13554:
I looked closer to the sqlite3 benchmark.
Interestingly, the division (in the no-magic version) and its magic instruction sequence (in the upstream version) were never executed!
So, in other words, it is true that the optimized instruction sequence should take less cycles (obviously)... but, in this case, those were never run.
This points to other possibilities..The hightest function is
wasm[0]::function[191]+0x14d60(this is probably thesqlite3VdbeExecinterpreter loop -- this fact is vibe-guessed by codex-gpt-5.6-sol).The magic-div/rem optimizations are confirmed to be working well:
;; no-magic version 7ffb7544cbfb: mov 0x70(%rsp),%rax 7ffb7544cc00: mov 0x78(%rsp),%rdx 7ffb7544cc05: sub %edx,%eax 7ffb7544cc07: cltd 7ffb7544cc08: mov $0x14,%r8d 7ffb7544cc0e: idiv %r8dturns into
;; upstream 76fc4e9b2bd6: mov 0x70(%rsp),%rax 76fc4e9b2bdb: mov 0x78(%rsp),%rdx 76fc4e9b2be0: sub %edx,%eax 76fc4e9b2be2: imull 0x12378(%rip) # 14de0 <wasm[0]::function[191]+0x14d60> 76fc4e9b2be8: sar $0x3,%edx 76fc4e9b2beb: mov %rdx,%rax 76fc4e9b2bee: shr $0x1f,%eax 76fc4e9b2bf1: add %edx,%eaxIt was interesting that stack frame size did not change unlike previous performance gaps.
Both producedsub $0x230,%rsp:;; upstream 0000000000000080 <wasm[0]::function[191]>: 80: 55 push %rbp 81: 48 89 e5 mov %rsp,%rbp 84: 4c 8b 57 08 mov 0x8(%rdi),%r10 88: 4d 8b 52 18 mov 0x18(%r10),%r10 8c: 49 81 c2 40 02 00 00 add $0x240,%r10 93: 4c 3b d4 cmp %rsp,%r10 96: 0f 87 09 4d 01 00 ja 14da5 <wasm[0]::function[191]+0x14d25> 9c: 48 81 ec 30 02 00 00 sub $0x230,%rsp;; no-magic Disassembly of section .text: 0000000000000080 <wasm[0]::function[191]>: 80: 55 push %rbp 81: 48 89 e5 mov %rsp,%rbp 84: 4c 8b 57 08 mov 0x8(%rdi),%r10 88: 4d 8b 52 18 mov 0x18(%r10),%r10 8c: 49 81 c2 40 02 00 00 add $0x240,%r10 93: 4c 3b d4 cmp %rsp,%r10 96: 0f 87 2c 42 01 00 ja 142c8 <wasm[0]::function[191]+0x14248> 9c: 48 81 ec 30 02 00 00 sub $0x230,%rspNotably, the number of data movements increased.
I uploaded the disasssembly files for verifying the result:
wasm-function-191.x86_64.nomagic.txt
wasm-function-191.x86_64.upstream.txt
%rsp-related moves: 6,035 -> 6,442 (+6.7%)- stack loads: 4,641 -> 5,044 (+8.7%)
- total code (.text) size: 82,752 byte -> 85,536 byte
The increase of %rsp-moves and stack (re-)loads looks suspicious, possibly indicating that this regression relates to register allocation.
I'm not aware of the regalloc2 implementation, so I'm not quite sure that this interpretation is valid at the moment.Thanks for reading!
Last updated: Jul 29 2026 at 05:03 UTC