Stream: git-wasmtime

Topic: wasmtime / issue #8233 Cranelift: incosistent execution r...

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 09:12):

candymate added the bug label to Issue #8233.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 09:12):

candymate added the fuzz-bug label to Issue #8233.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 09:12):

candymate opened issue #8233:

Test Case

// main.rs
use wasmtime::*;

fn main() -> Result<()> {
    let mut config = Config::default();
    config.strategy(Strategy::Cranelift);
    config.cranelift_opt_level(OptLevel::None);

    let engine1 = Engine::new(&config)?;
    let mut new_config = config.clone();
    new_config.cranelift_opt_level(OptLevel::Speed);
    let engine2 = Engine::new(&new_config)?;
    let wat = r#"
    (module
        (type (;0;) (func (param i64 i64 i64) (result i64)))
        (import "mem" "mem" (memory (;0;) 1))
        (func (;0;) (type 0) (param i64 i64 i64) (result i64)
          i64.const -1
          i64.const 0
          local.get 0
          local.get 1
          i64.lt_s
          select
          local.get 2
          i64.shr_u
          i64.extend8_s)
        (export "main" (func 0)))
    "#;
    let module1 = Module::new(&engine1, wat)?;
    let module2 = Module::new(&engine2, wat)?;
    let mut store1 = Store::new(&engine1, ());
    let mut store2 = Store::new(&engine2, ());

    let memory_ty = MemoryType::new(1, None);
    let memory1 = Memory::new(&mut store1, memory_ty.clone())?;
    let memory2 = Memory::new(&mut store2, memory_ty)?;

    let instance1 = Instance::new(&mut store1, &module1, &[memory1.into()])?;
    let instance2 = Instance::new(&mut store2, &module2, &[memory2.into()])?;
    let main1 = instance1.get_func(&mut store1, "main")
        .expect("`main` was not an exported function");
    let main2 = instance2.get_func(&mut store2, "main")
        .expect("`main` was not an exported function");
    let params = vec![
        Val::I64(0),
        Val::I64(1),
        Val::I64(2),
    ];
    let mut results1 = vec![Val::I64(0)];
    let mut results2 = vec![Val::I64(0)];
    println!("Opt level None: {:?}", main1.call(
        &mut store1,
        &params,
        &mut results1
    ));
    println!("{:?}", results1);
    println!("--------------");
    println!("Opt level Speed: {:?}", main2.call(
        &mut store2,
        &params,
        &mut results2
    ));
    println!("{:?}", results2);

    Ok(())
}

[package]
name = "wasmtime-wrapper"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wasmtime = { path = "../wasmtime/crates/wasmtime" }
#wasmtime = "19.0.0"

Steps to reproduce

cargo run --release --target=aarch64-unknown-linux-gnu

QEMU run options I'm currently using is the following:

qemu-aarch64 -L /usr/aarch64-linux-gnu -E LD_LIBRARY_PATH=/usr/aarch64-linux-gnu/lib -E WASMTIME_TEST_NO_HOG_MEMORY=1 target/aarch64-unknown-linux-gnu/release/wasmtime-wrapper

Expected Results

Both results from none and speed optimizations should show the same result -1

Opt level None: Ok(())
[I64(-1)]
--------------
Opt level Speed: Ok(())
[I64(-1)]

Actual Results

The result from speed optimization is 0x3fffffffffffffff, which is the wrong value.

Opt level None: Ok(())
[I64(-1)]
--------------
Opt level Speed: Ok(())
[I64(4611686018427387903)]

Versions and Environment

Extra Info

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 14:42):

alexcrichton commented on issue #8233:

Thanks for the report! I'm had a surprising amount of trouble reproducing this though, so I'm wondering if this isn't perhaps a different bug lurking somewhere. Locally I can't reproduce on a macOS arm64 machine, a native arm64 Linux machine, or on an x86_64 Linux machine with QEMU emulation. I noticed though that my QEMU was 8.1.5 (what we use in CI) rather than your local 8.2.1.

That appears to be the cause of the issue though, that QEMU 8.2.1 is required. I wonder if this is perhaps QEMU bug in that case?

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 19:05):

alexcrichton commented on issue #8233:

Bisection on qemu itself shows qemu/qemu@95bf306e3a058a38f5adb27be2ac598134b159d9 is the first "bad" commit. Not that I understand anything about QEMU source itself...

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 19:08):

candymate commented on issue #8233:

@alexcrichton I tried to downgrade QEMU to 8.1.5 and found that I'm getting the correct results (which is -1). Maybe this is a QEMU bug rather than a bug in Cranelift?

FYI, CLIF I'm getting now is:

function u0:0(i64 vmctx, i64, i64, i64, i64) -> i64 fast {
    gv0 = vmctx
    gv1 = load.i64 notrap aligned readonly gv0+8
    gv2 = load.i64 notrap aligned gv1
    sig0 = (i64 vmctx, i32 uext, i32 uext) -> i32 uext system_v
    sig1 = (i64 vmctx, i32 uext) -> i32 uext system_v
    stack_limit = gv2

                                block0(v0: i64, v1: i64, v2: i64, v3: i64, v4: i64):
@0033                               v6 = iconst.i64 -1
@0035                               v7 = iconst.i64 0
@003b                               v8 = icmp slt v2, v3
@003b                               v9 = uextend.i32 v8
@003c                               v10 = select v9, v6, v7  ; v6 = -1, v7 = 0
@003f                               v11 = ushr v10, v4
@0040                               v12 = ireduce.i8 v11
@0040                               v13 = sextend.i64 v12
@0041                               jump block1(v13)

                                block1(v5: i64):
@0041                               return v5
}

Now I'm trying to upgrade QEMU to try again...

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 19:11):

alexcrichton commented on issue #8233:

Yeah that's what I'm thinking. I'll note that on the master branch of QEMU I still see a non-minus-one return value, so I think for this you may need to downgrade to 8.1.5 rather than upgrade perhaps? That such a bug has gone unnoticed for so long though is also suspicious...

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 19:21):

candymate commented on issue #8233:

I also tested on QEMU v8.2.2 and v9.0.0-rc0, and now I'm pretty sure that this is a newly introduced QEMU bug. I think I'll need to downgrade QEMU to v8.1.5 for now...

Anyway, thanks for your work! I think you can close this issue whenever you want.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 19:28):

alexcrichton closed issue #8233:

Test Case

// main.rs
use wasmtime::*;

fn main() -> Result<()> {
    let mut config = Config::default();
    config.strategy(Strategy::Cranelift);
    config.cranelift_opt_level(OptLevel::None);

    let engine1 = Engine::new(&config)?;
    let mut new_config = config.clone();
    new_config.cranelift_opt_level(OptLevel::Speed);
    let engine2 = Engine::new(&new_config)?;
    let wat = r#"
    (module
        (type (;0;) (func (param i64 i64 i64) (result i64)))
        (import "mem" "mem" (memory (;0;) 1))
        (func (;0;) (type 0) (param i64 i64 i64) (result i64)
          i64.const -1
          i64.const 0
          local.get 0
          local.get 1
          i64.lt_s
          select
          local.get 2
          i64.shr_u
          i64.extend8_s)
        (export "main" (func 0)))
    "#;
    let module1 = Module::new(&engine1, wat)?;
    let module2 = Module::new(&engine2, wat)?;
    let mut store1 = Store::new(&engine1, ());
    let mut store2 = Store::new(&engine2, ());

    let memory_ty = MemoryType::new(1, None);
    let memory1 = Memory::new(&mut store1, memory_ty.clone())?;
    let memory2 = Memory::new(&mut store2, memory_ty)?;

    let instance1 = Instance::new(&mut store1, &module1, &[memory1.into()])?;
    let instance2 = Instance::new(&mut store2, &module2, &[memory2.into()])?;
    let main1 = instance1.get_func(&mut store1, "main")
        .expect("`main` was not an exported function");
    let main2 = instance2.get_func(&mut store2, "main")
        .expect("`main` was not an exported function");
    let params = vec![
        Val::I64(0),
        Val::I64(1),
        Val::I64(2),
    ];
    let mut results1 = vec![Val::I64(0)];
    let mut results2 = vec![Val::I64(0)];
    println!("Opt level None: {:?}", main1.call(
        &mut store1,
        &params,
        &mut results1
    ));
    println!("{:?}", results1);
    println!("--------------");
    println!("Opt level Speed: {:?}", main2.call(
        &mut store2,
        &params,
        &mut results2
    ));
    println!("{:?}", results2);

    Ok(())
}

[package]
name = "wasmtime-wrapper"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wasmtime = { path = "../wasmtime/crates/wasmtime" }
#wasmtime = "19.0.0"

Steps to reproduce

cargo run --release --target=aarch64-unknown-linux-gnu

QEMU run options I'm currently using is the following:

qemu-aarch64 -L /usr/aarch64-linux-gnu -E LD_LIBRARY_PATH=/usr/aarch64-linux-gnu/lib -E WASMTIME_TEST_NO_HOG_MEMORY=1 target/aarch64-unknown-linux-gnu/release/wasmtime-wrapper

Expected Results

Both results from none and speed optimizations should show the same result -1

Opt level None: Ok(())
[I64(-1)]
--------------
Opt level Speed: Ok(())
[I64(-1)]

Actual Results

The result from speed optimization is 0x3fffffffffffffff, which is the wrong value.

Opt level None: Ok(())
[I64(-1)]
--------------
Opt level Speed: Ok(())
[I64(4611686018427387903)]

Versions and Environment

Extra Info

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 19:28):

alexcrichton commented on issue #8233:

To confirm this is a QEMU issue I ended up mimizing to:

// foo.c
#include <stdio.h>
#include <stdint.h>

int64_t callme(size_t _1, size_t _2, int64_t a, int64_t b, int64_t c);

int main() {
    int64_t ret = callme(0, 0, 0, 1, 2);
    printf("%ld\n", ret);
    return 0;
}

// foo.S
.global callme
callme:
  cmp   x2, x3
  cset  x12, lt
  and   w11, w12, #0xff
  cmp   w11, #0x0
  csetm x14, ne
  lsr   x13, x14, x4
  sxtb  x0, w13
  ret

compiled as:

$ aarch64-linux-gnu-gcc foo.c foo.S -o foo

On native that prints -1 and on qemu 8.2.1 it prints 4611686018427387903.

Interestingly if I enable -g with QEMU and single-step in qemu it also prints -1.

In any case I agree it's QEMU so I'm going to close this.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 19:55):

bjorn3 commented on issue #8233:

Are you going to open a qemu issue to get this fixed?

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 21:17):

alexcrichton commented on issue #8233:

Ah no I wasn't planning on personally doing that, no

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 23:49):

candymate commented on issue #8233:

I'll report this to QEMU by today and let you know about it. Thank you.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 25 2024 at 23:50):

candymate edited a comment on issue #8233:

I'll report this to QEMU on today and let you know about it. Thank you.

view this post on Zulip Wasmtime GitHub notifications bot (Mar 26 2024 at 04:58):

candymate edited a comment on issue #8233:

I'll report this to QEMU on today and let you know about it. Thank you.

Update: I reported this to QEMU: https://gitlab.com/qemu-project/qemu/-/issues/2248

view this post on Zulip Wasmtime GitHub notifications bot (Apr 09 2024 at 12:46):

pm215 commented on issue #8233:

just FYI, this QEMU bug will be fixed in the upcoming 9.0 release and backported to the relevant stable branches.

Last updated: Nov 22 2024 at 16:03 UTC