Stream: git-wasmtime

Topic: wasmtime / issue #4307 New Fuzz Target for Exercising Int...


view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 18:31):

fitzgen opened issue #4307:

To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.

This fuzz target should:

This tests that we can round trip interface values of type T both as arguments
and returns to and from Wasm.

Here is a diagram showing what the fuzz target should do:

+-------------------+                   +------------------+
| Host              |                   | Wasm             |
|                   |   argument        |                  |
|     arbitrary ------------------------------>            |
|     value         |                   |      |           |
|                   |                   |      |           |
|                   |                   |      |           |
|         +-----------------+           |      |           |
|         | host function   |           |      |           |
|         | imported by     |           |      |           |
|         | Wasm            |  argument |      V           |
|         |            <-----------------------            |
|         |           |     |           |                  |
|         | assert    |     |           |                  |
|         | value is  |     |           |                  |
|         | equal     V     |  return   |                  |
|         |            ----------------------->            |
|         |                 |           |      |           |
|         +-----------------+           |      |           |
|                   |                   |      |           |
|                   |                   |      V           |
|   assert    <--------------------------------            |
|   value is        |                   |                  |
|   equal           |                   |                  |
|                   |                   |                  |
+-------------------+                   +------------------+

Test Case Generator

The test case generator will be responsible for generating

Oracle

There are two versions of the oracle:

  1. a version that uses the dynamic interface types API to pass values to Wasm,
    and

  2. a version that uses the static interface types API to pass values to Wasm.

Dynamic API Oracle

The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent of wasmtime::Val).

Static API Oracle

The static API oracle will use the test case generator at build time to
generate, say, 1000 interface types T to fuzz, and then generate arbitrary
instances of T at runtime. This means that if T is a struct, for example, we
can define a Rust type like

#[derive(InterfaceType)]
struct MyRustType {
    foo: u32,
    bar: String,
}

whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance of MyRustType at runtime
in this scenario, but not new types T.

(Note that derive(InterfaceType) does not exist at the time of writing.)

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 18:36):

fitzgen commented on issue #4307:

We might actually want to have two instances of the Wasm component, so that values flow like

argument: host ---> wasm A ---> wasm B ---> host import
return: host import ---> wasm B ---> wasm A ---> host

so that we are exercising component-to-component trampolines and all that.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 20:15):

dicej commented on issue #4307:

Thanks, for writing this up, @fitzgen. A few of questions for you:

  1. Given that neither the dynamic API for interface type values nor derive(InterfaceType) yet exist, I assume each of those things will need to be implemented before the corresponding oracle is created. Is that correct?
  2. Is it worthwhile to do any work on this issue before one of the above features is implemented? I.e. is there any point in writing the test case generator without any oracles to use it?
  3. For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 20:16):

dicej edited a comment on issue #4307:

Thanks, for writing this up, @fitzgen. A few of questions for you:

  1. Given that neither the dynamic API for interface type values nor derive(InterfaceType) yet exist, I assume each of those things will need to be implemented before the corresponding oracle is created. Is that correct?
  2. Is it worthwhile to do any work on this issue before one of the above features is implemented? I.e. is there any value in writing the test case generator prior to being able to write oracles that use it?
  3. For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 20:26):

fitzgen commented on issue #4307:

Yeah, I think it makes sense to unblock this before working on it. We could probably technically implement some (most?) of the test case generator without the oracles, but I think it would be best if they were developed in together since I think the shape of the generator may be informed by the oracles.

For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?

I think a build.rs would be better, since it would generally just be easier to write, read, and debug.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 20:29):

fitzgen edited issue #4307:

To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.

This fuzz target should:

This tests that we can round trip interface values of type T both as arguments
and returns to and from Wasm.

Here is a diagram showing what the fuzz target should do:

+-------------------+                   +------------------+
| Host              |                   | Wasm             |
|                   |   argument        |                  |
|     arbitrary ------------------------------>            |
|     value         |                   |      |           |
|                   |                   |      |           |
|                   |                   |      |           |
|         +-----------------+           |      |           |
|         | host function   |           |      |           |
|         | imported by     |           |      |           |
|         | Wasm            |  argument |      V           |
|         |            <-----------------------            |
|         |           |     |           |                  |
|         | assert    |     |           |                  |
|         | value is  |     |           |                  |
|         | equal     V     |  return   |                  |
|         |            ----------------------->            |
|         |                 |           |      |           |
|         +-----------------+           |      |           |
|                   |                   |      |           |
|                   |          return   |      V           |
|   assert    <--------------------------------            |
|   value is        |                   |                  |
|   equal           |                   |                  |
|                   |                   |                  |
+-------------------+                   +------------------+

Test Case Generator

The test case generator will be responsible for generating

Oracle

There are two versions of the oracle:

  1. a version that uses the dynamic interface types API to pass values to Wasm,
    and

  2. a version that uses the static interface types API to pass values to Wasm.

Dynamic API Oracle

The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent of wasmtime::Val).

Static API Oracle

The static API oracle will use the test case generator at build time to
generate, say, 1000 interface types T to fuzz, and then generate arbitrary
instances of T at runtime. This means that if T is a struct, for example, we
can define a Rust type like

#[derive(InterfaceType)]
struct MyRustType {
    foo: u32,
    bar: String,
}

whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance of MyRustType at runtime
in this scenario, but not new types T.

(Note that derive(InterfaceType) does not exist at the time of writing.)

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 20:32):

dicej edited a comment on issue #4307:

Thanks, for writing this up, @fitzgen. A few questions for you:

  1. Given that neither the dynamic API for interface type values nor derive(InterfaceType) yet exist, I assume each of those things will need to be implemented before the corresponding oracle is created. Is that correct?
  2. Is it worthwhile to do any work on this issue before one of the above features is implemented? I.e. is there any value in writing the test case generator prior to being able to write oracles that use it?
  3. For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 21:03):

dicej commented on issue #4307:

@alexcrichton Are you planning to create an issue for derive(InterfaceType? I'd be happy to create a placeholder, but I'm still fuzzy on the details, so I'll need help fleshing it out.

Also, it sounds like we might also need an issue for the interface-type-equivalent of wasmtime::Val Nick mentioned above. I _think_ I understand what's involved there (e.g. copying values.rs and adapting it for interface types), but I could use guidance there, too.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 21:12):

alexcrichton commented on issue #4307:

I do indeed want to open an issue! Sorry I got caught up all day in a debugging adventure but that's over now so I'll go make an issue next.

view this post on Zulip Wasmtime GitHub notifications bot (Jun 23 2022 at 21:12):

alexcrichton labeled issue #4307:

To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.

This fuzz target should:

This tests that we can round trip interface values of type T both as arguments
and returns to and from Wasm.

Here is a diagram showing what the fuzz target should do:

+-------------------+                   +------------------+
| Host              |                   | Wasm             |
|                   |   argument        |                  |
|     arbitrary ------------------------------>            |
|     value         |                   |      |           |
|                   |                   |      |           |
|                   |                   |      |           |
|         +-----------------+           |      |           |
|         | host function   |           |      |           |
|         | imported by     |           |      |           |
|         | Wasm            |  argument |      V           |
|         |            <-----------------------            |
|         |           |     |           |                  |
|         | assert    |     |           |                  |
|         | value is  |     |           |                  |
|         | equal     V     |  return   |                  |
|         |            ----------------------->            |
|         |                 |           |      |           |
|         +-----------------+           |      |           |
|                   |                   |      |           |
|                   |          return   |      V           |
|   assert    <--------------------------------            |
|   value is        |                   |                  |
|   equal           |                   |                  |
|                   |                   |                  |
+-------------------+                   +------------------+

Test Case Generator

The test case generator will be responsible for generating

Oracle

There are two versions of the oracle:

  1. a version that uses the dynamic interface types API to pass values to Wasm,
    and

  2. a version that uses the static interface types API to pass values to Wasm.

Dynamic API Oracle

The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent of wasmtime::Val).

Static API Oracle

The static API oracle will use the test case generator at build time to
generate, say, 1000 interface types T to fuzz, and then generate arbitrary
instances of T at runtime. This means that if T is a struct, for example, we
can define a Rust type like

#[derive(InterfaceType)]
struct MyRustType {
    foo: u32,
    bar: String,
}

whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance of MyRustType at runtime
in this scenario, but not new types T.

(Note that derive(InterfaceType) does not exist at the time of writing.)

view this post on Zulip Wasmtime GitHub notifications bot (Jul 07 2022 at 16:40):

jameysharp labeled issue #4307:

To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.

This fuzz target should:

This tests that we can round trip interface values of type T both as arguments
and returns to and from Wasm.

Here is a diagram showing what the fuzz target should do:

+-------------------+                   +------------------+
| Host              |                   | Wasm             |
|                   |   argument        |                  |
|     arbitrary ------------------------------>            |
|     value         |                   |      |           |
|                   |                   |      |           |
|                   |                   |      |           |
|         +-----------------+           |      |           |
|         | host function   |           |      |           |
|         | imported by     |           |      |           |
|         | Wasm            |  argument |      V           |
|         |            <-----------------------            |
|         |           |     |           |                  |
|         | assert    |     |           |                  |
|         | value is  |     |           |                  |
|         | equal     V     |  return   |                  |
|         |            ----------------------->            |
|         |                 |           |      |           |
|         +-----------------+           |      |           |
|                   |                   |      |           |
|                   |          return   |      V           |
|   assert    <--------------------------------            |
|   value is        |                   |                  |
|   equal           |                   |                  |
|                   |                   |                  |
+-------------------+                   +------------------+

Test Case Generator

The test case generator will be responsible for generating

Oracle

There are two versions of the oracle:

  1. a version that uses the dynamic interface types API to pass values to Wasm,
    and

  2. a version that uses the static interface types API to pass values to Wasm.

Dynamic API Oracle

The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent of wasmtime::Val).

Static API Oracle

The static API oracle will use the test case generator at build time to
generate, say, 1000 interface types T to fuzz, and then generate arbitrary
instances of T at runtime. This means that if T is a struct, for example, we
can define a Rust type like

#[derive(InterfaceType)]
struct MyRustType {
    foo: u32,
    bar: String,
}

whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance of MyRustType at runtime
in this scenario, but not new types T.

(Note that derive(InterfaceType) does not exist at the time of writing.)

view this post on Zulip Wasmtime GitHub notifications bot (Jul 13 2022 at 22:36):

dicej commented on issue #4307:

FYI, I'm planning to start working on this tomorrow, assuming no significant additional work is needed on #4442.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 15:48):

dicej commented on issue #4307:

Looks like #4442 needs more work, so I won't be able to tackle this one quite yet.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 15 2022 at 17:53):

dicej commented on issue #4307:

Quick question: Should the test case generator deliberately generate instances and components which do _not_ match the generated type?

For example, if the generated type is a record with three fields, should the instance generator (sometimes) generate values with too many, not enough, or mis-typed field values (or values which are instances of an entirely different sort of type), and should the component generator (sometimes) generate a component which expects a different record type (or a different sort of type)?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 15 2022 at 18:02):

fitzgen commented on issue #4307:

You could do that with very low probability (something like u.ratio(1, 1_000)) if you want, but I don't think it is super critical. Wouldn't work on this kind of thing ahead of working on the main bits.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 15 2022 at 20:03):

dicej commented on issue #4307:

Regarding the static API oracle: neither libFuzzer nor cargo-fuzz will be involved, correct? I'm envisioning adding code to the build.rs at the root of the repo which generates a .rs file under the tests directory containing a single test that exercises 1000 random test cases, each of which has been generated in build.rs using the test case generator with an arbitrary::Unstructured initialized with a random byte slice. Does that sound right?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 18 2022 at 13:28):

alexcrichton commented on issue #4307:

Yeah that sounds right to me. We'll want the seed of the byte slice to be deterministic somewhat in the sense that if something hits a bug on oss-fuzz we want to be able to reproduce it locally by overriding the seed. For now I think having an optional env var to specify the seed would be fine, we can always fiddle with it later too.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 19 2022 at 16:05):

dicej commented on issue #4307:

I'm making good progress on this, but getting hung up on the asymmetry between the ComponentType impls for str and WasmStr. The former only implements Lower, and the latter only implements Lift, so there's currently no available string type that implements both AFAICT. This would be an issue any time someone wanted to derive both Lift and Lower for a type containing a string, and it's especially annoying when generating a fuzz test like this one.

My first thought was to create my own type wrapper (e.g. struct MyString(Box<str>)) and then manually implement both Lift and Lower for it, but that's getting ugly fast, and I don't think we should expect wasmtime embedders to do that anyway. Another option is to have the test case generator generate two versions of each type: one for lifting and one for lowering -- just in case the type has a string somewhere in the hierarchy. I'd also need to generate a PartialEq impl for each such pair of types, which would be extremely tedious.

@alexcrichton I think life would be a lot easier if String (and Box<str>, etc.) implemented Lift, raising an error if the bytes aren't valid UTF-8. What do you think?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 19 2022 at 17:06):

alexcrichton commented on issue #4307:

That actually seems quite reasonable to me, the WasmStr type needed to exist as a base-level of support, but building String on top of Lower for str and Lift for WasmStr I think would work great!

view this post on Zulip Wasmtime GitHub notifications bot (Jul 20 2022 at 22:34):

dicej commented on issue #4307:

That actually seems quite reasonable to me, the WasmStr type needed to exist as a base-level of support, but building String on top of Lower for str and Lift for WasmStr I think would work great!

I tried to do this, but given that Lift::load only receives a Memory and a &[u8] (and no store), there doesn't seem to be a way to access the memory of the string body, so there's no way to construct an actual non-empty String. Am I missing something?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 20 2022 at 22:38):

dicej commented on issue #4307:

Regarding the above, would it be reasonable to add a &StoreOpaque parameter to Lift::load?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 21 2022 at 14:37):

alexcrichton commented on issue #4307:

I think Memory::as_slice can be used to get the view into the entire linear memory for indirect pointers?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 21 2022 at 14:47):

dicej commented on issue #4307:

I think Memory::as_slice can be used to get the view into the entire linear memory for indirect pointers?

Oh, good call. I missed that Memory had a StoreOpaque field. Thanks!

view this post on Zulip Wasmtime GitHub notifications bot (Jul 29 2022 at 16:00):

dicej commented on issue #4307:

@alexcrichton Things are coming together nicely. The fuzz tests have found a few bugs, which I've been debugging and fixing. I could use your help with dependency management, though.

The publish CI job isn't happy with the "component-test-util/component-model" line I added to Cargo.toml, and my cargo skills aren't sharp enough to know how to address it. In this case component-test-util is a dev depencency, but it's only used when the component-model feature is enabled. However, apparently dev dependencies can't be optional, so I'm not sure what to do. Any ideas?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 29 2022 at 16:04):

alexcrichton commented on issue #4307:

Nice! I'm also happy to help out with debugging if you'd like as well.

Otherwise though I think the best fix is to just unconditionally enable the component-model feature when testing. We already do this for other optional features like async and it'll avoid us having to deal with features-in-tests. In short there's no other easy answer for having an optional dev-dependency.

I'll try to work on a PR to unconditionally enable the component-model when testing.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 29 2022 at 16:18):

dicej commented on issue #4307:

Here's a fun one:

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: item nesting too deep
     --> <anon>:66:1005
      |
   66 |             (import "echo" (func $f (param (variant (case "C0" (variant (case "C0" (variant (case "C0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" string) (case "C1" unit))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))) (result unit)))
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ^', crates/fuzzing/src/oracles.rs:1088:6

view this post on Zulip Wasmtime GitHub notifications bot (Jul 29 2022 at 16:44):

alexcrichton commented on issue #4307:

Ah yeah for that we'll want to constrain the Arbitrary for the type hierarchy to have some maximum depth. For that we'll either need to throw out test cases earlier or otherwise change the Arbitrary impl to constrain how deep types can be.

I suppose another idea is to use the binary encoding instead of the text encoding of components since that specific error is coming from the wast parser.

... This makes me think that we may stack overflow with super deep type hierarchies now...

view this post on Zulip Wasmtime GitHub notifications bot (Aug 04 2022 at 18:49):

dicej commented on issue #4307:

This is done as of #4537, I believe.

view this post on Zulip Wasmtime GitHub notifications bot (Aug 04 2022 at 19:10):

alexcrichton commented on issue #4307:

Indeed, and thanks again @dicej!

view this post on Zulip Wasmtime GitHub notifications bot (Aug 04 2022 at 19:10):

alexcrichton closed issue #4307:

To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.

This fuzz target should:

This tests that we can round trip interface values of type T both as arguments
and returns to and from Wasm.

Here is a diagram showing what the fuzz target should do:

+-------------------+                   +------------------+
| Host              |                   | Wasm             |
|                   |   argument        |                  |
|     arbitrary ------------------------------>            |
|     value         |                   |      |           |
|                   |                   |      |           |
|                   |                   |      |           |
|         +-----------------+           |      |           |
|         | host function   |           |      |           |
|         | imported by     |           |      |           |
|         | Wasm            |  argument |      V           |
|         |            <-----------------------            |
|         |           |     |           |                  |
|         | assert    |     |           |                  |
|         | value is  |     |           |                  |
|         | equal     V     |  return   |                  |
|         |            ----------------------->            |
|         |                 |           |      |           |
|         +-----------------+           |      |           |
|                   |                   |      |           |
|                   |          return   |      V           |
|   assert    <--------------------------------            |
|   value is        |                   |                  |
|   equal           |                   |                  |
|                   |                   |                  |
+-------------------+                   +------------------+

Test Case Generator

The test case generator will be responsible for generating

Oracle

There are two versions of the oracle:

  1. a version that uses the dynamic interface types API to pass values to Wasm,
    and

  2. a version that uses the static interface types API to pass values to Wasm.

Dynamic API Oracle

The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent of wasmtime::Val).

Static API Oracle

The static API oracle will use the test case generator at build time to
generate, say, 1000 interface types T to fuzz, and then generate arbitrary
instances of T at runtime. This means that if T is a struct, for example, we
can define a Rust type like

#[derive(InterfaceType)]
struct MyRustType {
    foo: u32,
    bar: String,
}

whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance of MyRustType at runtime
in this scenario, but not new types T.

(Note that derive(InterfaceType) does not exist at the time of writing.)


Last updated: Dec 23 2024 at 12:05 UTC