fitzgen opened issue #4307:
To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.This fuzz target should:
- generate an arbitrary interface type
T
,generate a Wasm component that
- imports a host function
T -> T
, and- exports a function
T -> T
that passes its argument to the import function
and returns the result of the import function,generate some number of arbitrary values of type
T
,and for each value:
- call the Wasm's exported function,
when the host function is called, assert that the argument is equal to the
original value,return the argument from the host function,
when the Wasm function returns to the host, assert that the argument is
equal to the original value,assert that the host function was actually called.
This tests that we can round trip interface values of type
T
both as arguments
and returns to and from Wasm.Here is a diagram showing what the fuzz target should do:
+-------------------+ +------------------+ | Host | | Wasm | | | argument | | | arbitrary ------------------------------> | | value | | | | | | | | | | | | | | | +-----------------+ | | | | | host function | | | | | | imported by | | | | | | Wasm | argument | V | | | <----------------------- | | | | | | | | | assert | | | | | | value is | | | | | | equal V | return | | | | -----------------------> | | | | | | | | +-----------------+ | | | | | | | | | | | V | | assert <-------------------------------- | | value is | | | | equal | | | | | | | +-------------------+ +------------------+
Test Case Generator
The test case generator will be responsible for generating
- the arbitrary interface type
T
,the Wasm component that takes values of type
T
, passes them to the imported
host function, and returns the result of the host function,and instances of type
T
.Oracle
There are two versions of the oracle:
a version that uses the dynamic interface types API to pass values to Wasm,
anda version that uses the static interface types API to pass values to Wasm.
Dynamic API Oracle
The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent ofwasmtime::Val
).Static API Oracle
The static API oracle will use the test case generator at build time to
generate, say, 1000 interface typesT
to fuzz, and then generate arbitrary
instances ofT
at runtime. This means that ifT
is a struct, for example, we
can define a Rust type like#[derive(InterfaceType)] struct MyRustType { foo: u32, bar: String, }
whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance ofMyRustType
at runtime
in this scenario, but not new typesT
.(Note that
derive(InterfaceType)
does not exist at the time of writing.)
fitzgen commented on issue #4307:
We might actually want to have two instances of the Wasm component, so that values flow like
argument: host ---> wasm A ---> wasm B ---> host import return: host import ---> wasm B ---> wasm A ---> host
so that we are exercising component-to-component trampolines and all that.
dicej commented on issue #4307:
Thanks, for writing this up, @fitzgen. A few of questions for you:
- Given that neither the dynamic API for interface type values nor
derive(InterfaceType)
yet exist, I assume each of those things will need to be implemented before the corresponding oracle is created. Is that correct?- Is it worthwhile to do any work on this issue before one of the above features is implemented? I.e. is there any point in writing the test case generator without any oracles to use it?
- For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?
dicej edited a comment on issue #4307:
Thanks, for writing this up, @fitzgen. A few of questions for you:
- Given that neither the dynamic API for interface type values nor
derive(InterfaceType)
yet exist, I assume each of those things will need to be implemented before the corresponding oracle is created. Is that correct?- Is it worthwhile to do any work on this issue before one of the above features is implemented? I.e. is there any value in writing the test case generator prior to being able to write oracles that use it?
- For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?
fitzgen commented on issue #4307:
Given that neither the dynamic API for interface type values nor
derive(InterfaceType)
yet exist, I assume each of those things will need to be implemented before the corresponding oracle is created. Is that correct?Is it worthwhile to do any work on this issue before one of the above features is implemented? I.e. is there any value in writing the test case generator prior to being able to write oracles that use it?
Yeah, I think it makes sense to unblock this before working on it. We could probably technically implement some (most?) of the test case generator without the oracles, but I think it would be best if they were developed in together since I think the shape of the generator may be informed by the oracles.
For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?
I think a
build.rs
would be better, since it would generally just be easier to write, read, and debug.
fitzgen edited issue #4307:
To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.This fuzz target should:
- generate an arbitrary interface type
T
,generate a Wasm component that
- imports a host function
T -> T
, and- exports a function
T -> T
that passes its argument to the import function
and returns the result of the import function,generate some number of arbitrary values of type
T
,and for each value:
- call the Wasm's exported function,
when the host function is called, assert that the argument is equal to the
original value,return the argument from the host function,
when the Wasm function returns to the host, assert that the argument is
equal to the original value,assert that the host function was actually called.
This tests that we can round trip interface values of type
T
both as arguments
and returns to and from Wasm.Here is a diagram showing what the fuzz target should do:
+-------------------+ +------------------+ | Host | | Wasm | | | argument | | | arbitrary ------------------------------> | | value | | | | | | | | | | | | | | | +-----------------+ | | | | | host function | | | | | | imported by | | | | | | Wasm | argument | V | | | <----------------------- | | | | | | | | | assert | | | | | | value is | | | | | | equal V | return | | | | -----------------------> | | | | | | | | +-----------------+ | | | | | | | | | | return | V | | assert <-------------------------------- | | value is | | | | equal | | | | | | | +-------------------+ +------------------+
Test Case Generator
The test case generator will be responsible for generating
- the arbitrary interface type
T
,the Wasm component that takes values of type
T
, passes them to the imported
host function, and returns the result of the host function,and instances of type
T
.Oracle
There are two versions of the oracle:
a version that uses the dynamic interface types API to pass values to Wasm,
anda version that uses the static interface types API to pass values to Wasm.
Dynamic API Oracle
The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent ofwasmtime::Val
).Static API Oracle
The static API oracle will use the test case generator at build time to
generate, say, 1000 interface typesT
to fuzz, and then generate arbitrary
instances ofT
at runtime. This means that ifT
is a struct, for example, we
can define a Rust type like#[derive(InterfaceType)] struct MyRustType { foo: u32, bar: String, }
whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance ofMyRustType
at runtime
in this scenario, but not new typesT
.(Note that
derive(InterfaceType)
does not exist at the time of writing.)
dicej edited a comment on issue #4307:
Thanks, for writing this up, @fitzgen. A few questions for you:
- Given that neither the dynamic API for interface type values nor
derive(InterfaceType)
yet exist, I assume each of those things will need to be implemented before the corresponding oracle is created. Is that correct?- Is it worthwhile to do any work on this issue before one of the above features is implemented? I.e. is there any value in writing the test case generator prior to being able to write oracles that use it?
- For the static API oracle, were you thinking of creating (or extending) a build.rs that generates Rust code or writing a procedural macro?
dicej commented on issue #4307:
@alexcrichton Are you planning to create an issue for
derive(InterfaceType
? I'd be happy to create a placeholder, but I'm still fuzzy on the details, so I'll need help fleshing it out.Also, it sounds like we might also need an issue for the interface-type-equivalent of
wasmtime::Val
Nick mentioned above. I _think_ I understand what's involved there (e.g. copying values.rs and adapting it for interface types), but I could use guidance there, too.
alexcrichton commented on issue #4307:
I do indeed want to open an issue! Sorry I got caught up all day in a debugging adventure but that's over now so I'll go make an issue next.
alexcrichton labeled issue #4307:
To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.This fuzz target should:
- generate an arbitrary interface type
T
,generate a Wasm component that
- imports a host function
T -> T
, and- exports a function
T -> T
that passes its argument to the import function
and returns the result of the import function,generate some number of arbitrary values of type
T
,and for each value:
- call the Wasm's exported function,
when the host function is called, assert that the argument is equal to the
original value,return the argument from the host function,
when the Wasm function returns to the host, assert that the argument is
equal to the original value,assert that the host function was actually called.
This tests that we can round trip interface values of type
T
both as arguments
and returns to and from Wasm.Here is a diagram showing what the fuzz target should do:
+-------------------+ +------------------+ | Host | | Wasm | | | argument | | | arbitrary ------------------------------> | | value | | | | | | | | | | | | | | | +-----------------+ | | | | | host function | | | | | | imported by | | | | | | Wasm | argument | V | | | <----------------------- | | | | | | | | | assert | | | | | | value is | | | | | | equal V | return | | | | -----------------------> | | | | | | | | +-----------------+ | | | | | | | | | | return | V | | assert <-------------------------------- | | value is | | | | equal | | | | | | | +-------------------+ +------------------+
Test Case Generator
The test case generator will be responsible for generating
- the arbitrary interface type
T
,the Wasm component that takes values of type
T
, passes them to the imported
host function, and returns the result of the host function,and instances of type
T
.Oracle
There are two versions of the oracle:
a version that uses the dynamic interface types API to pass values to Wasm,
anda version that uses the static interface types API to pass values to Wasm.
Dynamic API Oracle
The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent ofwasmtime::Val
).Static API Oracle
The static API oracle will use the test case generator at build time to
generate, say, 1000 interface typesT
to fuzz, and then generate arbitrary
instances ofT
at runtime. This means that ifT
is a struct, for example, we
can define a Rust type like#[derive(InterfaceType)] struct MyRustType { foo: u32, bar: String, }
whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance ofMyRustType
at runtime
in this scenario, but not new typesT
.(Note that
derive(InterfaceType)
does not exist at the time of writing.)
jameysharp labeled issue #4307:
To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.This fuzz target should:
- generate an arbitrary interface type
T
,generate a Wasm component that
- imports a host function
T -> T
, and- exports a function
T -> T
that passes its argument to the import function
and returns the result of the import function,generate some number of arbitrary values of type
T
,and for each value:
- call the Wasm's exported function,
when the host function is called, assert that the argument is equal to the
original value,return the argument from the host function,
when the Wasm function returns to the host, assert that the argument is
equal to the original value,assert that the host function was actually called.
This tests that we can round trip interface values of type
T
both as arguments
and returns to and from Wasm.Here is a diagram showing what the fuzz target should do:
+-------------------+ +------------------+ | Host | | Wasm | | | argument | | | arbitrary ------------------------------> | | value | | | | | | | | | | | | | | | +-----------------+ | | | | | host function | | | | | | imported by | | | | | | Wasm | argument | V | | | <----------------------- | | | | | | | | | assert | | | | | | value is | | | | | | equal V | return | | | | -----------------------> | | | | | | | | +-----------------+ | | | | | | | | | | return | V | | assert <-------------------------------- | | value is | | | | equal | | | | | | | +-------------------+ +------------------+
Test Case Generator
The test case generator will be responsible for generating
- the arbitrary interface type
T
,the Wasm component that takes values of type
T
, passes them to the imported
host function, and returns the result of the host function,and instances of type
T
.Oracle
There are two versions of the oracle:
a version that uses the dynamic interface types API to pass values to Wasm,
anda version that uses the static interface types API to pass values to Wasm.
Dynamic API Oracle
The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent ofwasmtime::Val
).Static API Oracle
The static API oracle will use the test case generator at build time to
generate, say, 1000 interface typesT
to fuzz, and then generate arbitrary
instances ofT
at runtime. This means that ifT
is a struct, for example, we
can define a Rust type like#[derive(InterfaceType)] struct MyRustType { foo: u32, bar: String, }
whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance ofMyRustType
at runtime
in this scenario, but not new typesT
.(Note that
derive(InterfaceType)
does not exist at the time of writing.)
dicej commented on issue #4307:
FYI, I'm planning to start working on this tomorrow, assuming no significant additional work is needed on #4442.
dicej commented on issue #4307:
Looks like #4442 needs more work, so I won't be able to tackle this one quite yet.
dicej commented on issue #4307:
Quick question: Should the test case generator deliberately generate instances and components which do _not_ match the generated type?
For example, if the generated type is a record with three fields, should the instance generator (sometimes) generate values with too many, not enough, or mis-typed field values (or values which are instances of an entirely different sort of type), and should the component generator (sometimes) generate a component which expects a different record type (or a different sort of type)?
fitzgen commented on issue #4307:
You could do that with very low probability (something like
u.ratio(1, 1_000)
) if you want, but I don't think it is super critical. Wouldn't work on this kind of thing ahead of working on the main bits.
dicej commented on issue #4307:
Regarding the static API oracle: neither
libFuzzer
norcargo-fuzz
will be involved, correct? I'm envisioning adding code to the build.rs at the root of the repo which generates a .rs file under the tests directory containing a single test that exercises 1000 random test cases, each of which has been generated in build.rs using the test case generator with anarbitrary::Unstructured
initialized with a random byte slice. Does that sound right?
alexcrichton commented on issue #4307:
Yeah that sounds right to me. We'll want the seed of the byte slice to be deterministic somewhat in the sense that if something hits a bug on oss-fuzz we want to be able to reproduce it locally by overriding the seed. For now I think having an optional env var to specify the seed would be fine, we can always fiddle with it later too.
dicej commented on issue #4307:
I'm making good progress on this, but getting hung up on the asymmetry between the
ComponentType
impls forstr
andWasmStr
. The former only implementsLower
, and the latter only implementsLift
, so there's currently no available string type that implements both AFAICT. This would be an issue any time someone wanted to derive bothLift
andLower
for a type containing a string, and it's especially annoying when generating a fuzz test like this one.My first thought was to create my own type wrapper (e.g.
struct MyString(Box<str>)
) and then manually implement bothLift
andLower
for it, but that's getting ugly fast, and I don't think we should expectwasmtime
embedders to do that anyway. Another option is to have the test case generator generate two versions of each type: one for lifting and one for lowering -- just in case the type has a string somewhere in the hierarchy. I'd also need to generate aPartialEq
impl for each such pair of types, which would be extremely tedious.@alexcrichton I think life would be a lot easier if
String
(andBox<str>
, etc.) implementedLift
, raising an error if the bytes aren't valid UTF-8. What do you think?
alexcrichton commented on issue #4307:
That actually seems quite reasonable to me, the
WasmStr
type needed to exist as a base-level of support, but buildingString
on top ofLower for str
andLift for WasmStr
I think would work great!
dicej commented on issue #4307:
That actually seems quite reasonable to me, the
WasmStr
type needed to exist as a base-level of support, but buildingString
on top ofLower for str
andLift for WasmStr
I think would work great!I tried to do this, but given that
Lift::load
only receives aMemory
and a&[u8]
(and no store), there doesn't seem to be a way to access the memory of the string body, so there's no way to construct an actual non-emptyString
. Am I missing something?
dicej commented on issue #4307:
Regarding the above, would it be reasonable to add a
&StoreOpaque
parameter toLift::load
?
alexcrichton commented on issue #4307:
I think
Memory::as_slice
can be used to get the view into the entire linear memory for indirect pointers?
dicej commented on issue #4307:
I think
Memory::as_slice
can be used to get the view into the entire linear memory for indirect pointers?Oh, good call. I missed that
Memory
had aStoreOpaque
field. Thanks!
dicej commented on issue #4307:
@alexcrichton Things are coming together nicely. The fuzz tests have found a few bugs, which I've been debugging and fixing. I could use your help with dependency management, though.
The publish CI job isn't happy with the "component-test-util/component-model" line I added to Cargo.toml, and my cargo skills aren't sharp enough to know how to address it. In this case
component-test-util
is a dev depencency, but it's only used when thecomponent-model
feature is enabled. However, apparently dev dependencies can't beoptional
, so I'm not sure what to do. Any ideas?
alexcrichton commented on issue #4307:
Nice! I'm also happy to help out with debugging if you'd like as well.
Otherwise though I think the best fix is to just unconditionally enable the
component-model
feature when testing. We already do this for other optional features likeasync
and it'll avoid us having to deal with features-in-tests. In short there's no other easy answer for having an optional dev-dependency.I'll try to work on a PR to unconditionally enable the
component-model
when testing.
dicej commented on issue #4307:
Here's a fun one:
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: item nesting too deep --> <anon>:66:1005 | 66 | (import "echo" (func $f (param (variant (case "C0" (variant (case "C0" (variant (case "C0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (record (field "f0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" (variant (case "C0" string) (case "C1" unit))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))) (result unit))) | ^', crates/fuzzing/src/oracles.rs:1088:6
alexcrichton commented on issue #4307:
Ah yeah for that we'll want to constrain the
Arbitrary
for the type hierarchy to have some maximum depth. For that we'll either need to throw out test cases earlier or otherwise change theArbitrary
impl to constrain how deep types can be.I suppose another idea is to use the binary encoding instead of the text encoding of components since that specific error is coming from the
wast
parser.... This makes me think that we may stack overflow with super deep type hierarchies now...
dicej commented on issue #4307:
This is done as of #4537, I believe.
alexcrichton commented on issue #4307:
Indeed, and thanks again @dicej!
alexcrichton closed issue #4307:
To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.This fuzz target should:
- generate an arbitrary interface type
T
,generate a Wasm component that
- imports a host function
T -> T
, and- exports a function
T -> T
that passes its argument to the import function
and returns the result of the import function,generate some number of arbitrary values of type
T
,and for each value:
- call the Wasm's exported function,
when the host function is called, assert that the argument is equal to the
original value,return the argument from the host function,
when the Wasm function returns to the host, assert that the argument is
equal to the original value,assert that the host function was actually called.
This tests that we can round trip interface values of type
T
both as arguments
and returns to and from Wasm.Here is a diagram showing what the fuzz target should do:
+-------------------+ +------------------+ | Host | | Wasm | | | argument | | | arbitrary ------------------------------> | | value | | | | | | | | | | | | | | | +-----------------+ | | | | | host function | | | | | | imported by | | | | | | Wasm | argument | V | | | <----------------------- | | | | | | | | | assert | | | | | | value is | | | | | | equal V | return | | | | -----------------------> | | | | | | | | +-----------------+ | | | | | | | | | | return | V | | assert <-------------------------------- | | value is | | | | equal | | | | | | | +-------------------+ +------------------+
Test Case Generator
The test case generator will be responsible for generating
- the arbitrary interface type
T
,the Wasm component that takes values of type
T
, passes them to the imported
host function, and returns the result of the host function,and instances of type
T
.Oracle
There are two versions of the oracle:
a version that uses the dynamic interface types API to pass values to Wasm,
anda version that uses the static interface types API to pass values to Wasm.
Dynamic API Oracle
The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent ofwasmtime::Val
).Static API Oracle
The static API oracle will use the test case generator at build time to
generate, say, 1000 interface typesT
to fuzz, and then generate arbitrary
instances ofT
at runtime. This means that ifT
is a struct, for example, we
can define a Rust type like#[derive(InterfaceType)] struct MyRustType { foo: u32, bar: String, }
whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance ofMyRustType
at runtime
in this scenario, but not new typesT
.(Note that
derive(InterfaceType)
does not exist at the time of writing.)
Last updated: Nov 22 2024 at 17:03 UTC