Stream: git-wasmtime

Topic: wasmtime / issue #4444 Enable cargo-vet


view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 14:49):

alexcrichton commented on issue #4444:

Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with cargo vet, so would you be up to adding a small README.md to the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).

Additionally, what's the general "next steps" here? Should we start performing reviews of dependencies and updating the exemptions list to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?

Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default wasi-crypto feature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything that wasi-crypto might need and we'll deal with this later" to make it clear to us which are the most important crates to review?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 17:35):

jameysharp commented on issue #4444:

If I'm reading the cargo-vet documentation correctly, it looks like we should be able to say something like this in supply-chain/config.toml:

[policy.wasi-crypto]
criteria = []

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 21:24):

bholley commented on issue #4444:

Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with cargo vet, so would you be up to adding a small README.md to the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).

Sure, happy to do that.

In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.

Additionally, what's the general "next steps" here?

As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.

Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.

Should we start performing reviews of dependencies and updating the exemptions list to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?

There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of cargo vet suggest.

Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default wasi-crypto feature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything that wasi-crypto might need and we'll deal with this later" to make it clear to us which are the most important crates to review?

As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just [safe-to-run](https://mozilla.github.io/cargo-vet/built-in-criteria.html#safe-to-run). This is the default for dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-level cargo test to install malware on a developer's workstation.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 21:26):

bholley edited a comment on issue #4444:

Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with cargo vet, so would you be up to adding a small README.md to the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).

Sure, happy to do that.

In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.

Additionally, what's the general "next steps" here?

As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.

Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.

Should we start performing reviews of dependencies and updating the exemptions list to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?

There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of cargo vet suggest.

Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default wasi-crypto feature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything that wasi-crypto might need and we'll deal with this later" to make it clear to us which are the most important crates to review?

As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default for dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-level cargo test to install malware on a developer's workstation.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 21:26):

bholley edited a comment on issue #4444:

Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with cargo vet, so would you be up to adding a small README.md to the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).

Sure, happy to do that.

In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.

Additionally, what's the general "next steps" here?

As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.

Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.

Should we start performing reviews of dependencies and updating the exemptions list to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?

There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of cargo vet suggest.

Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default wasi-crypto feature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything that wasi-crypto might need and we'll deal with this later" to make it clear to us which are the most important crates to review?

As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want by requiring []. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default for dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-level cargo test to install malware on a developer's workstation.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 21:27):

bholley edited a comment on issue #4444:

Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with cargo vet, so would you be up to adding a small README.md to the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).

Sure, happy to do that.

In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.

Additionally, what's the general "next steps" here?

As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.

Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.

Should we start performing reviews of dependencies and updating the exemptions list to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?

There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of cargo vet suggest.

Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default wasi-crypto feature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything that wasi-crypto might need and we'll deal with this later" to make it clear to us which are the most important crates to review?

As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want by requiring the empty list. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default for dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-level cargo test to install malware on a developer's workstation.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 14 2022 at 21:27):

bholley edited a comment on issue #4444:

Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with cargo vet, so would you be up to adding a small README.md to the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).

Sure, happy to do that.

In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.

Additionally, what's the general "next steps" here?

As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.

Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.

Should we start performing reviews of dependencies and updating the exemptions list to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?

There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of cargo vet suggest.

Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default wasi-crypto feature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything that wasi-crypto might need and we'll deal with this later" to make it clear to us which are the most important crates to review?

As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want by setting the required criteria to the empty list. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default for dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-level cargo test to install malware on a developer's workstation.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 15 2022 at 18:52):

alexcrichton commented on issue #4444:

Ok that all sounds great! With a small README I'm happy to merge this and get the ball rolling. I'm looking forward to exploring this!

Currently I get:

$ cargo vet suggest
...
estimated audit backlog: 9591434 lines

so oof we have quite the backlog!

view this post on Zulip Wasmtime GitHub notifications bot (Jul 15 2022 at 21:08):

bholley commented on issue #4444:

Ok that all sounds great! With a small README I'm happy to merge this and get the ball rolling.

Done!

Currently I get:

$ cargo vet suggest ... estimated audit backlog: 9591434 lines

so oof we have quite the backlog!

Note that it's an uneven distribution. About a third of that is v8, for example, which I don't think anyone has any intention of auditing. cargo vet suggest shows the smallest audits first, since it's generally higher ROI to knock out audits for 20 small crates than one big one.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 18 2022 at 03:36):

cfallin commented on issue #4444:

At the risk of somewhat complicating things (sorry! getting through backlog now and just catching this), a suggestion: would you (@bholley) be willing to write a Bytecode Alliance RFC for this?

I personally fully support the goals of cargo-vet and think this is a great thing; but I worry that introducing it like this all at once in an "enforcing mode" (i.e. with a CI job) without much in the way of project-specific guidance or policy or documentation may be a bit confusing for some contributors. I personally wouldn't know what to do if I were to, say, need to bump the version of a utility crate (can that invalidate an audit?) or add a dependency on a new one. Do I just add an exemption for now? Is that our blanket policy until we have something better / more specific, like an organized audit effort? Or do we do something else? Are we implicitly shifting now to a policy where we prefer not to add new dependencies at all? Are we eventually actually going to audit various upstream crates?

I may not be privy to various other conversations that have brought this effort here, and context they bring, but then if so, I guess I'm representative of other contributors also not in the loop. Some basic discussion on these points would be great, I think, to help educate and get everyone onboard with basic policies and procedures.

So, a basic RFC that outlines what our (BA's) use of cargo-vet would look like, with basic workflows, policies and such, who can approve new deps (or other approval-gated actions I guess), how we might eventually actually audit code, etc., would I think be great to see before we turn this on. We've had RFCs for similar other refinements and updates to our policies, e.g. for example Wasmtime 1.0 scope/procedures, or a security-incident response runbook, and this seems at least as consequential as those. What do you think?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 18 2022 at 03:38):

cfallin edited a comment on issue #4444:

At the risk of somewhat complicating things (sorry! getting through backlog now and just catching this), a suggestion: would you (@bholley) be willing to write a Bytecode Alliance RFC for this?

I personally fully support the goals of cargo-vet and think this is a great thing; but I worry that introducing it like this all at once in an "enforcing mode" (i.e. with a CI job) without much in the way of project-specific guidance or policy or documentation may be a bit confusing for some contributors. I personally wouldn't know what to do if I were to, say, need to bump the version of a utility crate (can that invalidate an audit?) or add a dependency on a new one. Do I just add an exemption for now? Is that our blanket policy until we have something better / more specific, like an organized audit effort? Or do we do something else? Are we implicitly shifting now to a policy where we prefer not to add new dependencies at all? Are we eventually actually going to audit various upstream crates?

I may not be privy to various other conversations that have brought this effort here, and context they bring, but then if so, I guess I'm representative of other contributors also not in the loop. Some basic discussion on these points would be great, I think, to help educate and get everyone onboard with basic policies and procedures.

So, a basic RFC that outlines what our (BA's) use of cargo-vet would look like, with basic workflows, policies and such, who can approve new deps (or other approval-gated actions I guess), how we might eventually actually audit code, etc., would I think be great to see before we turn this on. We've had RFCs for similar other refinements and updates to our policies, e.g. Wasmtime 1.0 scope/procedures, or a security-incident response runbook, and this seems at least as consequential as those. What do you think?

view this post on Zulip Wasmtime GitHub notifications bot (Jul 18 2022 at 20:32):

jameysharp commented on issue #4444:

I think most procedural details are covered well enough by https://mozilla.github.io/cargo-vet/performing-audits.html. What we'd need is mostly policy. I'm only coming up with two such questions, which I hope is a small enough scope that an RFC wouldn't be too difficult. Specifically:

@bholley, in thinking about those questions I'm now curious about other things:

view this post on Zulip Wasmtime GitHub notifications bot (Jul 19 2022 at 22:19):

alexcrichton commented on issue #4444:

@cfallin, @jameysharp, @bholley and I all had a great chat today about cargo vet and its usage in Wasmtime. The current thinking is that we will bring up the usage of cargo vet in CI at the upcoming Wasmtime and Cranelift meetings. Assuming there's no objections and everyone's on board we'll go ahead and merge this PR.

I think we'll also grow some documentation in the contributing section of the Wasmtime book for cargo vet around how we manage the audits.toml and such, but I think that's probably best added to after experience over time.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 20 2022 at 14:31):

alexcrichton commented on issue #4444:

I opened https://github.com/mozilla/cargo-vet/issues/271 and https://github.com/mozilla/cargo-vet/issues/272 from our discussion yesterday if others have other thoughts they'd like to contribute as well.

view this post on Zulip Wasmtime GitHub notifications bot (Jul 25 2022 at 16:29):

alexcrichton commented on issue #4444:

Ok we've discussed this at both the Cranelift and Wasmtime meetings there were no objections to landing this, so @bholley if you wouldn't mind rebasing this I think it should be good to go!

view this post on Zulip Wasmtime GitHub notifications bot (Jul 25 2022 at 19:31):

bholley commented on issue #4444:

@alexcrichton done!

view this post on Zulip Wasmtime GitHub notifications bot (Jul 25 2022 at 19:31):

bholley commented on issue #4444:

@alexcrichton done!

view this post on Zulip Wasmtime GitHub notifications bot (Jul 25 2022 at 19:31):

bholley deleted a comment on issue #4444:

@alexcrichton done!


Last updated: Dec 23 2024 at 12:05 UTC