Wasmtime GitHub notifications bot (Jul 14 2022 at 14:49): Wasmtime GitHub notifications bot (Jul 14 2022 at 14:49):alexcrichton commented on issue #4444:
Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with
cargo vet, so would you be up to adding a smallREADME.mdto the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).Additionally, what's the general "next steps" here? Should we start performing reviews of dependencies and updating the
exemptionslist to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default
wasi-cryptofeature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything thatwasi-cryptomight need and we'll deal with this later" to make it clear to us which are the most important crates to review?
Wasmtime GitHub notifications bot (Jul 14 2022 at 17:35):jameysharp commented on issue #4444:
If I'm reading the cargo-vet documentation correctly, it looks like we should be able to say something like this in
supply-chain/config.toml:[policy.wasi-crypto] criteria = []
Wasmtime GitHub notifications bot (Jul 14 2022 at 21:24):bholley commented on issue #4444:
Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with
cargo vet, so would you be up to adding a smallREADME.mdto the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).Sure, happy to do that.
In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.
Additionally, what's the general "next steps" here?
As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.
Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.
Should we start performing reviews of dependencies and updating the
exemptionslist to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of
cargo vet suggest.Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default
wasi-cryptofeature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything thatwasi-cryptomight need and we'll deal with this later" to make it clear to us which are the most important crates to review?As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just
[safe-to-run](https://mozilla.github.io/cargo-vet/built-in-criteria.html#safe-to-run). This is the default fordev-dependencies, for example, since they're not shipped to users, but you also don't want a top-levelcargo testto install malware on a developer's workstation.
Wasmtime GitHub notifications bot (Jul 14 2022 at 21:26):bholley edited a comment on issue #4444:
Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with
cargo vet, so would you be up to adding a smallREADME.mdto the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).Sure, happy to do that.
In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.
Additionally, what's the general "next steps" here?
As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.
Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.
Should we start performing reviews of dependencies and updating the
exemptionslist to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of
cargo vet suggest.Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default
wasi-cryptofeature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything thatwasi-cryptomight need and we'll deal with this later" to make it clear to us which are the most important crates to review?As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default for
dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-levelcargo testto install malware on a developer's workstation.
Wasmtime GitHub notifications bot (Jul 14 2022 at 21:26):bholley edited a comment on issue #4444:
Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with
cargo vet, so would you be up to adding a smallREADME.mdto the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).Sure, happy to do that.
In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.
Additionally, what's the general "next steps" here?
As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.
Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.
Should we start performing reviews of dependencies and updating the
exemptionslist to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of
cargo vet suggest.Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default
wasi-cryptofeature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything thatwasi-cryptomight need and we'll deal with this later" to make it clear to us which are the most important crates to review?As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want by requiring
[]. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default fordev-dependencies, for example, since they're not shipped to users, but you also don't want a top-levelcargo testto install malware on a developer's workstation.
Wasmtime GitHub notifications bot (Jul 14 2022 at 21:27):bholley edited a comment on issue #4444:
Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with
cargo vet, so would you be up to adding a smallREADME.mdto the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).Sure, happy to do that.
In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.
Additionally, what's the general "next steps" here?
As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.
Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.
Should we start performing reviews of dependencies and updating the
exemptionslist to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of
cargo vet suggest.Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default
wasi-cryptofeature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything thatwasi-cryptomight need and we'll deal with this later" to make it clear to us which are the most important crates to review?As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want by requiring the empty list. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default for
dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-levelcargo testto install malware on a developer's workstation.
Wasmtime GitHub notifications bot (Jul 14 2022 at 21:27):bholley edited a comment on issue #4444:
Thanks for this! I suspect many others on Wasmtime, like me, are pretty unfamiliar with
cargo vet, so would you be up to adding a smallREADME.mdto the supply-chain directory describing the tool a little? (or otherwise mostly just leaving a link to the preexisting book).Sure, happy to do that.
In case it's helpful context, I'll also link here to the announcement we made to Firefox engineers when we deployed this on mozilla-central.
Additionally, what's the general "next steps" here?
As a very first step, the wasmtime team should also determine a set of people who can approve audit submissions, similar to what Mozilla did here. I think this should be enforceable on github with a CODEOWNERS file.
Once the group has been established, I might recommend we set up a call with the counterparts at Mozilla so that we can ensure we're aligned enough on expectations that we're comfortable sharing audits bidirectionally.
Should we start performing reviews of dependencies and updating the
exemptionslist to something that's not exempted? Should we wait for imports-from-other-projects to work before we start reviewing other things? For now if we add a dependency should we say it's ok to just add an exemption like has been done already?There are various ways to approach this. The approach we've adopted at Mozilla, which I'd certainly recommend here, is to generally aim for "no new unaudited code" — so if somebody adds a new dependency, they have to audit it (absent some compelling extenuating circumstance). And if they update an existing dependency, they have to either audit the delta (even if the base version of the delta is an exemption entry) or audit the whole package and remove the corresponding exemption entry. In parallel, we're aiming to slowly burn down our exemptions backlog with the help of
cargo vet suggest.Currently some parts of our dependency tree are quite large and something we don't audit much. For example the off-by-default
wasi-cryptofeature pulls in a pretty large family of crates which are only used for that one feature and isn't something we necessarily want to sink too much time into auditing and reviewing at this point since it's an experimental feature anyway. Is there a way to have a separate configuration file with "here's everything thatwasi-cryptomight need and we'll deal with this later" to make it clear to us which are the most important crates to review?As @jameysharp noted, cargo-vet allows you to configure different policies for difference subtrees in your build graph. So you can entirely exempt the wasi-crypto stuff if you want by setting the required criteria to the empty list. Alternatively, for stuff that's not shipped but might reasonably be executed by a developer, you might want to require just safe-to-run. This is the default for
dev-dependencies, for example, since they're not shipped to users, but you also don't want a top-levelcargo testto install malware on a developer's workstation.
Wasmtime GitHub notifications bot (Jul 15 2022 at 18:52):alexcrichton commented on issue #4444:
Ok that all sounds great! With a small README I'm happy to merge this and get the ball rolling. I'm looking forward to exploring this!
Currently I get:
$ cargo vet suggest ... estimated audit backlog: 9591434 linesso oof we have quite the backlog!
Wasmtime GitHub notifications bot (Jul 15 2022 at 21:08):bholley commented on issue #4444:
Ok that all sounds great! With a small README I'm happy to merge this and get the ball rolling.
Done!
Currently I get:
$ cargo vet suggest ... estimated audit backlog: 9591434 linesso oof we have quite the backlog!
Note that it's an uneven distribution. About a third of that is v8, for example, which I don't think anyone has any intention of auditing.
cargo vet suggestshows the smallest audits first, since it's generally higher ROI to knock out audits for 20 small crates than one big one.
Wasmtime GitHub notifications bot (Jul 18 2022 at 03:36):cfallin commented on issue #4444:
At the risk of somewhat complicating things (sorry! getting through backlog now and just catching this), a suggestion: would you (@bholley) be willing to write a Bytecode Alliance RFC for this?
I personally fully support the goals of
cargo-vetand think this is a great thing; but I worry that introducing it like this all at once in an "enforcing mode" (i.e. with a CI job) without much in the way of project-specific guidance or policy or documentation may be a bit confusing for some contributors. I personally wouldn't know what to do if I were to, say, need to bump the version of a utility crate (can that invalidate an audit?) or add a dependency on a new one. Do I just add an exemption for now? Is that our blanket policy until we have something better / more specific, like an organized audit effort? Or do we do something else? Are we implicitly shifting now to a policy where we prefer not to add new dependencies at all? Are we eventually actually going to audit various upstream crates?I may not be privy to various other conversations that have brought this effort here, and context they bring, but then if so, I guess I'm representative of other contributors also not in the loop. Some basic discussion on these points would be great, I think, to help educate and get everyone onboard with basic policies and procedures.
So, a basic RFC that outlines what our (BA's) use of cargo-vet would look like, with basic workflows, policies and such, who can approve new deps (or other approval-gated actions I guess), how we might eventually actually audit code, etc., would I think be great to see before we turn this on. We've had RFCs for similar other refinements and updates to our policies, e.g. for example Wasmtime 1.0 scope/procedures, or a security-incident response runbook, and this seems at least as consequential as those. What do you think?
Wasmtime GitHub notifications bot (Jul 18 2022 at 03:38):cfallin edited a comment on issue #4444:
At the risk of somewhat complicating things (sorry! getting through backlog now and just catching this), a suggestion: would you (@bholley) be willing to write a Bytecode Alliance RFC for this?
I personally fully support the goals of
cargo-vetand think this is a great thing; but I worry that introducing it like this all at once in an "enforcing mode" (i.e. with a CI job) without much in the way of project-specific guidance or policy or documentation may be a bit confusing for some contributors. I personally wouldn't know what to do if I were to, say, need to bump the version of a utility crate (can that invalidate an audit?) or add a dependency on a new one. Do I just add an exemption for now? Is that our blanket policy until we have something better / more specific, like an organized audit effort? Or do we do something else? Are we implicitly shifting now to a policy where we prefer not to add new dependencies at all? Are we eventually actually going to audit various upstream crates?I may not be privy to various other conversations that have brought this effort here, and context they bring, but then if so, I guess I'm representative of other contributors also not in the loop. Some basic discussion on these points would be great, I think, to help educate and get everyone onboard with basic policies and procedures.
So, a basic RFC that outlines what our (BA's) use of cargo-vet would look like, with basic workflows, policies and such, who can approve new deps (or other approval-gated actions I guess), how we might eventually actually audit code, etc., would I think be great to see before we turn this on. We've had RFCs for similar other refinements and updates to our policies, e.g. Wasmtime 1.0 scope/procedures, or a security-incident response runbook, and this seems at least as consequential as those. What do you think?
Wasmtime GitHub notifications bot (Jul 18 2022 at 20:32):jameysharp commented on issue #4444:
I think most procedural details are covered well enough by https://mozilla.github.io/cargo-vet/performing-audits.html. What we'd need is mostly policy. I'm only coming up with two such questions, which I hope is a small enough scope that an RFC wouldn't be too difficult. Specifically:
- Do we require contributors who change our dependencies to audit the changes themselves, and if so, do we trust every contributor to audit well enough? Or do we reject PRs modifying anything in
supply-chain/unless the audit updates are in a (signed?) commit by a core contributor?- How do we decide which other projects we trust enough to import their audits?
@bholley, in thinking about those questions I'm now curious about other things:
What stops someone from adding things to
imports.lockthat didn't actually come from imports? Maybe there could be a variant ofcargo vet --lockedthat checks that everything in the lock file corresponds to something from the current version of some import, so CI can catch this case. Better yet if it can check only those entries added in the current PR.When
cargo vet suggestproposes adding an import, it'd be nice if it could offer a specific command to run. (e.g.cargo vet import firefox?)Can I easily figure out what diff/version somebody else was reviewing when they recorded a certification?
Wasmtime GitHub notifications bot (Jul 19 2022 at 22:19):alexcrichton commented on issue #4444:
@cfallin, @jameysharp, @bholley and I all had a great chat today about cargo vet and its usage in Wasmtime. The current thinking is that we will bring up the usage of
cargo vetin CI at the upcoming Wasmtime and Cranelift meetings. Assuming there's no objections and everyone's on board we'll go ahead and merge this PR.I think we'll also grow some documentation in the contributing section of the Wasmtime book for
cargo vetaround how we manage theaudits.tomland such, but I think that's probably best added to after experience over time.
Wasmtime GitHub notifications bot (Jul 20 2022 at 14:31):alexcrichton commented on issue #4444:
I opened https://github.com/mozilla/cargo-vet/issues/271 and https://github.com/mozilla/cargo-vet/issues/272 from our discussion yesterday if others have other thoughts they'd like to contribute as well.
Wasmtime GitHub notifications bot (Jul 25 2022 at 16:29):alexcrichton commented on issue #4444:
Ok we've discussed this at both the Cranelift and Wasmtime meetings there were no objections to landing this, so @bholley if you wouldn't mind rebasing this I think it should be good to go!
Wasmtime GitHub notifications bot (Jul 25 2022 at 19:31):bholley commented on issue #4444:
@alexcrichton done!
Wasmtime GitHub notifications bot (Jul 25 2022 at 19:31):bholley commented on issue #4444:
@alexcrichton done!
Wasmtime GitHub notifications bot (Jul 25 2022 at 19:31):bholley deleted a comment on issue #4444:
@alexcrichton done!
Last updated: Nov 22 2024 at 17:03 UTC