Alex Crichton (Dec 13 2024 at 17:22): Alex Crichton (Dec 13 2024 at 17:22):I'm at a point where I need to at least vent a bit of frustration with cargo vet. As time goes on I feel like we know less and less of how cargo vet works and it becomes more and more of a black box that everyone just punts questions to someone else for. I also feel like if I can't figure out how to get something working with cargo vet then it just doesn't get done and we have to work around it one way or another, which isn't a great feeling for me as I'm generally just as much in the dark about others when it comes to cargo vet.
In the immediate term I have no idea why this PR is failing for cargo vet. If I run the CI commands locally I can't get the same failure. The PR changes supply-chain/*.toml files which seems highly likely to be relevant but if I revert the changes locally and run cargo vet then it changes the files again. I can't figure out what the difference is.
Additionally in that PR I'm adding a new crate which I'm now dreading because I know it's going to cause issues in the future. We have a common issue where cargo vet is going to pass until that crate is published. Once it's published it's going to break our CI on main and there's nothing we can do about it. We can't add annotations now because it's not published, and it's not published because it has to land in-tree first.
Overall I'm personally relatively frustrated with cargo vet in that it's a really great property to have for the project and I don't want to give it up but I feel like it's always suffered from key usability issues that haven't gotten resolved. Forcibly breaking our CI every time we add a new crate is quite bad IMO and we've had a relatively long history of being unable to reproduce CI issues locally. I suspect I'm missing something here but everything is so opaque I have no idea where to start to figure out what the difference is.
Alex Crichton (Dec 13 2024 at 17:24):I'm going to try to hack around the failure in my PR, but it means I'm about to check in a state to the repo where if anyone runs cargo vet locally then it just breaks and can't land the PR (the same state that PR is in). Doesn't feel great
Chris Fallin (Dec 13 2024 at 17:39):It might be worth re-evaluating the assumptions we made at the time we adopted the tool, too: the idea was that there would be a shared effort in vetting and that the burden would be relatively minimal. Could we take an objective look at that -- has it happened, how many vets have we had to do, what's the opportunity cost (which features, PRs, ... have we declined because they would bring in more vetting overhead), etc. Good topic for next Wasmtime meeting?
fitzgen (he/him) (Dec 13 2024 at 17:43):fwiw, the last few dep updates I've done haven't run into any issues with cargo vet but you also do that kind of thing way more than I do
Alex Crichton (Dec 13 2024 at 17:43):I don't personally want to champion anything to say we should remove the tool, but I also at the same time don't think we're in a great spot. I do think that factually we haven't really been doing many vets relative to the amount of time we've been using cargo vet
fitzgen (he/him) (Dec 13 2024 at 17:44):it is probably worth filing an issue upstream and seeing what they have to say
Alex Crichton (Dec 13 2024 at 17:44):Personally I find it difficult to talk about issues with cargo vet because everything is so unclear, the (reasonable) knee-jerk reaction is "well how can we fix that problem" and I can't even articulate what the problem is really
Alex Crichton (Dec 13 2024 at 17:44):I filed an issue awhile ago for the major issue we have (a new great guarantees a CI break in the future), lemme find it
fitzgen (he/him) (Dec 13 2024 at 17:44):I do think it is a really important part of our overall security story, so I'd much prefer working with upstream to resolve issues than giving it up
Alex Crichton (Dec 13 2024 at 17:45):https://github.com/mozilla/cargo-vet/issues/604
Alex Crichton (Dec 13 2024 at 17:45):my impression is that development on cargo-vet has slowed down a lot
fitzgen (he/him) (Dec 13 2024 at 17:47):yeah :-/
Chris Fallin (Jun 15 2026 at 16:44):Re-enlivening this old thread for some discussion (cc @Alex Crichton @fitzgen (he/him)): In the verification folks' mega-merge-to-main PR (here), I innocently set out to do a little bit of cargo vet'ing before lunch, just to find
which needless to say is a cost way too high to be practical (IMHO). Most of the big deltas are in crates we already use, as far as I can tell. I've asked to see if we can minimize the diff (avoid incidental upgrades) but even if those are avoidable now, it is still latent "unresolved upgrade" diff that we'll have to pay down at some point if we stay up-to-date in the steady state.
I learned about our >10k-download-per-day rule the other day when discussing this but even with that, this is still a heavy lift as I have to go check download counts and do manual actions for 144 crates.
Against that, vetting is more important than ever in the age of "supply-chain attacks are actually real and insidious", but I just thought I would raise the sometimes-seemingly-impossible cost we've signed up for and see if folks have thoughts...
fitzgen (he/him) (Jun 15 2026 at 16:46):oof yikes :-/
Is there a reason that there are so many crate versions bumped? Seems possible (likely?) that some stuff got accidentally bumped unnecessarily
Chris Fallin (Jun 15 2026 at 16:46):Yeah, I commented asking for us to regen Cargo.lock on suspicion that there was a cargo upgrade at some point on the branch
Chris Fallin (Jun 15 2026 at 16:47):but even then, this is now a gap between our vet frontier and upstreams' main, so we'll have to pay it at some point (unless we freeze deps forever)
fitzgen (he/him) (Jun 15 2026 at 16:48):agreed it is not a permanent solution
Pat Hickey (Jun 15 2026 at 16:52):part of the long term solution is allegedly that the community of other projects whose vets we trust will take care of some of that audit burden, particularly on the most used crates
fitzgen (he/him) (Jun 15 2026 at 16:52):Pat Hickey said:
part of the long term solution is allegedly that the community of other projects whose vets we trust will take care of some of that audit burden, particularly on the most used crates
which does happen, fwiw. I've updated deps a couple times and had the vet already done by mozilla or whoever
Alex Crichton (Jun 15 2026 at 16:59):On one hand this is how you backdoor wasmtime, you update everything where something in the soup is backdoor'd, but on the other hand I agree that there's a lot of friction here to updating deps
Alex Crichton (Jun 15 2026 at 16:59):what I've settled on is to every-so-often make sure we're on the latest semver-major track of all deps
Alex Crichton (Jun 15 2026 at 17:00):but I otherwise basically ignore point releases until there's some other forcing function
Alex Crichton (Jun 15 2026 at 17:00):but I suspect you're right in this case that there's a cargo update somewhere in the history by accident which can pretty easily reduce this vet burden
Alex Crichton (Jun 15 2026 at 17:01):we might also be able to exclude ISLE-specific deps if something like Z3 is coming in and it's a million lines
Alex Crichton (Jun 15 2026 at 17:01):we already exclude vets on fuzzing-specific deps IIRC
Chris Fallin (Jun 15 2026 at 17:03):Yeah, strong +1 to the aversion to update-everything-at-once; if nothing else, my reviewer fatigue would not be a good defense against an actual issue. So on principle I am not going to take on such a large vet at once in any case; hopefully we can make the new functionality compatible with that
Chris Fallin (Jun 15 2026 at 17:04):I agree it's important we do this, to be clear, and am not arguing against vetting ("vetting is more important than ever"); just... trying to square the labor asked of us with all of that
Alex Crichton (Jun 15 2026 at 17:04):yeah and I don't think anyone should expect to all-of-a-sudden vet 100kloc
Alex Crichton (Jun 15 2026 at 17:05):if we genuinely take a massive new runtime dep that's something we should probably discuss in a meeting or something
Last updated: Jul 29 2026 at 05:03 UTC