Stream: git-cranelift

Topic: cranelift / Issue #1408 Tracking issue for merging the cr...

view this post on Zulip GitHub (Feb 25 2020 at 18:49):

alexcrichton opened Issue #1408:

Discussed at yesterday's meeting I wanted to open an issue tracking progress for merging the wasmtime/cranelift repositories. I think there are three major work items that need to happen:

Performing the actual merge

There's a few different ways to do the merge here, but I'd like to propose that the following steps are done.

I suspect the last part is going to be pretty significant here, so I'll be prototyping that before we actually do the merge since we want the "shut down window" to be as small as possible. I'll post here with the work when I get it set up. If folks have workflows though that go beyond cargo test and want to make sure any scripts and such keep working, let me know! I'll try to make sure they work in the new repository.

Transferring open issues

This will involve creating a script to migrate all open issues from the bytecodealliance/cranelift repository to bytecodealliance/wasmtime. I don't believe such a script exists, but it'll get gisted here before the move is done.

Making it easier to migrate open PRs

I'm on the hook for creating something to make it easy-ish to transfer open PRs to the wasmtime repository. My current thinking is the script would look like:

I'll run through this myself and confirm this works, posting the results here.

view this post on Zulip GitHub (Feb 25 2020 at 18:54):

iximeow commented on Issue #1408:

Has there been discussion of _why_ we want to merge the Cranelift and Wasmtime repos somewhere? I know it's been on the cranelift meeting agenda for the past few weeks but I never saw context for it there either.

view this post on Zulip GitHub (Feb 25 2020 at 18:57):

bjorn3 commented on Issue #1408:

view this post on Zulip GitHub (Feb 25 2020 at 18:58):

bjorn3 edited a comment on Issue #1408:

view this post on Zulip GitHub (Feb 25 2020 at 19:01):

bjorn3 edited a comment on Issue #1408:

view this post on Zulip GitHub (Feb 25 2020 at 20:03):

alexcrichton edited Issue #1408:

Discussed at yesterday's meeting I wanted to open an issue tracking progress for merging the wasmtime/cranelift repositories. I think there are three major work items that need to happen:

Performing the actual merge

There's a few different ways to do the merge here, but I'd like to propose that the following steps are done.

If folks have workflows though that go beyond cargo test and want to make sure any scripts and such keep working, let me know! I'll try to make sure they work in the new repository.

Transferring open issues

This will involve creating a script to migrate all open issues from the bytecodealliance/cranelift repository to bytecodealliance/wasmtime. I don't believe such a script exists, but it'll get gisted here before the move is done.

I think the general guidelines for this will be:

Making it easier to migrate open PRs

I'm on the hook for creating something to make it easy-ish to transfer open PRs to the wasmtime repository. My current thinking is the script would look like:

I'll run through this myself and confirm this works, posting the results here.

view this post on Zulip GitHub (Feb 25 2020 at 20:22):

tschneidereit commented on Issue #1408:

The merge was discussed a few months ago. Originally we had wanted to do this as part of the move to the Bytecode Alliance GitHub org, but then had to put it on the backburner because it became clear that the complications @alexcrichton mentions above would take longer to work through.

I'll give a longer explanation below, but to put one thing front-and-center, the merge is not in any way about tightly coupling Cranelift to Wasmtime. Speaking with my Mozilla hat on, that doesn't make sense, because SpiderMonkey is, if anything, the more important embedding. And neither does it make sense from a Bytecode Alliance perspective: as @bjorn3 points out, with Lucet we have another embedding that's in production and really important. And we're also big fans of rustc_codegen_cranelift, and certainly don't want that to not be supported.

The current setup causes us to have significant inefficiencies in our workflows, because we often have to do things in Cranelift and a runtime in parallel, and coordinate landing and CI runs, etc. We realized this will become much worse over time, because we need to be able to fully test features that have tight compiler/runtime interactions.

Right now, we have somewhat rudimentary support for testing these kinds of features in Cranelift, but with various upcoming features—such as Interface Types, Linking, and GC—we need something much more sophisticated. We came to the conclusion that by far the best way to do this is to have that runtime be Wasmtime, because otherwise we'd end up duplicating a lot of functionality. And we'd not really have something that's production-quality, meaning that the quality of the integration would always be worse than what it could be with Wasmtime.

Unfortunately, this is currently not possible: circular dependencies are hugely painful across separate repositories, and just doesn't seem viable. Hence the plan to merge the repositories.

We believe the downsides this brings are solidly outweighed by the advantage of improving the productivity of people working on both Cranelift and Wasmtime—and thus ultimately also benefit other embeddings of Cranelift.

We’ll of course continue doing individual crate releases on crates.io, and we’ll make sure to have good Cranelift documentation independent from Wasmtime. We’ll provide more info about good workflows (including things around notification handling) based on these merged repositories once we get closer to actually doing the merge.

view this post on Zulip GitHub (Feb 26 2020 at 01:10):

alexcrichton edited Issue #1408:

Discussed at yesterday's meeting I wanted to open an issue tracking progress for merging the wasmtime/cranelift repositories. I think there are three major work items that need to happen:

Performing the actual merge

There's a few different ways to do the merge here, but I'd like to propose that the following steps are done.

If folks have workflows though that go beyond cargo test and want to make sure any scripts and such keep working, let me know! I'll try to make sure they work in the new repository.

Transferring open issues

This will involve creating a script to migrate all open issues from the bytecodealliance/cranelift repository to bytecodealliance/wasmtime.

I think the general guidelines for this will be:

script: https://gist.github.com/alexcrichton/ef102cb315e548c92fd671c198da9aa2

Making it easier to migrate open PRs

I'm on the hook for creating something to make it easy-ish to transfer open PRs to the wasmtime repository. My current thinking is the script would look like:

I'll run through this myself and confirm this works, posting the results here.

view this post on Zulip GitHub (Feb 26 2020 at 01:11):

alexcrichton commented on Issue #1408:

Ok I've created a sample repository of what the final merge looks like -- https://github.com/alexcrichton/wasmtime/commits/cranelift-merge-afterwards -- which has merges of CI, fuzzers, documentation, etc. Feel free to play around in that repo and let me know if you find any issues!

I've also created a script -- https://gist.github.com/alexcrichton/ef102cb315e548c92fd671c198da9aa2 -- to transfer all issues from one repository to another, while preserving labels. Tomorrow I'll work on how to transfer PRs in an "easy" fashion.

view this post on Zulip GitHub (Feb 26 2020 at 16:19):

alexcrichton edited Issue #1408:

Discussed at yesterday's meeting I wanted to open an issue tracking progress for merging the wasmtime/cranelift repositories. I think there are three major work items that need to happen:

Performing the actual merge

There's a few different ways to do the merge here, but I'd like to propose that the following steps are done.

If folks have workflows though that go beyond cargo test and want to make sure any scripts and such keep working, let me know! I'll try to make sure they work in the new repository.

Transferring open issues

This will involve creating a script to migrate all open issues from the bytecodealliance/cranelift repository to bytecodealliance/wasmtime.

I think the general guidelines for this will be:

script: https://gist.github.com/alexcrichton/ef102cb315e548c92fd671c198da9aa2

Making it easier to migrate open PRs

I've created a script which should ideally make it easy to transfer PRs from one repository to another. The general gist of that script is something like this:

At that point you should have a local git commit you can push up to your wasmtime fork and send as a PR. We'll need to, however, close PRs as they're reopened on wasmtime.

Once this all actually happens I can drop a comment on each PR with precise instructions with an updated script that has all the variables sorted out.

view this post on Zulip GitHub (Feb 26 2020 at 16:21):

alexcrichton commented on Issue #1408:

Ok I've now also created a script -- https://gist.github.com/alexcrichton/8cb3f4ef7c25317ba6824ee31e3e53aa -- to rebase a PR on the wasmtime repository. The tl;dr; is that you use git rebase to move all the files for you, then you cherry-pick all the commits onto the wasmtime repo.

I've updated the description above, and that should make it so we're all set from a tooling perspective. I think now we just need a good time to pull the trigger!

view this post on Zulip GitHub (Feb 26 2020 at 22:37):

alexcrichton commented on Issue #1408:

I've created a zulip stream if folks want to chat about this as well, and also as a less "email the world on everything" way to coordinate.

view this post on Zulip GitHub (Feb 28 2020 at 16:36):

alexcrichton commented on Issue #1408:

Ok unless there's any objects that come up, I plan on tackling this in the afternoon today and performing the actual merge. If you've got any concerns it's best to bring them up in zulip and @-mention me.

One final question being considered is the final crate organization. The example above has crates/cranelift/$crate, but it sounds like we're now leaning towards cranelift/$crate or cranelift/crates/$crate

view this post on Zulip GitHub (Feb 28 2020 at 23:29):

alexcrichton transferred Issue #1408:

Discussed at yesterday's meeting I wanted to open an issue tracking progress for merging the wasmtime/cranelift repositories. I think there are three major work items that need to happen:

Performing the actual merge

There's a few different ways to do the merge here, but I'd like to propose that the following steps are done.

If folks have workflows though that go beyond cargo test and want to make sure any scripts and such keep working, let me know! I'll try to make sure they work in the new repository.

Transferring open issues

This will involve creating a script to migrate all open issues from the bytecodealliance/cranelift repository to bytecodealliance/wasmtime.

I think the general guidelines for this will be:

script: https://gist.github.com/alexcrichton/ef102cb315e548c92fd671c198da9aa2

Making it easier to migrate open PRs

I've created a script which should ideally make it easy to transfer PRs from one repository to another. The general gist of that script is something like this:

At that point you should have a local git commit you can push up to your wasmtime fork and send as a PR. We'll need to, however, close PRs as they're reopened on wasmtime.

Once this all actually happens I can drop a comment on each PR with precise instructions with an updated script that has all the variables sorted out.

Last updated: Dec 23 2024 at 13:07 UTC