Stream: cranelift

Topic: out of tree backend

Théo Degioanni (Oct 21 2022 at 15:35):

Hello! I've been curious about using Cranelift as a codegen backend for an exotic architecture (for fun). From my understanding of the backend, the list of supported architectures is hard-coded. Are out-of-tree architecture backends intended in Cranelift? The interface is so easy I feel like there is a great opportunity for easy-to-plug-in target architectures.
If not I will just fork Cranelift for my fun project, but I was wondering if there is a better solution to have it be more modular.

Chris Fallin (Oct 21 2022 at 16:01):

@Théo Degioanni -- that's an interesting question!

Right now the simple answer is "no, not supported", but if we were to find a way to make this work, I wouldn't be opposed to it, I think.

A few considerations off the top of my head:

• Backend registration: we'd want to replace the hardcoded backend lookup that you've noted with some sort of registry. Or maybe we don't need that, and cranelift_codegen::isa::lookup continues to stand as a way to get a backend for one of the architectures in cranelift-codegen, and we just expose a way to provide a Box<dyn MachBackend> directly when constructing a compiler context. The latter feels a little cleaner actually.
• Build system: we currently have a build.rs that hardcodes a list of ISLE files to build; we'd probably want some sort of helper that a build.rs in a crate containing another backend could invoke.
• Versioning: we are currently free to update internal interfaces and refactor as we see the need; I don't want to lose this freedom and ossify the compiler's internals in order to provide a stability guarantee to backends (the value tradeoff still seems clearly in the direction of "let the compiler evolve" rather than "hold it steady so many backends can arise"). So an out-of-tree backend would likely need to version-pin and might need sometimes significant work to upgrade to newer versions.

The last part seems like the most potentially questionable part. I'd want to set a norm that the cost of keeping up is solely on the out-of-tree project, and we don't consider this cost when deciding to make changes. (In our tiers hierarchy, this is a kind of "tier 4": not only is it expected not to interfere with work on features/platforms/evolution of higher tiers, but we don't even try to keep things compiling because we don't see them.) But if that's an acceptable situation, then yeah, this could work -- it basically amounts to opening up visibility a bit (throwing some pubs in) and giving an interface to pass in a backend trait impl.

Chris Fallin (Oct 21 2022 at 16:01):

Curious what others think too -- cc @fitzgen (he/him) @Trevor Elliott @Jamey Sharp @Andrew Brown

Andrew Brown (Oct 21 2022 at 16:03):

Yeah, if someone wants to build a more flexible backend registration, I think it would be a great addition

fitzgen (he/him) (Oct 21 2022 at 16:08):

I feel like this is unnecessary complexity that isn't super well motivated by any project stakeholders or core use cases. If this is just a for-fun project, I don't see the harm in OP forking. If this is a more serious, long-term-supported project, then I'd prefer to have the conversation about adding it as a proper first-class supported backend for cranelift.

Trevor Elliott (Oct 21 2022 at 16:09):

It might be a nice way to show backend support and maturity. Backends that we support would be in-tree while side-projects could be separate packages that wouldn't affect our own build system.

Chris Fallin (Oct 21 2022 at 16:13):

The main concern is ecosystem dynamics IMHO (the versioning bit above). If it becomes a defacto way to build plugin backends that many folks end up depending on, then we have a strictly worse world where we can't evolve because changes have to be coordinated across repos and teams. I think with enough care though (outlining exactly what expectations are, not accepting issues for "broken builds" when we update internal interfaces) it could work to foster a sense of openness and experimentation

Chris Fallin (Oct 21 2022 at 16:14):

I guess the question for us is: what really is the burden. If it's some pubs and an alternate constructor that takes a Box<dyn MachBackend>, that doesn't really qualify as "unnecessary complexity" IMHO. The social aspects are more important to consider I think

Chris Fallin (Oct 21 2022 at 16:17):

@Théo Degioanni a question for you: have you experimented with what actually is needed to open up the interface? a minimum diff might help us understand better exactly how invasive this would be

Théo Degioanni (Oct 21 2022 at 16:22):

Thank you for the consideration!
@Chris Fallin So far I have not experimented with anything, as I am evaluating what tech stack I should hook onto (my ultimate goal is to have a Rust or wasm compiler to my exotic architecture, so Cranelift is a good option).
My understanding from reading the code though is that (assuming I pin to a specific Cranelift commit), the interface itself is already somewhat open beyond the target triple/target selection interface? The TargetIsa trait seems like it could be implemented by anyone. I feel like I want to go with Cranelift either way, so I can try to add my target in tree and see where I would have struggled if I had just provided a Box<dyn TargetIsa> (assuming that makes sense).

Chris Fallin (Oct 21 2022 at 16:23):

That sounds good; an outline/summary of what we might do to eventually enable this would be a good outcome regardless. Good luck with the backend hacking and let us know if you have questions :-)

Théo Degioanni (Oct 21 2022 at 16:25):

Ideally from an user perspective I would have liked the lookup and other target-selection to be dynamic map-like lookups that would be pre-populated with the in-tree targets, and then could be expanded at runtime with targets created by the caller (so that I could for example invoke wasmtime with additionally-registered targets, or something like that).

Théo Degioanni (Oct 21 2022 at 16:27):

Thank you, though!

Last updated: Nov 22 2024 at 17:03 UTC