Stream: warg

Topic: Federation dependency resolution


view this post on Zulip Lann Martin (Oct 04 2023 at 20:03):

view this post on Zulip Lann Martin (Oct 04 2023 at 20:05):

It seems like the answer has to include "Fermyon" at least as some kind of opt-in choice (if not the default or only option) in order to capture some of the use-cases we are going for.

view this post on Zulip Lann Martin (Oct 04 2023 at 20:14):

but you also would clearly want an option for Fermyon to mirror BA's wasi:core as ba:wasi:core and somehow signal to the client that it should use that to resolve wasi:http@1.0.0's dep on wasi:core (as we discussed in the meeting)

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:18):

Perhaps, the default behavior would be... ba:wasi:http originates from Registry "BA" and carries the context of that Registry which would resolve imports in the context of the source Registry. But if a package fermyon:http (published to "Fermyon" registry) imports wasi:core it would be resolve in the context of the "Fermyon" registry. How does that feel?

view this post on Zulip Lann Martin (Oct 04 2023 at 20:20):

Thinking about how that would be implemented, the implication is that a registry can instruct a client to remap any unlocked dependency to any other package, right?

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:23):

I was thinking the re-mapping would only be to make it possible for name conflicts to be resolved for packages published to its own registry. Not re-mapping other registries' packages imports to different ones and creating different binaries.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:24):

You could override that behavior in the CLI and remap things as you see fit. But I was thinking for Registry default behavior instructing the client.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:25):

I'm not certain about this. But how I was thinking about it, at the moment.

view this post on Zulip Lann Martin (Oct 04 2023 at 20:29):

OK, so this server-side mapping would be something like "everything named ba:X is mirrored from BA X and any dependencies you see from a ba:* package should be remapped to ba:"?

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:29):

There are trade-offs. If registry federating and re-mapping doesn't change how packages sourced from other registries resolve and compile, that might be a good thing.

view this post on Zulip Lann Martin (Oct 04 2023 at 20:33):

If that is what we're shooting for then I think we probably do need a "namespace mapping" log to maintain our transparency security model

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:35):

Lann Martin said:

OK, so this server-side mapping would be something like "everything named ba:X is mirrored from BA X and any dependencies you see from a ba:* package should be remapped to ba:"?

In this registry,ba namespace is for registry.bytecodealliance.org packages that also use ba namespace. For packages originating from that ba source registry, they would resolve just as if that was the registry you were talking to.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:35):

They would carry the context of the source registry.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:37):

So the re-mapping is really only for resolving name conflicts and disambiguating imports for packages published to your registry.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:38):

And not changing the behavior of imported packages and how they resolve their imports.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:38):

That's how I was thinking about it. But again, I'm not certain.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:40):

Thinking that through, if your registry changes the mappings, the packages in your registry could change behavior. That does suggest that this should be a verifiable log.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:44):

Do you think the registry should be able to change the behavior of how other registries' packages resolve their imports? Through importing it to your registry. Or just have that capability explicitly done in the CLI with remapping client-side?

view this post on Zulip Lann Martin (Oct 04 2023 at 20:48):

I guess that is a fundamental question about what we mean by "mirroring"

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:50):

Suppose it could also be an option that the registry specifies how the client should resolve imports from packages sourced from another registry. But not sure if we want that or not.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:54):

Thus far, I was not listing re-mapping other registries' package imports to new ones as a use case for federation. But it may be one. I'm not sure if that is desired behavior or not.

view this post on Zulip Lann Martin (Oct 04 2023 at 20:55):

Well, we don't have any way to ensure namespace uniqueness between registries. We either need a way to remap or we need to declare that you're just out of luck if you end up with a conflict :shrug:

view this post on Zulip Lann Martin (Oct 04 2023 at 20:56):

I suspect there are also reasons to intentionally shadow another registry's namespace(s) too

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 20:57):

Yeah, it a mechanism of control. I could imagine it could be useful.

view this post on Zulip Lann Martin (Oct 04 2023 at 20:59):

I'm also not sure what feature we are providing to users by telling them that we are mirroring another registry, if not to do that kind of remapping.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:03):

Without remapping, you have could have following other benefits: privacy of network activity, increased availability through replication/caching, data locality by having the API/Content local within a datacenter or network, and implementing access control policies.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:06):

But also, you'd really be talking to your "home" registry. Having mirroring simplifies the dev experience of accessing packages from across other registries.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:07):

I could see remapping as something that might be useful. But not the primary motivator.

view this post on Zulip Lann Martin (Oct 04 2023 at 21:08):

Sure but we don't need any new features to do "transparent mirroring"; a registry can just periodically sync upstream logs

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:09):

Not sure I follow that last statement about not needing any new features. Can you elaborate?

view this post on Zulip Lann Martin (Oct 04 2023 at 21:10):

I mean protocol features

view this post on Zulip Lann Martin (Oct 04 2023 at 21:10):

The mirroring server would still need to learn to sync

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:11):

You mean there wouldn't need to any changes to the logs and protobufs. Just API changes and server/client changes. Correct?

view this post on Zulip Lann Martin (Oct 04 2023 at 21:11):

Not even API changes necessarily; a mirroring server could use the existing API

view this post on Zulip Lann Martin (Oct 04 2023 at 21:12):

Might want some sort of "bulk fetch" in practice but not necessary at current scales

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:13):

I think we need to communicate to the client the mapping of namespaces to which registry. And have the client communicate what registry it wants to request data from, sort of a proxy.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:15):

There are a few options on the API changes.

view this post on Zulip Lann Martin (Oct 04 2023 at 21:16):

hmm yeah if you wanted client-initiated lazy mirroring then you'd need another endpoint

view this post on Zulip Lann Martin (Oct 04 2023 at 21:17):

that seems like it could be an entirely different API surface tho; it would only be applicable to certain kinds of registries

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:17):

Do you think we would need a verifiable log for the namespace mapping to which source registry?

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:17):

Kyle suggested largely keeping the current API and adding an HTTP Header in the request which specified which registry domain you were talking about.

view this post on Zulip Lann Martin (Oct 04 2023 at 21:17):

well, so apart from the "please mirror that registry" API, would this mirroring registry need to indicate that its doing any of this mirroring to clients?

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:19):

The registry would need to indicate to the client if it is aware of a namespace and can fulfill the requests for its packages.

view this post on Zulip Lann Martin (Oct 04 2023 at 21:24):

I feel like I'm missing something implicit in this proposal; would the mirror registry include mirrored package logs in its transparency log?

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:27):

Trying to avoid that, if we can. The client would request the operator and package logs and proof bundles for each registry mirror, but asking for those through the home registry that is mirroring those registries.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:27):

That could either be done as separate HTTP requests in parallel (with an HTTP header indicating the registry) or changing the API to do it in a single request.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:28):

I'm curious how that feels to you and if you see issues.

view this post on Zulip Lann Martin (Oct 04 2023 at 21:31):

hmm ok so lets split that up into two parts:

view this post on Zulip Lann Martin (Oct 04 2023 at 21:31):

ignoring mirroring for a moment, a registry could say "namespace X is actually managed by the BA registry"

view this post on Zulip Lann Martin (Oct 04 2023 at 21:31):

a redirect, more or less

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:32):

Yeah, redirect/proxy

view this post on Zulip Lann Martin (Oct 04 2023 at 21:34):

in the just-redirecting world, if the registry had its own package that pointed into that namespace, the client would see the redirect and fetch from the other registry

view this post on Zulip Lann Martin (Oct 04 2023 at 21:35):

layered on top of that is this warg-flavored "caching proxy", which semantically is the same as talking to the other registry

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:35):

Yeah

view this post on Zulip Lann Martin (Oct 04 2023 at 21:36):

can these be framed as two orthogonal features? I can see them being useful in different combinations

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:37):

By two features, do you mean redirect and caching being two different features?

view this post on Zulip Lann Martin (Oct 04 2023 at 21:38):

yeah, "namespace X actually means registry Y" and "I can proxy registry Y for you"

view this post on Zulip Lann Martin (Oct 04 2023 at 21:39):

I can easily imagine just wanting the proxy, or want to deploy the proxy and the registry-with-redirects separately

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:40):

Yeah, I think it is easy to think of them as different features if we go the route with using an HTTP header and keeping the API largely the same.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:40):

By redirect, do you mean an HTTP redirect or a mapping a namespace to a Registry?

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:41):

I wasn't thinking of using HTTP redirects

view this post on Zulip Lann Martin (Oct 04 2023 at 21:41):

I'm not sure if it would be an http redirect per se (though it would be nice if it could somehow be that simple)

view this post on Zulip Lann Martin (Oct 04 2023 at 21:41):

but a "warg redirect"

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:42):

Yeah, conceptually a redirect.

view this post on Zulip Lann Martin (Oct 04 2023 at 21:42):

splitting them up also hopefully makes it easier to understand the security implications

view this post on Zulip Lann Martin (Oct 04 2023 at 21:43):

like I can pretty easily see how the proxy part can be done securely

view this post on Zulip Lann Martin (Oct 04 2023 at 21:43):

without really changing much fundamentally

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:43):

Yeah, and the HTTP header specifying the Registry domain has a simplicity to it.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:46):

I suppose another benefit of thinking of it as two different features is that client may chose its own mappings and want to proxy the requests through its home Registry. That was related to the question that I asked today on where the mapping should live for namespaces to which Registry. Client-side or server-side. Doesn't need to be exclusive to one or the other.

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:48):

It sounds like the proxy feature is OK by you? And we are still figuring out the mapping feature?

view this post on Zulip Lann Martin (Oct 04 2023 at 21:48):

yep

view this post on Zulip Calvin Prewitt (Oct 04 2023 at 21:49):

That gets us most of the way there. I think.


Last updated: Dec 23 2024 at 12:05 UTC