Lann Martin (Oct 04 2023 at 20:03): Lann Martin (Oct 04 2023 at 20:03):wasi:http and wasi:corewasi:http@1.0.0 contains an unlocked import dep on wasi:core@1.0.0wasi:http from registry BA, naming it ba:wasi:httpwasi:core which may or may not be mirrored from BA.ba:wasi:http@1.0.0 from Fermyon. Should the unlocked dependency resolve to Fermyon's wasi:core or BA's wasi:core? Lann Martin (Oct 04 2023 at 20:05):It seems like the answer has to include "Fermyon" at least as some kind of opt-in choice (if not the default or only option) in order to capture some of the use-cases we are going for.
Lann Martin (Oct 04 2023 at 20:14):but you also would clearly want an option for Fermyon to mirror BA's wasi:core as ba:wasi:core and somehow signal to the client that it should use that to resolve wasi:http@1.0.0's dep on wasi:core (as we discussed in the meeting)
Calvin Prewitt (Oct 04 2023 at 20:18):Perhaps, the default behavior would be... ba:wasi:http originates from Registry "BA" and carries the context of that Registry which would resolve imports in the context of the source Registry. But if a package fermyon:http (published to "Fermyon" registry) imports wasi:core it would be resolve in the context of the "Fermyon" registry. How does that feel?
Lann Martin (Oct 04 2023 at 20:20):Thinking about how that would be implemented, the implication is that a registry can instruct a client to remap any unlocked dependency to any other package, right?
Calvin Prewitt (Oct 04 2023 at 20:23):I was thinking the re-mapping would only be to make it possible for name conflicts to be resolved for packages published to its own registry. Not re-mapping other registries' packages imports to different ones and creating different binaries.
Calvin Prewitt (Oct 04 2023 at 20:24):You could override that behavior in the CLI and remap things as you see fit. But I was thinking for Registry default behavior instructing the client.
Calvin Prewitt (Oct 04 2023 at 20:25):I'm not certain about this. But how I was thinking about it, at the moment.
Lann Martin (Oct 04 2023 at 20:29):OK, so this server-side mapping would be something like "everything named ba:X is mirrored from BA X and any dependencies you see from a ba:* package should be remapped to ba:"?
Calvin Prewitt (Oct 04 2023 at 20:29):There are trade-offs. If registry federating and re-mapping doesn't change how packages sourced from other registries resolve and compile, that might be a good thing.
Lann Martin (Oct 04 2023 at 20:33):If that is what we're shooting for then I think we probably do need a "namespace mapping" log to maintain our transparency security model
Calvin Prewitt (Oct 04 2023 at 20:35):Lann Martin said:
OK, so this server-side mapping would be something like "everything named
ba:Xis mirrored from BAXand any dependencies you see from aba:*package should be remapped toba:"?
In this registry,ba namespace is for registry.bytecodealliance.org packages that also use ba namespace. For packages originating from that ba source registry, they would resolve just as if that was the registry you were talking to.
Calvin Prewitt (Oct 04 2023 at 20:35):They would carry the context of the source registry.
Calvin Prewitt (Oct 04 2023 at 20:37):So the re-mapping is really only for resolving name conflicts and disambiguating imports for packages published to your registry.
Calvin Prewitt (Oct 04 2023 at 20:38):And not changing the behavior of imported packages and how they resolve their imports.
Calvin Prewitt (Oct 04 2023 at 20:38):That's how I was thinking about it. But again, I'm not certain.
Calvin Prewitt (Oct 04 2023 at 20:40):Thinking that through, if your registry changes the mappings, the packages in your registry could change behavior. That does suggest that this should be a verifiable log.
Calvin Prewitt (Oct 04 2023 at 20:44):Do you think the registry should be able to change the behavior of how other registries' packages resolve their imports? Through importing it to your registry. Or just have that capability explicitly done in the CLI with remapping client-side?
Lann Martin (Oct 04 2023 at 20:48):I guess that is a fundamental question about what we mean by "mirroring"
Calvin Prewitt (Oct 04 2023 at 20:50):Suppose it could also be an option that the registry specifies how the client should resolve imports from packages sourced from another registry. But not sure if we want that or not.
Calvin Prewitt (Oct 04 2023 at 20:54):Thus far, I was not listing re-mapping other registries' package imports to new ones as a use case for federation. But it may be one. I'm not sure if that is desired behavior or not.
Lann Martin (Oct 04 2023 at 20:55):Well, we don't have any way to ensure namespace uniqueness between registries. We either need a way to remap or we need to declare that you're just out of luck if you end up with a conflict :shrug:
Lann Martin (Oct 04 2023 at 20:56):I suspect there are also reasons to intentionally shadow another registry's namespace(s) too
Calvin Prewitt (Oct 04 2023 at 20:57):Yeah, it a mechanism of control. I could imagine it could be useful.
Lann Martin (Oct 04 2023 at 20:59):I'm also not sure what feature we are providing to users by telling them that we are mirroring another registry, if not to do that kind of remapping.
Calvin Prewitt (Oct 04 2023 at 21:03):Without remapping, you have could have following other benefits: privacy of network activity, increased availability through replication/caching, data locality by having the API/Content local within a datacenter or network, and implementing access control policies.
Calvin Prewitt (Oct 04 2023 at 21:06):But also, you'd really be talking to your "home" registry. Having mirroring simplifies the dev experience of accessing packages from across other registries.
Calvin Prewitt (Oct 04 2023 at 21:07):I could see remapping as something that might be useful. But not the primary motivator.
Lann Martin (Oct 04 2023 at 21:08):Sure but we don't need any new features to do "transparent mirroring"; a registry can just periodically sync upstream logs
Calvin Prewitt (Oct 04 2023 at 21:09):Not sure I follow that last statement about not needing any new features. Can you elaborate?
Lann Martin (Oct 04 2023 at 21:10):I mean protocol features
Lann Martin (Oct 04 2023 at 21:10):The mirroring server would still need to learn to sync
Calvin Prewitt (Oct 04 2023 at 21:11):You mean there wouldn't need to any changes to the logs and protobufs. Just API changes and server/client changes. Correct?
Lann Martin (Oct 04 2023 at 21:11):Not even API changes necessarily; a mirroring server could use the existing API
Lann Martin (Oct 04 2023 at 21:12):Might want some sort of "bulk fetch" in practice but not necessary at current scales
Calvin Prewitt (Oct 04 2023 at 21:13):I think we need to communicate to the client the mapping of namespaces to which registry. And have the client communicate what registry it wants to request data from, sort of a proxy.
Calvin Prewitt (Oct 04 2023 at 21:15):There are a few options on the API changes.
Lann Martin (Oct 04 2023 at 21:16):hmm yeah if you wanted client-initiated lazy mirroring then you'd need another endpoint
Lann Martin (Oct 04 2023 at 21:17):that seems like it could be an entirely different API surface tho; it would only be applicable to certain kinds of registries
Calvin Prewitt (Oct 04 2023 at 21:17):Do you think we would need a verifiable log for the namespace mapping to which source registry?
Calvin Prewitt (Oct 04 2023 at 21:17):Kyle suggested largely keeping the current API and adding an HTTP Header in the request which specified which registry domain you were talking about.
Lann Martin (Oct 04 2023 at 21:17):well, so apart from the "please mirror that registry" API, would this mirroring registry need to indicate that its doing any of this mirroring to clients?
Calvin Prewitt (Oct 04 2023 at 21:19):The registry would need to indicate to the client if it is aware of a namespace and can fulfill the requests for its packages.
Lann Martin (Oct 04 2023 at 21:24):I feel like I'm missing something implicit in this proposal; would the mirror registry include mirrored package logs in its transparency log?
Calvin Prewitt (Oct 04 2023 at 21:27):Trying to avoid that, if we can. The client would request the operator and package logs and proof bundles for each registry mirror, but asking for those through the home registry that is mirroring those registries.
Calvin Prewitt (Oct 04 2023 at 21:27):That could either be done as separate HTTP requests in parallel (with an HTTP header indicating the registry) or changing the API to do it in a single request.
Calvin Prewitt (Oct 04 2023 at 21:28):I'm curious how that feels to you and if you see issues.
Lann Martin (Oct 04 2023 at 21:31):hmm ok so lets split that up into two parts:
Lann Martin (Oct 04 2023 at 21:31):ignoring mirroring for a moment, a registry could say "namespace X is actually managed by the BA registry"
Lann Martin (Oct 04 2023 at 21:31):a redirect, more or less
Calvin Prewitt (Oct 04 2023 at 21:32):Yeah, redirect/proxy
Lann Martin (Oct 04 2023 at 21:34):in the just-redirecting world, if the registry had its own package that pointed into that namespace, the client would see the redirect and fetch from the other registry
Lann Martin (Oct 04 2023 at 21:35):layered on top of that is this warg-flavored "caching proxy", which semantically is the same as talking to the other registry
Calvin Prewitt (Oct 04 2023 at 21:35):Yeah
Lann Martin (Oct 04 2023 at 21:36):can these be framed as two orthogonal features? I can see them being useful in different combinations
Calvin Prewitt (Oct 04 2023 at 21:37):By two features, do you mean redirect and caching being two different features?
Lann Martin (Oct 04 2023 at 21:38):yeah, "namespace X actually means registry Y" and "I can proxy registry Y for you"
Lann Martin (Oct 04 2023 at 21:39):I can easily imagine just wanting the proxy, or want to deploy the proxy and the registry-with-redirects separately
Calvin Prewitt (Oct 04 2023 at 21:40):Yeah, I think it is easy to think of them as different features if we go the route with using an HTTP header and keeping the API largely the same.
Calvin Prewitt (Oct 04 2023 at 21:40):By redirect, do you mean an HTTP redirect or a mapping a namespace to a Registry?
Calvin Prewitt (Oct 04 2023 at 21:41):I wasn't thinking of using HTTP redirects
Lann Martin (Oct 04 2023 at 21:41):I'm not sure if it would be an http redirect per se (though it would be nice if it could somehow be that simple)
Lann Martin (Oct 04 2023 at 21:41):but a "warg redirect"
Calvin Prewitt (Oct 04 2023 at 21:42):Yeah, conceptually a redirect.
Lann Martin (Oct 04 2023 at 21:42):splitting them up also hopefully makes it easier to understand the security implications
Lann Martin (Oct 04 2023 at 21:43):like I can pretty easily see how the proxy part can be done securely
Lann Martin (Oct 04 2023 at 21:43):without really changing much fundamentally
Calvin Prewitt (Oct 04 2023 at 21:43):Yeah, and the HTTP header specifying the Registry domain has a simplicity to it.
Calvin Prewitt (Oct 04 2023 at 21:46):I suppose another benefit of thinking of it as two different features is that client may chose its own mappings and want to proxy the requests through its home Registry. That was related to the question that I asked today on where the mapping should live for namespaces to which Registry. Client-side or server-side. Doesn't need to be exclusive to one or the other.
Calvin Prewitt (Oct 04 2023 at 21:48):It sounds like the proxy feature is OK by you? And we are still figuring out the mapping feature?
Lann Martin (Oct 04 2023 at 21:48):yep
Calvin Prewitt (Oct 04 2023 at 21:49):That gets us most of the way there. I think.
Last updated: Nov 22 2024 at 16:03 UTC