Stream: wasi

Topic: `wasi-http` middleware is infeasible?


view this post on Zulip Joel Dice (Nov 06 2023 at 18:01):

Context: I'm working on adding resource support to wasm-compose and wasi-virt. In the process, I was going to update the middleware example in the wasm-tools repo to use wasi-http-style resources to represent requests and responses. However I immediately ran into a problem which leads me to wonder if implementing general-purpose middleware using component composition for wasi:http/incoming-handler might not be possible.

The example in the wasm-tools repo involves composing a service component with a middleware component such that the latter compresses the body of the response returned by the former. That works just fine when the response is represented as a record with a list<u8> body. With wasi-http, though, both requests and responses are represented as resources, and the signature of the wasi:http/incoming-handler#handle function takes IncomingRequest and ResponseOutparam resource handles. Ideally, the middleware component would export the handle function from the host and import it from the service component. Moreover, the middleware component should import the relevant resource types from the host and export its own "virtualized" versions of those resources to the service component. However, a component cannot import a function which refers to types that it is exporting, which means the handle function it imports from the service component can only refer to the host's versions of those resource types.

Is there a way around this? The best I can come up with so far (and this just occurred to me, so it's a half-baked idea) is to add a third component to the above scenario which exports virtualized versions of the relevant resources plus some kind of "hook" API the middleware component can use to intercept the resource method calls it cares about. Then both the middleware and the service component would import the resource types from the third component, which in turn would import the "real" versions from the host.

view this post on Zulip Joel Dice (Nov 07 2023 at 20:39):

Just had a good conversation with @Luke Wagner about this, which I'll try to summarize here (any errors are my own, which perhaps he'll be kind enough to correct).

First, you can create a component which both imports from and exports to another component. This can't be expressed in WIT and can't be done using wasm-compose (to my knowledge), but you could theoretically build a wrapper component around the original one such that the wrapper instantiates the original and handles the cyclical relationship using lifted functions which use call_indirect at the core Wasm level. In fact, you can create arbitrary graphs of components containing cycles as long as the cycles are "broken" using call_indirect. Presumably wasi-virt does something like this.

Second, you can always do service chaining the old-fashioned way by having the middleware component do an outbound HTTP request to the service component (which the host might optimize if it notices the service can be invoked without using the network at all). I.e. you don't need component composition to do service chaining, and you probably won't want to use it for that until Preview 3.

Once Preview 3 arrives, we'll be able to do efficient, local service chaining using component composition, and this will not require virtualizing resource types. In my middleware/service example, the middleware component can call the service component's handle function, which returns a Response resource, from which middleware can extract the body as a stream (and anything else it wants), create a new Response with new stream, and pipe from the original stream to the new one while applying any desired transformations (e.g. compression). None of that requires virtualizing any resource types, breaking cycles, etc. (i.e. it can all be done with standard, wasm-compose-style composition).

view this post on Zulip kellytk (Nov 14 2023 at 15:13):

Re: last paragraph, is the juggling kinda similar to how it goes with Rust's std::process::Command? (stdin et al)

view this post on Zulip Joel Dice (Nov 14 2023 at 17:06):

Yeah, that seems like a good analogy. You can create multiple Commands and pipe the stdout of one to the stdin of another, using the equivalent plumbing to what a POSIX shell would do when you use the | (pipe) operator.

view this post on Zulip Kate Goldenring (Nov 14 2023 at 22:25):

@Joel Dice thank you for the summary. Just a couple clarifying questions.
You mention, "I.e. you don't need component composition to do service chaining, and you probably won't want to use it for that until Preview 3." Service chaining seems to be a really big use case that WASI HTTP + preview 2 has unlocked. I am not sure I understand why we don't want to use it yet, even without WASI virt.

I just want to clarify why we think preview 3 will solve this. It sounds like with preview 3 we are avoiding needing to import/export Types? Is this a goal or side effect of preview 3. Just catching up.

view this post on Zulip Joel Dice (Nov 14 2023 at 22:26):

Guy and I just realized wasi-virt --allow-http is broken for exactly the reason I described above. It currently relies on wasm-compose, which can't handle cycles (as noted earlier). If you're only doing outbound requests, you're fine (or at least you will be once I PR a fix for that), but if you're handling inbound requests you hit the cycle issue.

In my summary above, I basically concluded that we don't need or want virtualization to implement middleware anyway, but Guy reminded me that if you're not virtualizing wasi-http, you can't use wasi-virt to virtualize anything involving streams. That means no virtual filesystem, etc. At most, you can virtualize the bits that don't involve streams (e.g. env, clocks, and random), but that's it. Kind of a sad state of affairs.

So now Guy and I are thinking we do want to support virtualizing all of wasi-http, including inbound, which I think means we need to do one of the following:

view this post on Zulip Joel Dice (Nov 14 2023 at 22:30):

@Kate Goldenring: Till and I believe the best way to do general purpose service chaining with Preview 2 is by locally connecting outbound requests to incoming handlers without going over the network. That allows us to do concurrent I/O with minimal buffering, which is not possible otherwise via composition given the current state of the component model. For middleware applications that don't need to stream request or response bodies across component boundaries, composition can still be a good option, though.

view this post on Zulip Joel Dice (Nov 14 2023 at 22:33):

@Kate Goldenring: The plan for Preview 3 is to include native async and stream support in the component model, which will allow concurrent I/O across composed component boundaries.

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:35):

one way to explain this that may be helpful: in preview 2, scheduling of concurrent IO is managed with wasi:io/poll and wasi:io/stream, which are component model interfaces that expose resources (pollable, input-stream, output-stream)

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:36):

in order to import a component model interface that exposes any sort of io (meaning, anything in the interface returns a pollable or stream, like the wasi-http interface does), you also need to import the wasi:io package's interfaces from that same component

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:37):

and you can only have a single import of the wasi:io package's imports per component

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:37):

this effectively means, every time you import an interface such as http, you have to import it from a component that provides the entire world of io and every other interface that uses io

view this post on Zulip Joel Dice (Nov 14 2023 at 22:38):

@Peter Huene do you have thoughts on the wasi-virt situation I described above?

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:39):

wasi-virt is the only component that tries to provide this complete wrapper of your world, but wasi-virt itself is incomplete so it doesnt actually do it yet. once joel (or someone else) fixes the problems with wasi-virt, it will provide the entire set of interfaces you import, even if all you need to do is override one part of the filesystem, it needs to virtualize your http as well

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:39):

ive called this style of component composition "onion composition" previous, because each layer you add on has to wrap the entire other layer, like an onion

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:40):

in short, it is not ideal. it is just the way these things have to work, at the moment, because wasi:io is an ordinary wit package.

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:41):

the goal for preview 3 (though, preview 3 is just whatever ships at the end of 2024, and who knows if this will be complete, so id actually prefer we start calling it something else like "first-class component model async" or something) is that instead of being a wit package, the primitives for scheduling and streams become component model primitives

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:41):

at which point we have a lot more power with how they can be implemented, they dont have to follow many of the rules of the canonical ABI as it exists today

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:41):

and then we can have composition of components that use the stream and scheduling primitives without having to do that onion composition

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:43):

among other things, that will make service chaining between http much simpler, because one element in the chain wont have to onion-wrap the other - if the inner element needs filesystem, the outer element doesnt have to virtualize that filesystem

view this post on Zulip Pat Hickey (Nov 14 2023 at 22:43):

hopefully all of that makes sense, please let me help clarify if it doesnt!

view this post on Zulip Joel Dice (Nov 16 2023 at 21:10):

Here's an update on the state of wasi-virt support for wasi:http/incoming-handler. First, a summary of the problem:

After discussing this with @Alex Crichton, we've concluded that although it's technically possible to implement the three-component solution described above with some low-level WASM file manipulation, it would be better to add syntax to WIT which allows it to be expressed at that level. That's not something we'll want to tackle until after WASI Preview 2 ships, though. So we feel that this is not the right time to try to support wasi:http/incoming-handler in wasi-virt. Instead, we plan to make the WIT syntax additions a priority post WASI Preview 2, at which point we'll have what we need to work on wasi-virt.

view this post on Zulip Joel Dice (Nov 16 2023 at 21:17):

Here's a diagram, in case it helps.
cycle.drawio1.png

view this post on Zulip Luke Wagner (Nov 20 2023 at 20:26):

Great to hear you're digging into virtualization, and these challenges make sense. Two ideas:

view this post on Zulip Ryan Levick (rylev) (Dec 01 2023 at 17:34):

I've created a POC for doing middleware style chaining using wasi-http: https://github.com/rylev/wasi-http-middleware-prototype

The middleware calls down the middleware chain by calling into its outgoing-handler import which is intercepted by the runtime and forwarded to the downstream component which receives the request (without knowing that it actually came from a middleware component).

Currently the demo code always forwards the outgoing-handler's request to the downstream components, but we'll want to come up with some protocol for distinguishing between passing a request down a middleware chain and making a legitimate HTTP call. @Lann Martin had the idea of using the special TLD .alt for this.

Contribute to rylev/wasi-http-middleware-prototype development by creating an account on GitHub.

Last updated: Dec 23 2024 at 12:05 UTC