Stream: general

Topic: Importee calling into importer

Surma (Mar 06 2023 at 13:43):

I am thinking about a plugin system(-ish), where I’d want the runtime to invoke the plugins, but some plugins will also need to call stuff in the runtime. However, Components don’t have higher-order functions, and the import graph is a DAG, so no cycles can be present. I also don’t think I can somehow shoehorn function pointers (passed as resources maybe?) would work, as I don’t think I can control how a language puts these functions into their tables and the tables won’t necessarily be shared... What is the expected/recommended pattern for this kind of architecture?

default world runtime {
  import plugin-a: interface {
    do-a-thing: func() // This function wants to call `change_state`
  }
  // ... more plugins with the same interface ...
  export main: func()
  export change_state: func(state: u32)
}

Ramon Klass (Mar 06 2023 at 14:26):

I am not sure if this is the only way, but I did that by defining the worlds from the viewpoint of a plugin

default world plugin {
    // these functions are from the host
    import log: func(msg: string)
    import error: func(msg: string)

    //these are implemented in each plugin
    export setup: func() -> list<u8>
    export handler: func(fname: string) -> list<u8>
    export help: func()
}

as an example. Each of my plugins can use the log functions and each plugin exports setup, handler and help

Lann Martin (Mar 06 2023 at 14:51):

Another thing to remember is that the import graph is only a DAG for component instances. A component can appear multiple places in a path through the DAG as long as it is instantiated multiple times (doesn't share state)

Surma (Mar 06 2023 at 15:04):

Hm, right, but I do explicitly want to share state :thinking:

Lann Martin (Mar 06 2023 at 15:10):

In the simple example above, do-a-thing could return the next state. If it needs multiple state changes it could return a option<u32> and be called multiple times until it returns nothing

Lann Martin (Mar 06 2023 at 15:10):

That could be replaced by a stream<u32> once streams exist

Surma (Mar 06 2023 at 16:16):

Yeah, polling is one way out of there I suppose. I do worry about the overhead of that, though, with the lowering and lifting.

Being able to pass around function references across instance boundaries seems like a pretty powerful primitive that’s missing. But I’m also sure that’s a lot more complicated than I make it sound :sweat_smile:

Lann Martin (Mar 06 2023 at 18:02):

It isn't so much about how complicated it is; the component model has a "no reentering from imports" invariant. I'd be happy if someone could point to some discussion of that rule because I haven't quite wrapped my head around the rationale for it yet

Surma (Mar 08 2023 at 15:35):

Oooh, that invariant does sound very relevant. I’d be curious to learn more.

It does seem a bit surprising at the higher level — if the component model is supposed to address linking problem space, then components should be able to function like libraries, and there are libraries that utilize the callback pattern — I am not sure how those are supposed to get mapped to the component space (unless it’s supposed to be via core functions + table and the library component imports the table?)

Dan Gohman (Mar 08 2023 at 16:16):

For many cases, libraries that would want callbacks will want to use stream, once the component model supports that. Since it's async, it can interleave work from the producer of the stream and the consumer of the stream in much the same way that a callback would. It may even be implemented inside the component as actual callbacks.

Dan Gohman (Mar 08 2023 at 16:17):

And a benefit of organizing it as a stream rather than a callback is that the system as a whole avoids the complexity of cross-component lifetime management for callback closure objects.

Last updated: Nov 22 2024 at 16:03 UTC