Stream: git-wasmtime

Topic: wasmtime / PR #4285 Add support for nested components

Wasmtime GitHub notifications bot (Jun 17 2022 at 18:05):

alexcrichton opened PR #4285 (assigned to fitzgen) from nested-components to main:

This commit is an implementation of a number of features of the
component model including:

• Defining nested components
• Outer aliases to components and modules
• Instantiating nested components

The implementation here is intended to be a foundational pillar of
Wasmtime's component model support since recursion and nested components
are the bread-and-butter of the component model. At a high level the
intention for the component model implementation in Wasmtime has long
been that the recursive nature of components is "erased" at compile time
to something that's more optimized and efficient to process. This commit
ended up exemplifying this quite well where the vast majority of the
internal changes here are in the "compilation" phase of a component
rather than the runtime instantiation phase. The support in the
wasmtime crate, the runtime instantiation support, only had minor
updates here while the internals of translation have seen heavy updates.

The translate module was greatly refactored here in this commit.
Previously it would, as a component is parsed, create a final
Component to hand off to trampoline compilation and get persisted at
runtime. Instead now it's a thin layer over wasmparser which simply
records a list of LocalInitializer entries for how to instantiate the
component and its index spaces are built. This internal representation
of the instantiation of a component is pretty close to the binary format
intentionally.

Instead of performing dataflow legwork the translate phase of a
component is now responsible for two primary tasks:

1. All components and modules are discovered within a component. They're
   assigned Static{Component,Module}Index depending on where they're
   found and a {Module,}Translation is prepared for each one. This
   "flattens" the recursive structure of the binary into an indexed list
   processable later.

2. The lexical scope of components is managed here to implement outer
   module and component aliases. This is a significant design
   implementation because when closing over an outer component or module
   that item may actually be imported or something like the result of a
   previous instantiation. This means that the capture of
   modules and components is both a lexical concern as well as a runtime
   concern. The handling of the "runtime" bits are handled in the next
   phase of compilation.

The next and currently final phase of compilation is a new pass where
much of the historical code in translate.rs has been moved to (but
heavily refactored). The goal of compilation is to produce one "flat"
list of initializers for a component (as happens prior to this PR) and
to achieve this an "inliner" phase runs which runs through the
instantiation process at compile time to produce a list of initializers.
This inline module is the main addition as part of this PR and is now
the workhorse for dataflow analysis and tracking what's actually
referring to what.

During the inline phase the local initializers recorded in the
translate phase are processed, in sequence, to instantiate a
component. Definitions of items are tracked to correspond to their root
definition which allows seeing across instantiation argument boundaries
and such. Handling "upvars" for component outer aliases is handled in
the inline phase as well by creating state for a component whenever a
component is defined as was recorded during the translate phase.
Finally this phase is chiefly responsible for doing all string-based
name resolution at compile time that it can. This means that at runtime
no string maps will need to be consulted for item exports and such.
The final result of inlining is a list of "global initializers" which is
a flat list processed during instantiation time. These are almost
identical to the initializers that were processed prior to this PR.

There are certainly still more gaps of the component model to implement
but this should be a major leg up in terms of functionality that
Wasmtime implements. This commit, however leaves behind a "hole" which
is not intended to be filled in at this time, namely importing and
exporting components at the "root" level from and to the host. This is
tracked and explained in more detail as part of #4283.

cc #4185 as this completes a number of items there

<!--

Please ensure that the following steps are all taken care of before submitting
the PR.

• [ ] This has been discussed in issue #..., or if not, please tell us why
  here.

• [ ] A short description of what this does, why it is needed; if the
  description becomes long, the matter should probably be discussed in an issue
  first.

• [ ] This PR contains test cases, if meaningful.

• [ ] A reviewer from the core maintainer team has been assigned for this PR.
  If you don't know who could review this, please indicate so. The list of
  suggested reviewers on the right can help you.

Please ensure all communication adheres to the code of conduct.
-->

Wasmtime GitHub notifications bot (Jun 17 2022 at 18:05):

alexcrichton assigned PR #4285 to fitzgen.

Wasmtime GitHub notifications bot (Jun 17 2022 at 18:06):

alexcrichton has marked PR #4285 as ready for review.

Wasmtime GitHub notifications bot (Jun 17 2022 at 18:29):

alexcrichton updated PR #4285 (assigned to fitzgen) from nested-components to main.

Wasmtime GitHub notifications bot (Jun 17 2022 at 19:03):

alexcrichton requested fitzgen for a review on PR #4285.

Wasmtime GitHub notifications bot (Jun 21 2022 at 17:59):

fitzgen submitted PR review.

Wasmtime GitHub notifications bot (Jun 21 2022 at 17:59):

fitzgen submitted PR review.

Wasmtime GitHub notifications bot (Jun 21 2022 at 17:59):

fitzgen created PR review comment:

We should clarify that this constraint is only at the root component, since importing and exporting components is just fine in between nested components.

// way to implement either importing or exporting a component from the root component itself. Currently

Wasmtime GitHub notifications bot (Jun 21 2022 at 17:59):

fitzgen created PR review comment:

// * Components can have arbitrary nesting and internally do instantiations via

Wasmtime GitHub notifications bot (Jun 21 2022 at 17:59):

fitzgen created PR review comment:

Can we rename this type GlobalInitializer just to be super clear about the local vs global distinction?

Wasmtime GitHub notifications bot (Jun 21 2022 at 17:59):

fitzgen created PR review comment:

//! nested component is instantiated. The inlining then refers to how this
//! stack of instantiations is flattened to one list of `Initializer` entries to
//! represent the process of instantiating a component graph, similar to how function
//! inlining removes call instructions and creates one giant function for a call graph. Here

Wasmtime GitHub notifications bot (Jun 21 2022 at 17:59):

fitzgen created PR review comment:

Can you add a test here that has something like

• root component
  • parent component, imports f
    • child component closes over f
  • instantiate parent with f1
  • instantiate parent with f2
  • assert that each parent's child closed over the appropriate f1/f2

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:00):

fitzgen created PR review comment:

either here or in the wast files (I didn't see anything that exercised outer aliases/closures in this particular way)

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:00):

fitzgen submitted PR review.

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:05):

alexcrichton updated PR #4285 (assigned to fitzgen) from nested-components to main.

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:07):

alexcrichton submitted PR review.

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:07):

alexcrichton created PR review comment:

That I think should be covered by this test if I'm understanding you right, but could you double-check to be sure?

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:12):

fitzgen submitted PR review.

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:12):

fitzgen created PR review comment:

Ah, so it is! Thanks!

Wasmtime GitHub notifications bot (Jun 21 2022 at 18:48):

alexcrichton merged PR #4285.

Last updated: Dec 23 2024 at 12:05 UTC