Stream: git-wasmtime

Topic: wasmtime / PR #5153 Refactor metadata storage in AOT arti...

Wasmtime GitHub notifications bot (Oct 28 2022 at 21:11):

alexcrichton opened PR #5153 from refactor-compilation-metadata to main:

This commit is a reorganization of how metadata is stored in Wasmtime's compiled artifacts. Currently Wasmtime's ELF artifacts have data appended after them to contain metadata about the Engine as well as type information for the module itself. This extra data at the end of the file is ignored by ELF-related utilities generally and is assembled during the module serialization process.

In working on AOT-compiling components, though, I've discovered a number of issues with this:

• Primarily it's possible to mistakenly change an artifact if it's deserialized and then serialized again. This issue is probably theoretical but the deserialized artifact records the Engine configuration at time of creation but when re-serializing that it serializes the current Engine state, not the original Engine state.

• Additionally the serialization strategy here is tightly coupled to Module and its serialization format. While this makes sense it is not conducive for future refactorings to use a similar serialization format for components. The engine metadata, for example, does not necessarily need to be tied up with type information.

• The storage for this extra metadata is a bit wonky by shoving it at the end of the ELF file. The original reason for this was to have a compiled artifact be multiple objects concatenated with each other to support serializing module-linking-using modules. Module linking is no longer a thing and I have since decided that for the component model all compilation artifacts will go into one object file to assist debugability. This means that the extra stick-it-at-the-end is no longer necessary.

To solve these issues this commit splits up the
module/serialization.rs file in two, mostly moving the logic to engine/serialization.rs. The engine serialization logic now handles everything related to Engine compatibility such as targets, compiler flags, wasm features, etc. The module serialization logic is now exclusively interested in type information.

The engine metadata and serialized type information additionally live in sections of the final file now instead of at the end. This means that there are three primary bincode-encoded sections that are parsed on deserializing a file:

1. The Engine-specific metadata. This will be the same for both modules and components.
2. The CompiledModuleInfo structure. For core wasm there's just one of these but for the component model there will be multiple, one per core wasm module.
3. The type information. For core wasm this is a ModuleTypes but for a component this will be a ComponentTypes.

No true functional change is expected from this commit. Binary artifacts might get inflated by a small handful of bytes due to using ELF sections to represent this now.

A related change I made during this commit as well was the plumbing of the is_branch_protection_enabled flag. This is technically Engine-level metadata but I didn't want to plumb it all over the place as was done now, so instead a new section was added to the final binary just for this bti information. This means that it no longer needs to be a parameter to CodeMemory::publish and additionally is more amenable to a Component-is-just-one-object world where no single module owns this piece of metadata.

<!--

Please ensure that the following steps are all taken care of before submitting
the PR.

• [ ] This has been discussed in issue #..., or if not, please tell us why
  here.

• [ ] A short description of what this does, why it is needed; if the
  description becomes long, the matter should probably be discussed in an issue
  first.

• [ ] This PR contains test cases, if meaningful.

• [ ] A reviewer from the core maintainer team has been assigned for this PR.
  If you don't know who could review this, please indicate so. The list of
  suggested reviewers on the right can help you.

Please ensure all communication adheres to the code of conduct.
-->

Wasmtime GitHub notifications bot (Oct 28 2022 at 21:11):

alexcrichton requested pchickey for a review on PR #5153.

Wasmtime GitHub notifications bot (Oct 28 2022 at 23:15):

pchickey submitted PR review.

Wasmtime GitHub notifications bot (Oct 29 2022 at 16:35):

alexcrichton updated PR #5153 from refactor-compilation-metadata to main.

Wasmtime GitHub notifications bot (Oct 29 2022 at 17:01):

alexcrichton has enabled auto merge for PR #5153.

Wasmtime GitHub notifications bot (Oct 29 2022 at 17:13):

alexcrichton merged PR #5153.

Last updated: Dec 23 2024 at 12:05 UTC