cfallin opened PR #12756 from cfallin:a-whole-wide-world-for-debuggers-to-play-in to bytecodealliance:main:
This PR "draws the rest of the owl" for the debug-main world (bytecodealliance/rfcs#45). This includes a WIT world that hosts debug components that have access to "host debug powers" via a debugging API, and the ability to load such a debug-component and give it control of the main program as a debuggee when using
wasmtime run.The WIT is namespaced to
bytecodealliance:wasmtimeand is slightly aspirational in places: for example, the host does not yet implement injection of early return values or exception-throws. I intend to fill out a series of TODO issues once this all lands to track followup ("post-MVP") work.This PR does not include any debug components. I separately have a gdbstub component, with which I tested and co-developed this host-side implementation. My plan is to land it in a followup PR as a component that will be embedded in/shipped with the Wasmtime CLI and available under an easy-to-use CLI option. Once we have that gdbstub component, we can also implement end-to-end integration tests that boot up LLDB and run through an expected interaction. (Separately, those integration tests will require a release of wasi-sdk to ship an LLDB binary that we can use.) As such, there are no real tests in this PR: interesting behaviors only really occur with a full end-to-end flow.
The integration with the CLI is a little awkward (we internally build another
wasmtime runcommand that invokes the debug component, and tie it together with the debuggee via a specialinvoke_debuggerAPI; this seemed less bad than reworking all of the WASI setup to be more reusable). Happy to take more ideas here.<!--
Please make sure you include the following information:
If this work has been discussed elsewhere, please include a link to that
conversation. If it was discussed in an issue, just mention "issue #...".Explain why this change is needed. If the details are in an issue already,
this can be brief.Our development process is documented in the Wasmtime book:
https://docs.wasmtime.dev/contributing-development-process.htmlPlease ensure all communication follows the code of conduct:
https://github.com/bytecodealliance/wasmtime/blob/main/CODE_OF_CONDUCT.md
-->
cfallin requested alexcrichton for a review on PR #12756.
cfallin requested wasmtime-core-reviewers for a review on PR #12756.
cfallin requested wasmtime-default-reviewers for a review on PR #12756.
cfallin commented on PR #12756:
(For reference, my plan is to stack a PR on top of this with the gdbstub component as soon as my last PR to upstream
gdbstublands.)
cfallin commented on PR #12756:
I considered ways to try to break this up into smaller pieces, but a few thoughts:
- The linecount on the diff is a little misleading because this (necessarily) replicates all of the wasip2 WITs in
deps/; the actual host impl is ~1.6kLoC.- Those 1.6kLoC are tedious but just a bunch of accessors, morally -- landing, say, half of the memory accessors separately wouldn't meaningfully reduce complexity IMHO.
Happy to take any thoughts you have on this though!
cfallin updated PR #12756.
cfallin updated PR #12756.
cfallin updated PR #12756.
alexcrichton submitted PR review:
Very nice, it's awesome to see this all come together!
W.r.t. PR size -- I think this one's fine. It's all safe code using preexisting APIs and the new boundaries here, primarily the CLI interface, are easy enough to review in isolation. While this could be split up I'm ok landing this pretty much as-is given that all the major foundational decisions have already been sorted out.
W.r.t. tests -- we talked about this but to write this down here too -- I think it'd be good to have a small test suite which tests a few "hello world" interactions of the CLI with custom debugger wasms. The intent here is to build out enough infrastructure to have a place to put regression tests added in the future as opposed to thoroughly testing the entire interface. Most of the testing will probably happen through the gdbstub component for now. My recommendation would be do drop debugger/debuggee pairs in
crates/test-programs/src/bin/*and that can be a mixture of Rust/C/C++/Wat. That'd all be woven together probably in a newtests/all/debug/cli.rsor something like that which would be modeled after the existingtests/all/cli_tests.rs.Also, for stylistic things in this PR (like flattening results) I think it would also be quite reasonable if you'd prefer to defer such changes to future PRs/issues. I think it's most valuable at this juncture to get all your initial work landed and positioned for iteration, and this is all sequestered/fenced off well enough to be pretty easily refactorable without affecting too much else. Basically, feel free to defer comments below where you feel appropriate.
alexcrichton created PR review comment:
One thing I might recommend is using the
trappable_error_typeconfiguration ofbindgen!to flatten theResult<Result<..>>here. That'd require defining an error type in this crate (similar toTrappableErrorin thewasmtime-wasicrate), but ergonomically it might be a net win since there'd only be one layer ofResulton all these functions and?would work a bit more well
alexcrichton created PR review comment:
Can you update
ci/vendor-wit.shto manage these files?
alexcrichton created PR review comment:
I realize I'm a bit of a swinging pendulum on this, but I might recommend breaking up this file into smaller portions. For example:
- A
bindingssubmodule with just thebindgen!-generated types and any goop necessary there- An
opaquemodule forOpaqueDebuggerand its concrete implementation- A
hostsubmodule for implementations of all the bindgen-generatedHosttraits- The main module would then have public API things like
add_to_linker,add_debuggee, etc.
alexcrichton created PR review comment:
Stylistically you can probably do away with most of this -- I realize much of this is probably copied over from WASI crates but for example:
- The
DebuggerImplwrapper should no longer be necessary due to various restrucurings- Above I believe you can directly to
impl Host for ResourceTable- You can probably nix
DebuggerViewin favor of using closures directlyOne thing I'd recommend copying over though is to have a dedicated
add_to_linkerfunction in this crate that's handwritten. That'd delegate to the bindgen-generated version and would use thisHasDebuggerViewtype (plus probably a projection closure passed in), and internally it'd wire it up to all necessary interfaces.
alexcrichton created PR review comment:
Should the
Debuggertype perhaps be renamed toDebuggee? It's a bit confusing here having this switch back and forth where it's taking the debuggee store and then the debugger right below becomes the debuggee
alexcrichton created PR review comment:
Can this branch return an error if
profileoptions ortimeoutoptions are set because they aren't respected?
alexcrichton created PR review comment:
This raises a few questions for me personally about what's going on here:
- How come stdout isn't inherited by the debugger? Wouldn't that mean that
println!by default wouldn't actually go anywhere?- I can imagine use cases where the debugger itself is reading stdin for commands (e.g. lldb compiled-to-wasm) in which case inheriting stdin to the debugger would be expected. Given that I'm not sure that this may want to be unconditionally hardcoded to
false. Do you know what other debuggers like gdb/lldb do about the stdin of the debuggee? Naively I'd actually expect stdin to go to the debugger by default rather than the debuggee, and then the debugger would control the ability to customize the stdin of the debuggee.- Combining inheritance of stdin/stdout into one option here feels a bit ad-hoc and too-specific to this use case. Could those be split up into
-Sinherit-stdinand-Sinherit-stdout? And would it make sense to also add-Sinherit-stderr?
alexcrichton created PR review comment:
For complete-ness I'd ideally like to see this handle the case that these options are previously
Some(false)(same for those below) to handle cases where certain option combinations don't make sense and/or aren't respected. I think it's reasonable for options to influence the defaults of others, but if-Ddebugger -Dguest-debug=nis passed then I think that should be an error because the second option is otherwise silently ignored.
alexcrichton created PR review comment:
Original comment for posterity
To bikeshed this a for a moment, in the CLI here we technically have 4 different things to configure:
- argv for the debugger and debuggee
- Wasmtime options for the debugger and the debuggee
Configuring options for the debugger is pretty unwieldy requiring
-Ddebugger-arg=...as a prefix for every option, but it does get the job done at least. I might bikeshed that to at least be-Darg=...as the-Dpart implies "debugger stuff" already in theory.Going further though another option is something like:
$ wasmtime debug -Sdebugger-option debugger-argv -- -Sdebugee-option guest.wasm guest-argvmore often-than-not that'd be
wasmtime debug -- guest.wasm. Basically though if we went with a different top-level subcommand that'd open up a bit more flexibilty in terms of argument parsing. Another possible option would be:$ wasmtime debug -d=-Sdebugger-option -d debugger-argv -S debuggee-option guest.wasm guest-argvor something like that with a lightweight option saying "this goes to the debugger" and everything else goes to the debuggee.
Personally I sort of like the idea of a top-level
wasmtime debugcommand rather than tacking it ontowasmtime run, although I would expect much of the implementation to look the same as what's here today (probably just reorganized slightly)
Follow-up
@cfallin and I talked after the Cranelift meeting today about this. Chris reminded me that the goal is to eventually enable debugging with
wasmtime serveas well in which case a top-leveldebugsubcommand probably no longer makes sense, and I agree. Given that what we settled on was:
- Change
-Ddebugger-arg=...to-Darg=...(minor bikeshed)- Plan to add
-g 1234as a top-level argument towasmtime {run,serve}which enables using the built-in gdbstub component- Otherwise leave things as-is
That way most users will likely interact with this as
wasmtime run -g 1234 foo.wasmwhich feels succinct enough to me at least to be a pleasant interface
alexcrichton created PR review comment:
Can this
.join()the thread to clean it up at the end here?
primoly submitted PR review:
Making the API more consistent by having the
debuggeeparameter in the same position in all functions.(Sorry for being pedantic, I just noticed those things while reading the API and thought it would be better to tidy this up now than after it has been merged and started to be used in tooling.)
primoly created PR review comment:
d: Resource<Debuggee>, addr: u64,
primoly created PR review comment:
d: Resource<Debuggee>, addr: u64,
primoly created PR review comment:
d: Resource<Debuggee>, addr: u64,
primoly created PR review comment:
d: Resource<Debuggee>, addr: u64,
primoly created PR review comment:
d: Resource<Debuggee>, addr: u64,
primoly created PR review comment:
d: Resource<Debuggee>, addr: u64,
primoly created PR review comment:
get-u16: func(d: borrow<debuggee>, addr: u64) -> result<u16, error>; /// Get a u32 (in little endian order) at an address. get-u32: func(d: borrow<debuggee>, addr: u64) -> result<u32, error>; /// Get a u64 (in little endian order) at an address. get-u64: func(d: borrow<debuggee>, addr: u64) -> result<u64, error>;
primoly created PR review comment:
d: Resource<Debuggee>, addr: u64,
primoly created PR review comment:
/// Set a u8 (byte) at an address. Returns `none` if out-of-bounds. set-u8: func(d: borrow<debuggee>, addr: u64, value: u8) -> result<_, error>; /// Set a u16 (in little endian order) at an address. set-u16: func(d: borrow<debuggee>, addr: u64, value: u16) -> result<_, error>; /// Set a u32 (in little endian order) at an address. set-u32: func(d: borrow<debuggee>, addr: u64, value: u32) -> result<_, error>; /// Set a u64 (in little endian order) at an address. set-u64: func(d: borrow<debuggee>, addr: u64, value: u64) -> result<_, error>;
primoly submitted PR review.
primoly created PR review comment:
Could
memoryhave something likeread/writemethods that return/takelist<u8>? The current implementation ofread_addrsin gdbstub-component that has to read each byte individually[^1] seems inefficient.
cfallin submitted PR review.
cfallin created PR review comment:
That's a great question. It's pretty ad-hoc at the moment. Initially I wanted to have no stdin/out/err attached to the debugger component at all, to keep things clean and isolated; but stderr was useful for debug-logging, so I opted to make that a special case. I agree it's pretty weird if you don't already know how it's magically configured.
As far as I can tell, native debuggers like lldb/gdb let a program with stdin read stdin when in "running" state and read themselves when paused at a prompt. If either side tries to do terminal configuration ioctls to disable echo or whatever, probably you just have a Bad Time. This feels like one of those POSIX things that we shouldn't aspire too eagerly to replicate. I guess I could add three inherit-std{in,out,err} options and let all be off by default -- that's probably the cleanest default if we're starting from a blank slate (and someone authoring their own debugger component will be "in it" enough to look up these options probably).
cfallin submitted PR review.
cfallin created PR review comment:
Yep, finagled this a bit more -- slightly cleaner now. Thanks!
cfallin submitted PR review.
cfallin created PR review comment:
Ah, that's much better ergonomics -- done!
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin updated PR #12756.
cfallin created PR review comment:
Done; thanks for noticing this!
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin submitted PR review.
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin submitted PR review.
cfallin created PR review comment:
Ah, yep, that's a good idea -- added
read-bytes/write-bytes.
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin updated PR #12756.
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Done (updated to
-Darg=...).-gwill come in a followup PR with the gdbstub component. Thanks for the discussion on this one!
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin updated PR #12756.
cfallin created PR review comment:
Yep, that's a good point -- I only came to the terminology when writing the wit layer; propagated it downward to the
crate::Debugger->Debuggee.
cfallin submitted PR review.
cfallin updated PR #12756.
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
OK, I've split
-S inherit-stdin-stdoutinto-S inherit-stdinand-S inherit-stdout, added-S inherit-stderrfor consistency, made them all true by default (for the main program), and added corresponding-D inherit_std{in,out,err}which are all false by default.
cfallin commented on PR #12756:
All feedback addressed, I believe (thanks!); I'll add some basic test components and tests to run them tomorrow (then hopefully get the gdbstub component up as my next PR).
alexcrichton submitted PR review.
alexcrichton created PR review comment:
I think this is a stray line now
alexcrichton created PR review comment:
This might better fit in
api.rs
alexcrichton created PR review comment:
(or maybe
bindings.rs? my hunch is notopaque.rsregardless)
alexcrichton created PR review comment:
Could these setters also have guards like above which fails if the stored value is in conflict with the previous value?
cfallin submitted PR review.
cfallin created PR review comment:
Ah, these are explicitly copying over settings to the new
debugger_runthat we're creating from scratch -- the initial value isn't user-controlled at all (it's just the default forwasmtime run, but we're overriding that for debuggers).
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Sure, moved.
cfallin submitted PR review.
cfallin created PR review comment:
Good catch; thanks!
alexcrichton submitted PR review.
alexcrichton created PR review comment:
What I was worried about here is
-Darg=-Sinherit-stdio=n -Dinherit-stdio=y. In a sense-Dinherit-stdioi the same as-Darg=-Sinherit-stdioexcept easier to type, but I think we'll want to handle the theoretical conflicts here as well -- (I'm imagining a sort of generic function that handles the above cases plus these)
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Oh, interesting, I hadn't considered passthrough in that way. I just tested and it seems that one can't actually pass through "wasmtime args" using
-Darg=because we internally build and parse a commandwasmtime run debug_component.wasm <args>. Still, I like the idea of unifying all of these checks; see latest commit for an inner helper that handles setting debug-specific overrides and erroring on conflict.Also in the process of testing inherit-std{out,err} options I noticed that
inherit_stdin_stdout(and now my splitinherit_std{in,out,err}) were used in the wasip1 context setup but there was an unconditionalinherit_stdioin the wasip2 setup (pre-existing). I went ahead and fixed that as well.
cfallin updated PR #12756.
cfallin commented on PR #12756:
@alexcrichton I've added some basic tests with a very simple debug component (one that single-steps a few times to smoke-test everything, one that interrupts an infinite loop to check that the async machinery works properly). I think this should be ready for final review now -- thanks!
alexcrichton submitted PR review.
alexcrichton created PR review comment:
One thing I might recommend is to change the world itself for the debugger to just mention
wasi:io/pollperhaps? That way you don't have to deal with any of these other interfaces and it also wouldn't require so much vendored WIT. The debugger world would be a subset of the actual world but that's also relatively common and also accurate (e.g. this dosen't cover-Darg=-Shttpwhich enables the debugger to send http requests anyway)
alexcrichton created PR review comment:
Looking below it looks like the only true source of
.awaitis blocking on a pollable and otherwise nothing else is concurrent, so in the meantime it might be reasonable to skip thewstddependency, write this all with sync code, and usepollable.block_on()to do the awaiting?
alexcrichton created PR review comment:
Does this actually do anything? I figured that if the debugger is blocked then the guest is blocked as well, so doesn't the guest only actually execute code once it hits teh
wait()below? (which I suppose would be interrupted quickly with theinterrupt())
alexcrichton created PR review comment:
We need to sort this out at some point regardless because many of these tests should be built for
wasm32-wasip2instead ofwasm32-wasip1too... I suspect that this is required for thewstddependency, but if below I'm right that we can remove the dependency for now could you fold this back into the loop above? It's something we'll have to figure out at some point but I'd prefer to defer that to handling other wasm32-wasip2 parts too.
alexcrichton created PR review comment:
Can this file also get an addition of
foreach_debuggee!(...)similar to other "test groups" throughout the repo? The test-artifacts build should produce this macro to get invoked.Also I think it's fine to lift the
cfg(not(miri))annotation to this entire file to avoid dealing with any confusion there. (e.g.#![cfg(not(miri))]at the top)
alexcrichton created PR review comment:
Another way to put this, I probably have the wires crossed in my head for how this works, so can you remind me how this ends up working? For example how are the debugger and debugee running concurrently?
cfallin submitted PR review.
cfallin created PR review comment:
Both debugger and debuggee are running as async tasks in Tokio and so in theory the "continue" might not be processed until after we reach
interruptbelow -- I wanted to throw a delay in there to make it more likely to catch any issues with yields, etc. Said another way, without this, it's very unlikely we test the case we actually want to test.
cfallin submitted PR review.
cfallin created PR review comment:
I've got good news and bad news for you!
The good news is that I think you're right -- there's no need for
wstdin this PR directly.The bad news is that I'm about to change that workspace-wise in #12771, wherein the gdbstub component needs wstd for real (for TCP sockets and for a single guest-side executor that balances those with debugger pollables) so we'll need to vet it and take it as a dep in any case.
cfallin updated PR #12756.
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Published vets for
wstdand its transitive deps in d0027a6ef3.
cfallin submitted PR review.
cfallin created PR review comment:
Unfortunately I think we need almost everything -- I got this list by importing
wasi:cli/imports@0.2.6incrates/debugger/wit/world.wit, and then ended up with this (slightly absurd) list at least on the host side by chasing wit-bindgen errors and adding interfaces until it was happy. I don't think we can get away withoutwasi:cli/imports-- the point of the debug-main world is that we give the debugger component full CLI powers, including e.g. stdio and sockets. Note that wasi-http already is not included above so this is pretty "minimal", just explicit...
cfallin updated PR #12756.
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Given the above (wstd is required for the very next step anyway) I think I'd prefer testing that rather than have something in the test here that doesn't match the actual use-case -- ok if we keep wasm32-wasip2 here? Or some other refactor?
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Done!
cfallin updated PR #12756.
cfallin commented on PR #12756:
@alexcrichton in addition to addressing review feedback I tracked down a cancelation-safety issue in
EventFuture(reproducible locally withtaskset 0x1 cargo test ...I guess because it requires a certain amount of serialization causing theDynPollable::readyfuture to be canceled at least once; the CI machine has less parallelism and triggered this). See c39e049a1b24791205764a35205deed7772028f1. Thanks!
cfallin updated PR #12756.
cfallin commented on PR #12756:
Also a bugfix discovered via CI failure in the stacked PR here -- see 6252848fa87e336990e59e04e522c55d0969da9c.
alexcrichton created PR review comment:
In the interest of keeping this test debugger minimal, could it be removed from here? Make sense it'd be needed in gdbstub and I think that's fine, so it's ok to keep the vets here, but for the minimal testing here I figured it'd be nice to keep this as small as can be
alexcrichton created PR review comment:
Ah ok, I think I see. So the debuggee forcibly has a thread incrementing the epoch not only for requests to
interruptbut also for infinite loops in the debuggee to ensure that it's yielding eventually to the debugger in case the debugger's tokio task needs to run.It'll take me a bit to wrap my brain around it, but seems reasonable!
alexcrichton created PR review comment:
Oh sure yeah the gdbstub itself needs the full
wasi:cli/imports, but my thinking is that thedebugger/wit/world.witonly needsimport wasi:cli/poll@0.2.6and that's it (for thepollableresource). The actual gdbstub debugger would pull in more things (TCP, stdio, etc), but that's all transitively through other dependencies (e.g. wasi-libc, libstd, etc). By havingdebugger/wit/world.witbe a slimmed down version it maens that most of these annotations here wouldn't be necessary, and AFAIK nothing would break either.Put another way: by minimizing the
debugger/wit/world.witto its bare bones that shouldn't impact the functionality of gdbstub or any other bits of implementation anywhere, and what it'll do is make thebindgen!s here and on the host a bit easier since you don't have to worry about all the WASI interfaces unrelated to debugging
alexcrichton submitted PR review:
I'll admit I don't really understand what happened with the cancellation safety bit (it looked like it should work before), but happy to trust you on that it's fixing things
alexcrichton created PR review comment:
FWIW with the use of a closure below I think this is a dead trait
cfallin created PR review comment:
Ah, right, removed; thanks.
cfallin submitted PR review.
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Sure, removed wstd in favor of blocking directly on the
EventFutures (viaevent-future.finish) for now.
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
Ah, got it -- I was missing that we could actually just not include most of wasip2 in the debug-main world definition, but we can still provide the imports in Wasmtime's environment and that works fine. "Mix-in compositionality" of worlds still bends my brain a little but this makes sense. Thanks!
cfallin added PR #12756 Debugging: add the debug-main world. to the merge queue
cfallin removed PR #12756 Debugging: add the debug-main world. from the merge queue
cfallin updated PR #12756.
cfallin submitted PR review.
cfallin created PR review comment:
OK, addressed in d7aa6a2e28.
cfallin has enabled auto merge for PR #12756.
cfallin added PR #12756 Debugging: add the debug-main world. to the merge queue
cfallin merged PR #12756.
cfallin removed PR #12756 Debugging: add the debug-main world. from the merge queue
Last updated: Mar 23 2026 at 16:19 UTC