Steve Williams (Oct 15 2021 at 13:39): Steve Williams (Oct 15 2021 at 13:39):so, from dwarf dump., our test float is at 0x26 which is here :
translations[0x00000000].debuginfo.dwarf.debug_info.debug_info_section.slice[0x00000026]
quick inspect of that & it looks right. might hardwire pull that one through as a means of figuring out what issues we have before nailing it more generically.
Steve Williams (Oct 15 2021 at 13:39):0x00000026: DW_TAG_variable [2]
DW_AT_name [DW_FORM_strp] ( .debug_str[0x00000078] = "my_global")
DW_AT_type [DW_FORM_ref4] (cu + 0x0037 => {0x00000037} "float")
DW_AT_external [DW_FORM_flag_present] (true)
DW_AT_decl_file [DW_FORM_data1] ("g:\dev\test\.\test.c")
DW_AT_decl_line [DW_FORM_data1] (1)
DW_AT_location [DW_FORM_exprloc] (DW_OP_addr 0x400)
Steve Williams (Oct 15 2021 at 13:47):so we just verbatim dump that out & guess ends up at same address (0x400), which can inspect once it shows up in debugger.
if that's wrong, second pass over it to figure out where it gets mapped to.
Steve Williams (Oct 15 2021 at 13:49):& can run
float *wheres_my_global = &my_global;
... in the C. to check / figure out where it is for verification.
Steve Williams (Oct 17 2021 at 12:12):Some code at last :)
Just grabs the globals thus far. Not suggesting a merge yet unless you want to help deliver this.
Steve Williams (Oct 17 2021 at 12:13):Need that & the toml in that module to add gimli dependency.
Steve Williams (Oct 17 2021 at 12:24):its not a straight verbatim dump as you have to grab the string identifier & add it to the destination string table, amending index accordingly.
Steve Williams (Oct 17 2021 at 12:24):& whatever needs doing with the address.
Steve Williams (Oct 18 2021 at 09:42):looking at clone_unit in unit.rs - does most of what we want, just for functions, not global variables.
Steve Williams (Oct 18 2021 at 12:10):looks like finish_compile is where we should emit this stuff.
Steve Williams (Oct 18 2021 at 12:38):going just above that as too much rust weirdness to resolve dropping it in there. still not a fan. bit too "clever".
Steve Williams (Oct 18 2021 at 13:27):moved it here.
Steve Williams (Oct 18 2021 at 15:06):clone_unit - clones a compile unit. each compile unit (equivalent of a source file) holds its own globals
that calls clone_die_attributes. unsure why we don't just get globals.
processing ...
Steve Williams (Oct 18 2021 at 15:51):clone_die_attributes is parsing the globals. so we don't get them because they're either filtered out or broken in some way.
need to visualize what we're actually getting for insight.
Steve Williams (Oct 18 2021 at 16:19):pending_die_refs & pending_di_refs from clone_die_attributes should contain our global.
Steve Williams (Oct 18 2021 at 16:28):Here's where we come out the other side after processing the die.
https://github.com/adv-sw/wasmtime/blob/95efcd83e0b3865df4306ee65e6a75b25f20bf21/crates/cranelift/src/debug/transform/unit.rs#L374
Sub-programs - processed.
Global variable isn't a DW_TAG_subprogram - ends up doing nothing here :
https://github.com/adv-sw/wasmtime/blob/95efcd83e0b3865df4306ee65e6a75b25f20bf21/crates/cranelift/src/debug/transform/unit.rs#L399
Steve Williams (Oct 18 2021 at 16:32):& that's all recursive. fun times :)
Steve Williams (Oct 18 2021 at 16:43):then into generate_simulated_dwarf where I can see what it gets. can quit whining now & just work thru it. sorry all :)
Steve Williams (Oct 18 2021 at 16:44):think it just needs peppering with diagnostics until its obvious what's missing, or if nothing's missing look elsewhere for clues.
Steve Williams (Oct 18 2021 at 17:52):generate_simulated_dwarf is writing header then functions. so gotta figure out how to pull globals into that.
Steve Williams (Oct 18 2021 at 18:55):I also don't want to work on this but the fundamentals need to be solid for this to be an effective internet scale project.
Steve Williams (Oct 18 2021 at 18:55):windows users are used to quality debugging experiences. others seem to settle for less. lets get it right :)
Steve Williams (Oct 18 2021 at 19:15):summary: cranelift bug.
Steve Williams (Oct 18 2021 at 19:39):Added
log_globals(&di)?;
to start of generate_simulated_dwarf.
log_globals defined below. seems to indicate global coming thru, so that function at fault with valid data. I say valid, its there, whether correct or not, don't yet know.
fn log_globals(
di: &DebugInfoData,
) -> Result<i32, Error>
{
// Scan .debuginfo for global variables which we'll record & translate to JIT sink.
let mut count : i32 = 0;
let dwarf = &di.dwarf;
let mut iter = dwarf.units();
// Iterate over the compilation units.
while let Some(header) = iter.next()? {
println!(
"Unit at <.debug_info+0x{:x}>",
header.offset().as_debug_info_offset().unwrap().0
);
let unit = dwarf.unit(header)?;
// Iterate over the Debugging Information Entries (DIEs) in the unit.
let mut depth = 0;
let mut entries = unit.entries();
while let Some((delta_depth, entry)) = entries.next_dfs()? {
depth += delta_depth;
if entry.tag() == gimli::DW_TAG_variable && depth == 1
{
count = count + 1;
println!("GLOBAL_{} <0x{:x}> {}", count, entry.offset().0, entry.tag());
// Iterate over the attributes in the DIE.
let mut attrs = entry.attrs();
while let Some(attr) = attrs.next()? {
println!(" {}: {:?}", attr.name(), attr.value());
}
}
}
}
Ok(count)
}```
think we need to create a gimli unit with a null line program, implicitly globals unit. tried that, needs gimli mods so just figured out how to override with local crate. compiles, so into it. not the ideal first rust project, but hey :)
one trivial fix to run with gimli trunk.
Steve Williams (Oct 19 2021 at 16:18):is a nice structure for collab work & the syntax isn't terrible. might even start to like it.
Steve Williams (Oct 21 2021 at 10:24):Early approximation - work in progress : https://github.com/adv-sw/wasmtime
clone_globals in wasmtime\crates\cranelift\src\debug\transform\unit.rs
Applied in wasmtime\crates\cranelift\src\debug\transform\mod.rs
Traverses cleanly, not yet working code. Got some placeholders in there whilst we figure out what goes where.
Corrections / refinements welcome.
Steve Williams (Oct 21 2021 at 10:30):Requires some minor gimli tweaks : https://github.com/adv-sw/gimli
Steve Williams (Oct 21 2021 at 13:34):dropping log_globals back into generate_simulated_dwarf, we get one - so +ve change in behaviour.
next to decode that as it comes out to figure out if its correct.
Steve Williams (Oct 22 2021 at 10:59):Extracted generated ELF that's sent to LLDB to a file for inspection via the following.
fn register_debug_and_profiling(&mut self, profiler: &dyn ProfilingAgent) -> Result<()> {
// Register GDB JIT images; initialize profiler and load the wasm module.
if self.meta.native_debug_info_present {
let code = self.code();
let bytes = create_gdbjit_image(self.mmap().to_vec(), (code.as_ptr(), code.len()))
.map_err(SetupError::DebugInfo)?;
{
// ADV_SW_DEBUG : save for inspection.
let path: &Path = Path::new("g:\\dev\\test\\emit.elf");
fs::write(path, &bytes).unwrap();
// ADV_SW_DEBUG[END]
}
profiler.module_load(self, Some(&bytes));
Looks less wrong than I expected, but not quite there yet.
Steve Williams (Oct 22 2021 at 11:57):with my nonsense turned off, the global does appear to come thru.
https://pastebin.com/ZsTK8De1
should have done this bit first :)
... but we don't see it. so another verify on that then into lldb to see why it's not registered.
Steve Williams (Oct 22 2021 at 12:35):comparing emitted dwarf vs. native equivalent, we spot some differences.
native
< 1><0x0000002a> DW_TAG_variable
DW_AT_name my_global
DW_AT_type <0x0000003f>
DW_AT_external yes(1)
DW_AT_decl_file 0x00000001 /media/x/DEV_LLVM/dev/test/./test.c
DW_AT_decl_line 0x00000001
DW_AT_location len 0x0009: 0x032840400000000000:
DW_OP_addr 0x00404028
wasmtime emit
< 1><0x00000073> DW_TAG_variable
DW_AT_name my_global
DW_AT_type <0x0000002f>
DW_AT_external yes(1)
DW_AT_decl_file 0x00000001
DW_AT_decl_line 0x00000001
Steve Williams (Oct 22 2021 at 12:45):specifically, DW_AT_location isn't coming thru, so debugger doesn't know where it is & that's likely the reason for reject.
Steve Williams (Oct 22 2021 at 14:11):that DW_AT_location is skipped bcoz hitting "// FIXME _expr contains invalid expression"
in wasmtime\crates\cranelift\src\debug\transform\attr.rs
Steve Williams (Oct 22 2021 at 14:27):operation Address is causing compile_expression to fail. have no idea what should be going on in there :)
Steve Williams (Oct 22 2021 at 14:39):not having a clue what I'm doing in there, I forced Address to pass & got the desired attribute
DW_TAG_variable
DW_AT_name ("my_global")
DW_AT_external (0x01)
DW_AT_decl_file ("g:\dev\test\.\test.c")
DW_AT_decl_line (1)
DW_AT_location (DW_OP_addr 0x0, <decoding error> 00 04 00 00)
DW_AT_type (0x0000002f "float")
value is incorrect but its there. next to figure out what its supposed to be & how to get it.
Steve Williams (Oct 22 2021 at 14:47):DW_AT_location does come through for locals, which will be on the stack rather than fixed location, but perhaps can look at what's going on there for clues.
Steve Williams (Oct 22 2021 at 16:33):trying this in attr.rs
// ADV_SW_PATCH: DW_AT_location is defined by global variables.
// We process directly as compile_expression doesn't like them for some reason.
AttributeValue::Exprloc(ref expr) if attr.name() == gimli::DW_AT_location => {
let buf = expr.0.to_slice()?;
// TODO1: Verify buf[0] == DW_OP_addr
// TODO2: This is for 32 bit wasm pointers. How to detect 64 bit & react accordingly ?
// TODO3: Consider endian requirements.
let u = u32::from_le_bytes([buf[1],buf[2],buf[3],buf[4]]) as u64;
let addr = addr_tr.translate(u).unwrap_or(write::Address::Constant(0));
write::AttributeValue::Address(addr)
}
coming out as zero address, though 0x0040000 in the wasm.
how should I look up mapping from global's local address to target address space, please.
Steve Williams (Oct 22 2021 at 16:44):wasmtime\crates\cranelift\src\debug\transform\attr.rs
Steve Williams (Oct 22 2021 at 16:55):feel free to steal the thunder if you've got it from here :)
Steve Williams (Oct 22 2021 at 16:57):... but that's the issue - at least one of them. might be multiple faults.
global variable DW_AT_location address locations are being rejected by compile_expression in trunk.
Steve Williams (Oct 22 2021 at 17:07):that mapping is what resolve_vmctx_memory does. so figure out how to run through that code, I guess.
Steve Williams (Oct 22 2021 at 17:08):not sure globals end up in address translation table. hence why we're getting 0. so compute directly.
Steve Williams (Oct 23 2021 at 09:28):a command line option for wasmtime app & API to copy debug mode generated elf to a file for inspection once resolved so easier next time, perhaps.
Steve Williams (Oct 23 2021 at 09:36):Please could master update to latest gimli. Requires stub for DebugCuIndex, DebugTuIndex
Steve Williams (Oct 23 2021 at 09:39): DebugCuIndex => ".debug_cu_index",
DebugTuIndex => ".debug_tu_index",
guess. something like that.
Steve Williams (Oct 23 2021 at 09:50):trunk broken for me running globals example from wasmtime rust sdk, with debug mode enabled.
let mut config = wasmtime::Config::new();
config.debug_info(true);
let engine = wasmtime::Engine::new(&config)?;
& slightly modified gimli. perhaps its that - nope, dropped regular in everywhere. still broken. back to my branch.
Steve Williams (Oct 23 2021 at 10:18):regression redacted. programming error in my new code causing panic. doh :)
Steve Williams (Oct 23 2021 at 11:35):corrected that coding error. still haven't translated the address, but get wasm address coming thru (insufficient, but progress)
& write_expr_addr in expression.rs
Steve Williams (Oct 23 2021 at 14:07):still haven't found the magic source to convert wasm pointer to runtime equivalent before instantiation.
found translate_raw but that just performs a function lookup which will fail coz not a function.
Steve Williams (Oct 23 2021 at 17:23):how do I grab raw address from Address please.
pub fn write_addr(&mut self, addr: gimli::write::Address) {
let addr64 = addr as u64;
let bytes = addr64.to_le_bytes();
self.write_u8(bytes[0]);
self.write_u8(bytes[1]);
self.write_u8(bytes[2]);
self.write_u8(bytes[3]);
self.write_u8(bytes[4]);
self.write_u8(bytes[5]);
self.write_u8(bytes[6]);
self.write_u8(bytes[7]);
}
can't do as u64 or anything else I've tried. stuck on simple stuff.
Steve Williams (Oct 23 2021 at 17:40):I've got it generating an address. I can't figure out how to get the correct address or how to write it when I get it.
clues are welcome.
Steve Williams (Oct 23 2021 at 17:51):there's this thing, but can't figure out how to use it.
write_address(&mut self, address: Address, size: u8)
if we were C, I'd just cast it or at the least be able to see what it is in debugger.
Steve Williams (Oct 23 2021 at 17:54):as I don't know how to transform an address, unsure if figuring out how to serialize one of those things is way to go anyway.
Steve Williams (Oct 23 2021 at 18:10):finally :)
pub fn write_expr_addr(addr : gimli::write::Address)-> gimli::write::Expression
{
let mut w = ExpressionWriter::new();
w.write_op(gimli::constants::DW_OP_addr);
match addr {
gimli::write::Address::Constant(val) => {
let bytes = val.to_le_bytes();
w.write_u8(bytes[0]);
w.write_u8(bytes[1]);
w.write_u8(bytes[2]);
w.write_u8(bytes[3]);
w.write_u8(bytes[4]);
w.write_u8(bytes[5]);
w.write_u8(bytes[6]);
w.write_u8(bytes[7]);
},
gimli::write::Address::Symbol { symbol, addend } => { /*not yet supported*/ }
}
gimli::write::Expression::raw(w.into_vec())
}
Got this, which at least serializes an Address now. Coming thru zero. Unsure how I should convert the address of a wasm global variable to target equivalent.
// ADV_SW_PATCH: DW_AT_location is defined by global variables.
// We process directly as compile_expression doesn't like them for some reason.
AttributeValue::Exprloc(ref expr) if attr.name() == gimli::DW_AT_location => {
let buf = expr.0.to_slice()?;
// Ignore unless supported op.
if buf[0] != (gimli::DW_OP_addr.0 as u8) { continue; }
// TODO: Detect 64 bit wasm & form source address accordingly.
let addr_wasm = u32::from_le_bytes([buf[1],buf[2],buf[3],buf[4]]) as u64;
// TODO: Translate the address.
// TODO2: Ensure endian correctness - this is a target pointer.
let addr_target = addr_tr.translate(addr_wasm).unwrap_or(write::Address::Constant(0));
// let addr_target = write::Address::Constant(0);
// TODO: Research use of the following - might be required.
// resume_with_relocated_address
// evaluate_one_operation
// op_wasm_global
let expr = write_expr_addr(addr_target);
write::AttributeValue::Exprloc(expr)
}
assuming no other issues, this is nailed once we have the correct address.
Steve Williams (Oct 24 2021 at 09:50):hmmm ... so that code is getting called by build_artifacts whilst creating a module. don't think we can get final address there. so need to figure out a way to park the expression & evaluate it in JIT callback instead of as module is built.
Steve Williams (Oct 24 2021 at 10:17):create_gdbjit_image does some kind of a relocate, so need to get head around that.
Steve Williams (Oct 24 2021 at 12:34):first call might result in invalid pointer if we don't have memory yet, but construction of dwarf is from jit callback, at a breakpoint, we will have an instance, so can form a valid target pointer.
Steve Williams (Oct 24 2021 at 14:27):ummm from z:\dev\wasm\wasmtime\crates\cranelift\src\debug\transform\attr.rs
want
use crate::wasmtime_runtime::{debug_builtins};
....
// so I can do something along lines of :
let addr_target = resolve_vmctx_memory(addr_wasm);
... which is unsafe & don't know how to resolve that either coz rust noob.
Steve Williams (Oct 24 2021 at 14:28):someone who knows how to drive would have this done by now :)
Steve Williams (Oct 24 2021 at 16:32):following example has 2 float global variables - their DW_AT_locations look fine. the 3 means op addresss.
this log from running under lldb at a breakpoint.
the addresses I need to remap are processed before the memory is initialized. so its not possible at that point.
& here, I think is where the __vmctx->set() thingy comes in. so gotta figure out how this all fits in there.
aside: if someone could explain what a trampoline is, that'd help. might not need one, but would like to understand what they are so I know that. they seem to be something to do with functions but this is a global not a function.
[global_requiring_target_resolve]
[global_requiring_target_resolve]
Unit at <.debug_info+0x0>
GLOBAL_1 <0x26> DW_TAG_variable
DW_AT_name: DebugStrRef(DebugStrOffset(120))
DW_AT_type: UnitRef(UnitOffset(55))
DW_AT_external: Flag(true)
DW_AT_decl_file: FileIndex(1)
DW_AT_decl_line: Udata(1)
DW_AT_location: Exprloc(Expression(EndianSlice { slice: [3, 0, 4, 0, 0], endian: LittleEndian }))
GLOBAL_2 <0x3e> DW_TAG_variable
DW_AT_name: DebugStrRef(DebugStrOffset(136))
DW_AT_type: UnitRef(UnitOffset(55))
DW_AT_external: Flag(true)
DW_AT_decl_file: FileIndex(1)
DW_AT_decl_line: Udata(2)
DW_AT_location: Exprloc(Expression(EndianSlice { slice: [3, 4, 4, 0, 0], endian: LittleEndian }))
[elf_emit]
[initialize_memories]
Steve Williams (Oct 24 2021 at 16:35):we're close :)
(lldb) p my_global
error: Couldn't materialize: couldn't get the value of variable my_global: read memory from 0x0 failed (0 of 4 bytes read)
that's expected, it is set at 0x0 right now. we can finally "see" the global in lldb.
Steve Williams (Oct 24 2021 at 16:38):(lldb) fr v -g -l
(float) my_other_global = <read memory from 0x0 failed (0 of 4 bytes read)>
(float) my_global = <read memory from 0x0 failed (0 of 4 bytes read)>
that's new. got nothing previously.
just that remapping to nail somehow.
Steve Williams (Oct 24 2021 at 17:03):we can't encode as a DW_OP_addr as we don't know the target address at the point of query. hence looking at DW_OP_call_ref which on initial reading might work. have no clue how to use it, so figuring that out is next.
Steve Williams (Oct 24 2021 at 17:14):DW_OP_call_ref (0x9a) : 1 operand : a 4- or 8-byte offset of DIE.
that'll do. can encode it with the wasm pointer then run whatever function it wants to decode it.
don't of course have a clue how to specify the helper function.
Steve Williams (Oct 24 2021 at 17:15):that's from dwarf4 spec, p167
Steve Williams (Oct 24 2021 at 17:17):p24 : DW_OP_call_ref performs subroutine calls during evaluation of a DWARF expression or location description
Steve Williams (Oct 24 2021 at 17:19):"The operand is used as the offset of a debugging information entry in a .debug_info or .debug_types section which may be contained in a shared object or executable other than that containing the operator."
nope, that's not it :)
Steve Williams (Oct 24 2021 at 17:39):DW_OP_push_object_address perhaps. if I can push wasm addr on the stack & get that as parameter to function in OP_call_ref, that'd do.
Steve Williams (Oct 24 2021 at 17:43):from the spec : "Values on the stack at the time of the call may be used as parameters by the called expression and values left on the stack by the called expression may be used as return values by prior agreement between the calling and called expressions"
might even work.
Steve Williams (Oct 24 2021 at 18:08):Got it :)
* thread #1, stop reason = breakpoint 1.1
frame #0: 0x0000017f1d1b127f JIT(0x17f1b52f6d0) main at test.c:11:12
8
9 int main() {
10 int i = my_global + my_other_global;
-> 11 return i;
12 }
(lldb) fr v -g -l
(float) my_other_global = <read memory from 0x404 failed (0 of 4 bytes read)>
(float) my_global = <read memory from 0x400 failed (0 of 4 bytes read)>
(lldb) p __vmctx->set()
(lldb) p *(float*) resolve_vmctx_memory(0x400)
(float) $1 = 5
decided to just pass the wasm pointer & fix it on the other side.
can hide the pointer ops behind the lldbg gui.
unsure if its safe to resolve like this.
perhaps we do need to use an evaluated expression to present the correct target address .
downside of that is the resolved address will be visible & we don't want to show that.
Steve Williams (Oct 24 2021 at 18:22):global variables delivered to JIT debug interface - going with that if it's stable, which I'll figure out when I try it on something more complex.
Steve Williams (Oct 24 2021 at 18:33): Steve Williams (Oct 25 2021 at 09:16):Perhaps extending DWARF format (& lldb) by adding support for virtual addresses is the way to do this cleanly.
New DW_AT_location op DW_OP_vaddr (virtual address) which lldb could resolve by calling a method in some die like we currently do manually with resolve_vmctx_memory.
That's more work than I can schedule right now, but the alternative is excessive complexity to work without it.
Steve Williams (Oct 25 2021 at 11:19):rust is growing on me. steep learning curve, but impressive ecosystem. apologies for foul mood - had to get this done as spotted a gap. can take a chill pill now & just work through it as & when. wasmtime/cranelift crazy tech too & I've only scratched the surface - great work all, appreciated.
with greatest respect, a few more comments in the code though, plz :)
it aids understanding & that aids tracking down bugs faster.
apologies to any I've offended by going at this harder than was perhaps necessary.
Steve Williams (Oct 25 2021 at 11:37):done, I think. at least first pass solution. apologies for pushing too hard. I wanted it fixed, so my problem, nobody else's.
some experimentation & tidy up, then happy to land this if it proves to be solid. concerned a virtual address could clash with a genuine system address & thus cause lldb to interpret incorrectly. ensuring sandbox memory ptr is always > 4GB address resolves that 32 bit wasmtime.
best solution is implement virtual addresses in DWARF spec & lldb then everything is clear & well structured. it'll happen at some point unless there's an objection.
Steve Williams (Oct 25 2021 at 16:46):
that's better :)
still optimizing out all my locals but that's the compiler. almost sane.
Steve Williams (Oct 25 2021 at 17:52):the above 4GB memory pool alloc for virtual address safety point above is incorrect. we have no idea if virtual addresses also represent a valid system address. hence lldb tweak once this is solid one way or another. politics of getting it in will be more hassle than the technical work, but can patch independently for our use. so can be done.
Last updated: Dec 23 2024 at 12:05 UTC