Stream: wasmtime

Topic: debugger hitting breakpoint behaviour changed ?

Steve Williams (Apr 28 2023 at 08:58):

Hi guys, just upgraded our stuff to latest wasmtime & seems breakpoint behaviour has changed.

Previously got lldb::eStopReasonBreakpoint event on a breakpoint.
Now apparently lldb::eStopReasonException

Does anyone know anything about this, please. Is it intentional ? A regression of some kind ?
Do I have something setup wrong ?

Haven't worked on this code for quite some time so very rusty & may be user error.

Steve Williams (Apr 28 2023 at 09:17):

Rebuilt my previous (working) wasmtime from scratch on same system to attempt to determine whether something else on my system responsible for regression. Triggers breakpoint as expected.

Steve Williams (Apr 28 2023 at 09:36):

reviewing previous releases - the one I was working from was pre-1.0 ... to attempt to figure out where the regression/change occured.

Steve Williams (Apr 28 2023 at 09:45):

weird. 1.0.1 also generates an exception - so change isnt recent.

Steve Williams (Apr 28 2023 at 10:02):

fresh pull of 0.3.0 which I was working from acting unexpectedly, so my issue for the time being.

Steve Williams (Apr 28 2023 at 12:48):

v0.31.0 triggers breakpoints correctly. v0.32.0, the next tagged release triggers an exception & subsequent releases trigger exceptions.

Steve Williams (Apr 28 2023 at 12:51):

[ above should have said 0.30.0 not 0.3.0 but in any case, that release is irrelevant - the change in behaviour occurs between 0.31.0 & 0.32.0 ]

Steve Williams (Apr 28 2023 at 14:10):

Seems this is the cause of the regression :

Revision: ce67e7fcd1d2d6da1899d2b46cc41ef877bd9462
Author: Amanieu d'Antras <amanieu@gmail.com>
Date: 02/12/2021 11:53:04
Message:
Fix ownership in *_vec_new functions in the C API

These functions are specified to take ownership of the objects in the given slice, not clone them.

Modified: crates/c-api/src/vec.rs

Steve Williams (Apr 28 2023 at 14:27):

yup, replacing those couple of lines with previous implementation restores Breakpoint behaviour (minimal testing thus far).
next will try migrating that fix to trunk to see if that resolves regression there.
beyond that, this is rust code I have no clue about so will have to defer to others for complete resolution.
I'm going as far as just get it working again.

Steve Williams (Apr 28 2023 at 14:57):

yup, revert of that change to trunk gets breakpoints running again. yay :)

Steve Williams (Apr 28 2023 at 14:58):

needs a little more testing, then will push if wanted.

Steve Williams (Apr 28 2023 at 15:01):

as to the clone/ownership issue the change was supposed to address, its implications & why it generates an exception, dunno.
I need working recent version. this can be considered in more depth later. worked before that change, works with it removed. what it's supposed to do needs further consideration to avoid debugger breakpoint regression.

Steve Williams (Apr 28 2023 at 15:06):

likely a duplicate release of something if was assumed cloned before & behaviour changed to ownership change. so could be fixed properly by tracking that down, but beyond scope of my current work.

Steve Williams (Apr 28 2023 at 15:52):

that works for me :) & gets this running wasmtime trunk.
advance-software.com/develop

Steve Williams (Apr 28 2023 at 16:00):

I dont have permission to push apparently so here's the change in its current form.

https://pastebin.com/2Yt3FG9E

Steve Williams (Apr 28 2023 at 17:21):

Referenced here, as perhaps the same issue : https://github.com/bytecodealliance/wasmtime/issues/3999

bjorn3 (Apr 28 2023 at 17:33):

If you run valgrind on your program built against the latest wasmtime version without your patch does it give any errors?

Steve Williams (Apr 28 2023 at 18:02):

Hi Bjorn, can't run valgrind as windows user but could run drmemory. cant go much deeper as we're severely resource constrained so interaction must be limited to fire fighting for now. fire out - debugging working again.

concerned this has regressed for 17 months now. does nobody else use a debugger ? do we have a test that ensures debugger breakpoints are emitted properly ?

link with issue 3999 perhaps false but the issue itself genuine & corrected with this patch.

Steve Williams (Apr 28 2023 at 18:12):

let me see if I can get some useful diagnostics out of this.

Jamey Sharp (Apr 28 2023 at 18:31):

What would be most helpful is a minimal test program demonstrating the issue. As far as I know, Wasmtime doesn't have anything to do with emitting breakpoints, so I don't understand what behavior you were expecting.

Steve Williams (Apr 28 2023 at 18:38):

Hi Jamey,

Ran lldbg under visual studio debugger & used it to identify the regression & locate when the code regressed. That's all I've got for now. Its using some lldb API I don't fully understand yet as inherited code. Perhaps as we have a behaviour change, the issue causing the exception is in the app. Will investigate further. Will see if I can get lldb command line version to exhibit same behaviour.

My version : https://github.com/adv-sw/lldbg

Steve Williams (Apr 28 2023 at 18:40):

With above fix, the following is hit : https://github.com/adv-sw/lldbg/blob/master/src/Application.cpp#L2127

without it, get an exception. That's all I currently know.

Steve Williams (Apr 28 2023 at 18:41):

Specifically a lldb::eStopReasonException which isn't handled by the above code, but I can see it in the Visual Studio debugger.

Looks like will need a debug build of lldb to figure out why that's getting emitted.

Steve Williams (Apr 29 2023 at 09:55):

I'll investigate this a little but with respect, its not my problem. The patch caused a regression in a key component - lldb debugger.
The patch should be backed out as debugger was functioning perfectly well before it was introduced and does so again with it removed.

Patch should be re-accepted when it doesnt cause failure side effect. Fixing may be non-trivial & production work needs to proceed.

Steve Williams (Apr 29 2023 at 10:09):

An issue of contention may be, is lldbg using lldb API correctly. perhaps there is a programming error there. dont know at this point if fault is exhibited by lldb itself. will attempt to repro there when I get a minute.

bjorn3 (Apr 29 2023 at 10:12):

Are you trying to reuse any of the objects passed to wasm_*_vec_new? If so make sure to clone them. Accoring to the commit message it was documented that wasm_*_vec_new is taking ownership, but the implementation incorrectly cloned them instead. That commit fixes the discrepancy by changing the implementation to take ownership as documented.

bjorn3 (Apr 29 2023 at 10:13):

See https://github.com/bytecodealliance/wasmtime/pull/3582

Steve Williams (Apr 29 2023 at 10:14):

lldbg is an inherited project. I didnt write it. its using lldb interface. like I say, its possible there's an error in api use. unknown at this point.

bjorn3 (Apr 29 2023 at 10:14):

I meant the use of wasm_*_vec_new in the code you wrote, not in the debugger.

bjorn3 (Apr 29 2023 at 10:16):

Note that the wasm_* api's are implemented not just by wasmtime, but also by wasmer and some other wasm engines. Before the commit in question wasmtime's implementation didn't match the other implementations.

Steve Williams (Apr 29 2023 at 10:18):

will simplify repro as far as possible so we can eliminate or otherwise that concern. thanks.

Steve Williams (Apr 30 2023 at 07:07):

building debug lldb is taking forever. left it going overnight & still hadnt linked in the morning. trying on ms azure cloud thing.
if I'm to undertake heavy development work, I need a machine that can handle it. happy to contribute but not a magician :)

Steve Williams (Apr 30 2023 at 07:08):

my quick fix is the patch I provided which is sufficient for our purposes for the time being. this is going beyond my needs.

Steve Williams (Apr 30 2023 at 08:44):

I'm running a very old version of wasi. could be that's the issue if nobody else experiencing a problem. will upgrade when I can figure out where to obtain latest wasi from.

Steve Williams (Apr 30 2023 at 08:54):

from this : https://docs.wasmtime.dev/wasm-c.html

I get latest wasi by building llvm/clang for webassembly target ?

Steve Williams (Apr 30 2023 at 09:02):

https://github.com/WebAssembly/wasi-sdk/releases

apologies, going a little all over the place at the moment getting up to date.

Steve Williams (Apr 30 2023 at 09:51):

built lldb & clang in azure cloud as my local machine apparently has issues. debug build completed. building release version too so we have clang tools latest for test compilation. slow progress, but progress. switched to wasi 20 & compiled a simple test successfully.

Steve Williams (Apr 30 2023 at 13:49):

doh. yesterday's local lldb build failed coz I over compensated for an out of memory error & made a huge swap file on that drive, leaving insufficient room for build directory, so its stalled during linking :smile:

Steve Williams (May 01 2023 at 09:05):

update: our stuff running latest wasmtime, latest wasi, latest llvm. now we're up to date, will investigate cause of the exception.

Steve Williams (May 01 2023 at 16:59):

latest everything same behaviour as previous, with patch, works, without exception.

Steve Williams (May 01 2023 at 17:35):

Finally, cause of that exception.

image.png

Steve Williams (May 01 2023 at 17:38):

dinner then will try to get my head around why its doing that.

Steve Williams (May 01 2023 at 18:01):

Error is : "RVA 0x0 for export ordinal table not found"

Currently means nothing to me. Dont yet understand what the problem is.

Steve Williams (May 01 2023 at 18:04):

not understanding why an ownership change rather than clone should cause that. its not a duplicate release issue as I first thought.

Steve Williams (May 01 2023 at 18:06):

next step, I think is to simplify the repro.

Jamey Sharp (May 01 2023 at 18:17):

I see Microsoft has documentation on the "export ordinal table" as part of their PE format documentation, if that helps any: https://learn.microsoft.com/en-us/windows/win32/debug/pe-format#export-ordinal-table

Steve Williams (May 02 2023 at 05:28):

So, the issue appears to be in liblldb.dll!ObjectFilePECOFF::AppendFromExportTable which is PE specific, which is Windows platform.

I had to patch lldb years ago to get it running on Windows. The patch wasnt accepted as they require a test for every patch & I'm swamped with work as it is. Practical upshot: Fairly sure nobody else is testing on windows.

Here's the patch : https://pastebin.com/L1Q1d4hS

Patch works latest lldb - 16.0.2

Not really sure what test can be provided to verify it works & hence why not accepted yet.

Steve Williams (May 02 2023 at 05:42):

On Windows, it appears you need to build lldb with x86 as well as web assembly target for it to function.

I'm using

cmake -S llvm -DLLVM_TARGETS_TO_BUILD="X86;WebAssembly" -DCMAKE_BUILD_TYPE="Debug" -DLLVM_ENABLE_PROJECTS="clang;lldb;lld" -B build_d -G Ninja

Then : ninja lldb

From the build_d directory.

Steve Williams (May 02 2023 at 05:44):

Seem to remember there were issues with debugging native windows binaries last time I looked at this & perhaps still now, though all this is irrelevant. We're not trying to do that. We're trying to figure out why the change of a memory op in wasmtime from clone to re-assign owner causes a fault in the debugger.

Steve Williams (May 02 2023 at 10:18):

update: compiled the simplest c program possible :

void _start() {}

ran directly under wasmtime.exe rather than our runtime, under lldbg debugger. runs clean.

so my bug. next steps - reduce our stuff to the point where it no longer faults to identify cause.

Ralph (May 02 2023 at 23:59):

sounds to me that I should try to find someone to have a look here.

Steve Williams (May 03 2023 at 06:08):

working on a minimal repro. should have something soon.

Steve Williams (May 03 2023 at 06:12):

the lldb patch is something I need help with. confirm resolves lldb functionality on windows & accept into trunk. can maintain out of tree but seems silly. helps everyone.

Steve Williams (May 03 2023 at 08:13):

windows support obv not important for server based delivery but is for client front end (web browsers, etc.), coz like it or not, windows is the primary end user platform. so we must be solid there.

Steve Williams (May 03 2023 at 16:48):

update: got simple test case. test case running standalone works correctly. test case running as plugin for our engine faults the same way our regular web assembly plugin faults. so its something configuration specific or perhaps a fault in our engine having a side effect.

fun times. will isolate.

Ralph (May 03 2023 at 17:10):

wheeeeeeeee

Steve Williams (May 04 2023 at 08:59):

hmmm ... suspect this is bug in our engine having a side effect & nothing for anyone else to worry about, but we'll see. narrowing in on it.

Ralph (May 04 2023 at 14:32):

You DID have my attention, but now I'm merely curious.

Steve Williams (May 04 2023 at 14:32):

think I've got something

http:/advance-software.com/misc/repro.zip

extracts into your [wasmtime]/examples directory

so you have [wasmtime]/examples/repro not [wasmtime]/examples/repro/repro

fault: on windows, when compiled as a win64 visual studio project using included project file, wasm_module_new fails.

when compiled using \repro\native\make_d or make_r works - console verison.

Steve Williams (May 04 2023 at 14:33):

this has been really fiddly to put together so still possibility of something silly on my part.

Steve Williams (May 04 2023 at 14:35):

for those not on windows or not wanting to download that large archive which includes full wasi sdk we're linking against & llvm compilation tools being used, here's the wasm native sdk code I'm using.

https://pastebin.com/R4JSYBPC

Steve Williams (May 04 2023 at 14:35):

so first step is quick skim over that anyone curious, plz. any obvious derp on my part, plz.

Steve Williams (May 04 2023 at 14:37):

happy to run it here so we can figure out why wasm_module_new fails & that's my next job. will figure it out all the way if I can.

Steve Williams (May 04 2023 at 14:40):

debug build of the native win64 .exe doesn't fail, release does which is always a fun situation.
they both compile to the same location as of that drop - should tweak that.

Steve Williams (May 04 2023 at 14:48):

update : http://advance-software.com/misc/repro_2.zip

exactly the same as the first drop except native win64 example links to repro_d.exe & repro_r.exe instead of both to repro.exe

Steve Williams (May 04 2023 at 14:49):

step 1 - am I doing anything obviously stupid.
step 2 - can anyone else reproduce the problem.

Steve Williams (May 04 2023 at 14:53):

repro doesn't contain or reference our engine so api use error, genuine wasmtime bug or I'm misunderstanding something.

Steve Williams (May 04 2023 at 15:10):

repro release build linking against wasmtime debug build fails so can step through wasmtime & hopefully understand why. more caffeine than a look at that.

Steve Williams (May 04 2023 at 15:20):

can step into it but I'm rust stupid so this will be fun :)

Steve Williams (May 04 2023 at 15:33):

debug native app, targeting debug wasmtime correctly hits this :
https://github.com/bytecodealliance/wasmtime/blob/4b3c50b147f9f96717becf44bcfcff2de9b06c68/crates/wasmtime/src/module.rs#L459

release native app, targeting debug wasmtime doesn't & fails.

Steve Williams (May 04 2023 at 15:39):

not sure this is the same issue. tried replacing the clone/ownership change to see if behaviour changes & it doesnt.
so I still perhaps have a stupid in here.

Steve Williams (May 04 2023 at 15:41):

though regardless, this is unexpected result as far as I can tell so suggest we nail whatever this is & onwards.

Steve Williams (May 04 2023 at 15:55):

native app release build failing here : https://github.com/bytecodealliance/wasmtime/blob/4b3c50b147f9f96717becf44bcfcff2de9b06c68/crates/wasmtime/src/module.rs#L590

Steve Williams (May 04 2023 at 16:06):

\wasmparser-0.103.0\src\parser.rs
pub fn parse_all(self, mut data: &[u8])

data is all 0s in the native app release build that's failing.

Steve Williams (May 04 2023 at 16:12):

this one is my derp. apologies. data is coming thru 0s release build. so test has an error.

Steve Williams (May 04 2023 at 16:25):

so, ummmm.

int count = fread(wasm.data, file_size,1, file);

fixed by doing this instead :
int count = fread(wasm.data, 1, file_size, file);

have no idea why that was failing release build.

so test fixed & currently working. which is good coz this wasnt a repro for the original problem.

Steve Williams (May 04 2023 at 16:26):

hence anyone looking at my repro, plz drop it for now. its not a repro for the problem we're looking at. was a test error.
now I have a sane test environment will see if I can repro the actual problem.

Steve Williams (May 04 2023 at 16:33):

& changed that to a size_t now its working

Steve Williams (May 04 2023 at 16:35):

doesnt require anyone else on this right now as we dont have a repro yet. will try again now test environment sane. update when I have something.

Steve Williams (May 05 2023 at 12:32):

Got it.

The call :

wasm_functype_new_1_1(val_i32, val_i32);

in : https://pastebin.com/ZGKuqzPt

faults with whatever trunk I'm running (will update so at tip), passes with my workaround patch : https://pastebin.com/2Yt3FG9E

Steve Williams (May 05 2023 at 12:33):

don't think its specific to the wasm I'm loading but we'll see. will reduce to minimal example & move to tip.

Steve Williams (May 05 2023 at 12:57):

fault exists wasmtime tip

Steve Williams (May 05 2023 at 13:06):

minimal repro.

this faults for me wasmtime trunk.

https://pastebin.com/m2KNqPEK

Steve Williams (May 05 2023 at 13:07):

windows - dunno about other o/s.

Steve Williams (May 05 2023 at 13:09):

passes with my workaround patch : https://pastebin.com/2Yt3FG9E

Steve Williams (May 05 2023 at 13:26):

this thread far more verbose than it needs to be. apologies.

reduced to the following hopefully actionable issue : https://github.com/bytecodealliance/wasmtime/issues/6347

Steve Williams (May 05 2023 at 13:49):

thanks bjorn3 :)

weird api tripped me up, apparently. will rework as advised. excess allocation that way, but if its the spec, its the spec :)

Steve Williams (May 05 2023 at 13:55):

perhaps debug build of wasmtime could fail gracefully in that api misuse case as its quite unintuitive.

Steve Williams (May 05 2023 at 13:55):

you'd think what's effectively a constant could be reused multiple times.

Steve Williams (May 05 2023 at 13:59):

so we never call wasm_valtype_delete ? faults when I do so assume not, so seems odd that's an api function.

Steve Williams (May 05 2023 at 14:34):

resolved. thanks guys. weird api but is what it is.

Steve Williams (May 05 2023 at 14:44):

from a language perspective, c has no way of saying after you pass this thing to me, you may never use it again - the api own keyword must be a null macro. best you can do is pass thing ** & inside the call *thing = NULL so caller cannot reuse.

Steve Williams (May 05 2023 at 14:55):

void *__owned = NULL;
void Consume(void **thing)
{
   __owned = *thing;
   *thing = NULL;
}

int main()
{
   void *my_thing = 1;
   Consume(&my_thing );
   assert(my_thing == NULL);
}

Steve Williams (May 05 2023 at 14:59):

or, I guess, upgrade C :)

Ralph (May 05 2023 at 17:03):

this is my new favorite soap opera

Steve Williams (May 05 2023 at 18:14):

apologies. haven't upgraded our wasmtime or wasi stuff in ages so had a bit of an omg what moment :)

but now we're up to date. thanks guys :)

http://advance-software.com/develop
http://advance-software.com/products/browser
http://advance-software.com/xsg

still early but we've got a next-gen browser & next-gen web grammar (xsg) featuring web assembly secure programming interface, currently powered by wasmtime.

we're only using the wasm api so theoretically could swap out for any of the others, but happy with how things are for the now.

all the hard work very much appreciated.

Steve Williams (May 05 2023 at 18:18):

happy to add some credits if there's a document for stuff like that or we can put one together.

Steve Williams (May 05 2023 at 18:20):

& this thread can go :)

Ralph (May 05 2023 at 18:39):

WUT NO

Ralph (May 05 2023 at 18:39):

DAMMIT STEVE, PLEASE KEEP DIGGING FOR MORE

Steve Williams (May 05 2023 at 18:43):

I'm done for now :) debugger needs refinement but that's a side project. lldb needs that patch pushing but can maintain out of tree.
fread weirdness - very odd but not my problem :) we're solid. thanks all :)

Steve Williams (May 05 2023 at 18:44):

& wasm sdk is what it is. not a big deal, just gotta know how to drive :)

Last updated: Nov 22 2024 at 16:03 UTC