Now I figered you only need to check for a debugger in one of them since its only checking comms and waiting a frame there doesn't matter, but the more I tested this BRK thing, the more confusing it became.
I was able to make a test where it was out right ignoring BRK instructions! I simply couldn't figure that out, that is until I realised you need to call the debug stub EVERY interrupt so it can ALSO check for BRK instructions. Otherwise it'll just exit the IRQ and carry on like nothing happened.
So, that really annoying bug fixed, and I can now single step/run/stop in a rock solid manner - at last! Now I need to emulate RTS and RTI instructions so that the debugger works properly, and I'll probably emulate branches as well since it means I only need one breakpoint. Although I'll allow RTI via the monitor, you wont't be able to debug interrupts, this is just for emulator support as it WILL be able to debug interrupts. Because a real machine uses BRK's for breakpoints, its stub is inside an interrupt, but emulators dont have to do that, they can single step anywhere and dont need to write BRK's, they can just have a breakpoint table and check to the current PC which makes them very powerful.
I have to say I'm relieved, it was starting to get to me. I usually don't spend long on bugs, but this one was pushing it.... Still, all fixed and now we can really push on with some monitor features! It'll be great to get real watches and multiple memory windows without having to do to much. The comm's speed will probably hurt the update of watch windows, but we can probably put in a bit of smarts and a FORCE UPDATE button or something to help. Watches are probably the most powerful thing I've missed as you can ask for a variable and tell it the type and number of entrys you want displayed. i.e.
SrcTable,w,3: $8002, $8100, $8132
SpriteData,64: $00,$01,$02,$03,$04....(up to 64 bytes)
The other great thing about watches is that you only get the memory you need, not a whole memory window just to watch a single byte! You'll also notice I hope to do expression evaluation like "[Temp,w]+Y". This would get the word contents of Temp and add on the Y - or something like this. this is of course way down the line, but features like this will come, and everyone that uses the STUB interface will benifit without having to do it themselves.