Your perfect debugging emulator.

[FAST6191]

Various people have considered thrashing out a debugging platform/API* that emulator developers can write to and there are the occasional interesting things that crop over from time to time (mostly disabled users seeing emulators as a way into games that might not otherwise be able to play), such goals are commendable and/or admirable but I am not really after that right now. Instead within the realms of realism (and despite serious efforts and very impressive work on the matter I will draw that line around decompilers, barring things like .net/C# and java of course) what would you like to see in a debug/hacker grade emulator for a system? Yeah it is a fluff thread that with probably not get anything done but hey.

*some of the cheat programs (emuhaste and artmoney being two of the more notable) have moved pretty far down this path actually.

Breakpoints
If
address read (ROM, RAM or otherwise)
address written
address modifications (break on change)
run to line/instruction (program counter?)
run next ? lines
within/equals/less/greater than (memory, program counter, registers (CPU or virtual/IO), flags, otherwise)
break on break/jump?
cycles (infinite loop detection?)?

Modifiers for the above? Granted you probably could manage if you wanted to say skip the middle of a range but it could get a bit unwieldy.

Cheat type functions
Hold location
If button pressed....
Scans
Exact in all the various formats (signed, unsigned, floats of various forms, raw hex), fuzzier stuff.
Stepping back in scans would be nice. So many times has an accidental double click messed things up for me.

Log points
BIOS calls?
In game function calls (think assembly)?
Logs the same as breakpoints?
Pointer code helpers?
Warnings? Can be quite nice if you are dodging anti piracy in a game.

Palette view/edit/dump
Tile view/edit/dump
Map view/edit/dump
3d model/matrix/stack view/edit/dump for those systems with 3d.
General memory view/editor (with table support).
Register view/edit/dump

Naturally auto/real time updates for any of those.

For the above what would a tracing (as in rewind/ram tracing/digital storage oscilloscope) be too far outside the lines? If I extended that to savestates (I would probably try for a branch type system here)? Technically a feature like this is kind of available as part of tool assisted speedrun aimed emulators but I would be opting for the hard drive space chewing options instead.

Also for the above search options (hex, decimal, ascii, table...)

Similarly corruption on any of the above?

Memory highlighting/visualisation? Various things exist for this on the PC, I have been known to use patterns and highlighting as well?

Sound channel editor. Waveforms, on/off/volume

Memory watcher for possible free space (think deadbeef padding).

Header viewer for the ROM?
Filesystem support for systems with known filesystems? Compression support (assuming BIOS/SDK/common/well known) there? Known archive formats? Some measure of programmability?

On the fly patching support? Available for many emulators already but often overlooked. For filesystems, especially on larger game sporting devices, anything preferential (load the iso but if there is an appropriately named file at proper location then load that in preference)?

Bookmarks?

Disassembler
Assembler
For the sake of a reasonably short post can we nod and say "basically IDA on a good day" (named locations, psuedo instructions, minor obfuscation workaround abilities....). Maybe a pipe to a general tool as well in case you end up debugging an emulator/interpreter or some extra hardware (the pocketstation for the PS1 uses an ARM processor onboard, the DC VMU was not SuperH, more generally arcades have some interesting things like daughter boards with actual processing from time to time).

Life is wonderful mode? Infinite stack/memory, the unreadable is readable internally, the unmapped is mapped internally, overclocked options..... Basically "not hardware accurate but enough that I can get things tested and then lock it down". I would agree this is potentially dangerous though. However if you are going for things like high res textures hacks...

Scripting for the above. FCEUX and Desmume both seem to favour lua for this sort of thing. Though I am not going to claim I am all that familiar with lua I can see why it was picked.

A pipe of sorts. Thinking text dumping, mainly with the disabled users in mind but that plus screenshots, timing, video.... could help a lot for the "give me context" type things.

[Azkadellia]

I'd say FCEUX is the gold standard for NES hacking.

[furrykef]

openMSX has one of the best features I've seen in any debugging emulator (which is pretty funny because it's otherwise a pretty poor debugger IMO). It allows you to rewind the emulation instruction by instruction, making it easy to answer the question, "How did the code get here?" I wish it also allowed me to run the program backwards until hitting a breakpoint, but coding it in a way that would run in a reasonable amount of time would be crazy difficult.

[Malias]

The feature I'd love in a debugging emulator would be one that could do a memory trace and say exactly where some data in memory came from.  It would be such a time saver.

[Kajitani-Eizan]

The feature I'd love in a debugging emulator would be one that could do a memory trace and say exactly where some data in memory came from.  It would be such a time saver.

+1, that would be amazing

presumably you hit a button and it savestates and starts logging both a tracelog and the player input, then you use breakpoints or just pause the emulator to get to the point where the value in memory is there, then hit magic reverse trace button to trace backwards to see where it came from. obviously it might get stuck if it reads/writes to intermediate memory steps, but then you just automatically load state and retry again with the new target memory address and the recorded player input. repeat until you find a source or reach the beginning of the log

[STARWIN]

You mentioned memory watching for free space. Not sure if you mentioned the FCEUX family feature of logging executed/read ROM bytes - it can be a nice thing for gaining confidence in ROM bytes being unused. I assume it might be a nice heuristic for the emulator to keep age data of RAM byte read/writes.

Not sure if you mentioned RAM dumps. I have used those for the purpose of associating a certain state in a game to RAM state (e.g. in-battle, cursor there, this thing done..), but it would be more convenient if the emulator could find the suspect bytes for me: assume 2 ingame states I want to compare, then I should just tell the emulator which state we are in and after doing that multiple times, the emulator should have ruled out most bytes as either unchanging (irrelevant) or changing too much (music, animations).

I have recently read some PS1 (dis)asm as opposed to NES/SNES. It could be about the program (or its complexity), but I suspect that systems with many registers simply have more difficult disassemblies to read. Especially with many registers it seems to me that in addition to disassembly there should be a pseudo-C representation.. not saying that it wouldn't be nice for simpler systems as well. If you consider decompiling to be something difficult, then this is not that.. something more local and not necessarily compilable should suffice.

If I were to code a debugger emulator, my thoughts would eventually be drawn to automated analysis of the program at some level. Simple things for UI like marking callable procedures with alternating colors and having a color of its own for unlikely-to-be-executable (bad opcodes) sequences, for example. Or potentially more complex things like visualizing causality. In general many technical tools probably have more capabilities than what can easily be deciphered from their UI, so you also have to think about the visual output if we are talking about perfection.

[FAST6191]

As with all of us I am sure were are more trained by the things we do and what we learned on and as most of my days are spent playing with GBA and DS games (and coming up on them too) it probably changed my opening post a bit, I tried not to have it happen and since gone back/sideways and read up on the other consoles but I guess it slipped through.
However on the GBA ROM space is not such a problem (the Mother 3 people probably had it about as hard as anybody ever would and were very much the exception, most people can more than double the available space by simply telling a hex editor to add more to the end of the file and knowing one quirk) and the GBA has carts fully memory mapped (no banks or other fun hardware) which means basic read/write logs/breaks/interrupts for normal addresses work here.

"be a pseudo-C representation"
On the GBA and DS thanks to Nintendo, no$GBA and crystaltile2 there is a format called NEF.
It does allow you to mark certain sections as functions (or whatever you like), assign names to memory locations (and easily enough for the virtual/IO stuff) and generally turn it back more towards a developer assembler rather than a more raw one as seen by the average hacker. I do not know if you ever played with HLA in the X86 assembly world but it is kind of like that.

On many registers, again I suppose it is what you are used to but in general I find it annoying to have to step through a whole bunch of setup code for a function or even simply to do something like load an immediate into a register where the immediate is the size of a register. It still happens and in some I am thankful it does but it is not half so prominent as the older stuff. Now if we are going to talk about the increase in things like SIMD instructions I will happily provide an example of someone looking to find a swift exit a lot of the time.

That said some automated analysis would be an interesting thing. Golden sun on the GBA is probably the furthest I have seen people go down this path though that was more header values, pointers and of a fairly well reverse engineered ROM. I have been known to bust out the tile editor even if I am not planning to touch graphics (spotting patterns in colours is easier than spotting patterns in hex) and a lot of my favoured text decoding/encoding finding methods are more probabilistic than precise.

[BRPXQZME]

I have recently read some PS1 (dis)asm as opposed to NES/SNES. It could be about the program (or its complexity), but I suspect that systems with many registers simply have more difficult disassemblies to read.

Well, MIPS is very much a RISC load/store ISA. The more registers does make things a little harder to keep in your head, but there is also the thing where instructions do as little as possible as generically as possible, which means different instructions are not quite as distinct in intent and it usually requires writing more instructions than in 65xx to accomplish the same thing.

On the other hand, putting aside the performance issues, more registers is probably better for function call abstraction in the long run. ABIs tend to set aside a few for doing just that, so they're not quite as general-purpose in practice as they are in theory.

On many registers, again I suppose it is what you are used to but in general I find it annoying to have to step through a whole bunch of setup code for a function or even simply to do something like load an immediate into a register where the immediate is the size of a register.

Interface-wise, do you envision something like Vim's folding for hiding this?

[FAST6191]

I had not really got that far but yeah a folding/collapsible section would probably do. Though I would probably also go for a sort of comment style lowering the contrast and a giant curly bracket leading to a comment/equivalent function in C or something along those lines, think how most of us do when we decide to add text to disassembler/hex window screenshots.

About the only indulgence I might do is if it is a function call or jump to one I might consider having a hyperlink of sorts to it or a hover to preview type thing, maybe even branch the current memory state and try to run it to see what comes back if it is only a few instructions without need for further input. I have seen a few things like this in some of the more inclined towards hand holding IDEs, I have not made it too far into that world though (notepad++/geany has syntax highlighting, compilers or the languages themselves will tend to have a nice bit of debug output available and together that sorts most of my needs out).

[ARM9]

Symbolic debugging. This would be very convenient for homebrewing with assemblers that generate debug symbol files. Could also write your own by hand for romhacking if the debugger uses a relatively simple format (see wla .sym files).

[cret]

gdb- and rap-backend for all archs, that your debugger supports. step back. rop-gadget-search. (tracing data)

[FAST6191]

Can't believe I forgot to mention GDB in the opening post.

Anyway GDB is a fantastic option if you are developing homebrew (assuming you compiled accordingly), I find it a tiny bit clunky for a lot of ROM hacking use though (it will largely do the job but with it not really being reverse engineer/rom hacking focused you have to work around quite a bit). Looking at your other thread I guess you attempted to solve that issue though.

Still any emu devs reading. If you are not inclined to add in the crazy debugging features I will absolutely take a GDB stub or something

[Gemini]

Anything that works or feels close to the no$XXX debugger series. Really, there isn't much more you'd ever want.

[cret]

if you want to prepare hacks like this http://www.youtube.com/watch?v=D3EvpRHL_vk&; you need the abillity to step back, search rop-gadgets and trace data, so that you can create a schedule for data-movements, cause you dont want the game to overwrite your payload.


@FAST6191: A plugin-system with lots of language bindings would be awesome. it would be nice, if it's scriptable (predefined moves via script). How do you want to display visual output, which lib are you going to use?

[FAST6191]

Though I would not turn down such abilities I am not sure they are required for such a hack, though I guess maybe if I was attacking an entirely unknown game (pokemon had long been seen to overwrite various memory locations by using in game functions with them being fairly well explored). That said I really like those injection hacks for pokemon.

What you do mean by plugin system? As it stands fceux and desmume have access to lua (lua gets a tiny bit fragmented but call it full lua) which can speak to memory locations in the game and other such things (not quite a trace/breakpoint level but nearly there and being a programming language it affords quite a bit of fancy stuff).
http://www.fceux.com/web/help/LuaGettingStarted.html (desmume mirrors much of the functionality and indeed links to the fceux site).

I can not say it has ever occurred to me to use graphics output in a ROM hack (though I must confess to not having pushed lua that far here) be it final or in porgress. I would probably just pass calls or at best load pictures and output text anyway. Similarly there are quite few bridges to other languages for lua, perhaps not as many as python or going to C? but enough that most devs using common use languages will not go wanting.

[cret]

there are still some pieces of the pokemon-roms, where nobody knows, what they're doing.

not directly speaking to memory, speaking to joypad would be cool

[FAST6191]

Where nobody has been able to figure it out (heavy obfuscation/bad programming/unused function.... they are assembly coded games which changes things a bit) or because they have not looked?

On speaking to controls and speaking to memory...
http://nocash.emubase.de/pandocs.htm#joypadinput
http://nocash.emubase.de/gbatek.htm#gbakeypadinput
http://wiki.nesdev.com/w/index.php/Controller_Reading
http://wiki.superfamicom.org/snes/show/Schematics%2C+Ports%2C+and+Pinouts
http://nocash.emubase.de/psx-spx.htm#controllerscommunicationsequence
http://hcs64.com/files/n64-hw.dox (do a page search for "-Channels-")
http://hitmen.c02.at/files/yagcd/yagcd/chap9.html#sec9 (si=serial) http://hitmen.c02.at/files/yagcd/yagcd/chap4.html#sec4
http://wiibrew.org/wiki/Hardware/Serial_Interface and http://wiibrew.org/wiki/Wiimote
http://free60.org/GamePad
http://nocash.emubase.de/pagezero.htm#c64interruptsandnmis (scroll up a bit)
....

In many of those cases things will have been debounced to somewhere in normal memory too. Similarly if that was not enough http://www.fceux.com/web/help/LuaFunctionsList.html (page search Joypad Library). http://tasvideos.org/LuaScripting.html is probably also worth a look, I tend not to have the greatest interest in speedruns (TAS or otherwise) but they do have some good stuff.

[rveach]

The feature I'd love in a debugging emulator would be one that could do a memory trace and say exactly where some data in memory came from.  It would be such a time saver.

+1, that would be amazing

presumably you hit a button and it savestates and starts logging both a tracelog and the player input, then you use breakpoints or just pause the emulator to get to the point where the value in memory is there, then hit magic reverse trace button to trace backwards to see where it came from. obviously it might get stuck if it reads/writes to intermediate memory steps, but then you just automatically load state and retry again with the new target memory address and the recorded player input. repeat until you find a source or reach the beginning of the log

My version of Nintedulator already does this. http://www.romhacking.net/utilities/476/
It doesn't use savestates, it just has a big memory cache and is always logging where every value goes. Intermediate memory is fine, and it logs up to 5 addresses incase one value is created from a combination of others.

[cret]

which api are you going to use, for basic-emu-foo, or are you going to code your own?

[Chippy2000]

Cret were you meant to be making ramulate  ? Why not implement debugging to? I think it's a great idea.