Romhacking.net

Romhacking => ROM Hacking Discussion => Topic started by: zonk47 on April 08, 2016, 02:11:16 pm

Title: Making hacking easier
Post by: zonk47 on April 08, 2016, 02:11:16 pm
What can we do to make the process of hacking easier and faster?
Title: Re: Making hacking easier
Post by: Chronosplit on April 08, 2016, 03:11:56 pm
Not much IMO.  Research ends up in better tools for automating the simpler tasks in certain games, but other than that eventually you're always going to end up doing the same thing either way.
Title: Re: Making hacking easier
Post by: zonk47 on April 08, 2016, 04:00:17 pm
I think better emulator debugging functionality is key.

Let me restate/extrapolate on some ideas I had about the FF Legend/SaGa disassembly, and its challenges, that I conducted last night:
- there should be a debugger view option which displays the last values of the registers (only those changed by the instruction) at the moment of execution alongside the code. Also, for the bit manipulations such as shifting and rolling, it would help if the actual decimal arithmetic equivalents were also displayed.
- as someone mentioned in the "How to increase interest in hacking" thread, there is need for the ability to make the main routine auto-trace its loops. I can think of a few methods by which to implement this functionality, each with their own purposes.
- the code view window should only display bytes already executed as ASM, and mark areas read into RAM appropriately, so as to "map out" the ROM as the game is played. While there are limited cases of self-modifying code/RAM execution, I think work-arounds could be made for these. The former stipulation is vital, as the status quo is confusing as hell to everyone and makes hacking a pain generally.

Also I think there needs to be a "trace on input" function, in that this would help the hacker to invoke the debugger when triggering circumstances (such as map enters/exits and NPC triggers) that require decompression of data. We could crack compression easier this way, and find addresses for things like HP/shop data/exp with less trouble.

And I think we need to make these changes ourselves, because I don't think we can necessarily count on emulator authors to do them for us. If we do do them, however, hacking suddenly becomes loads easier and takes exponentially less time to pull off. We do need to stop relying on tools like IDA Pro though, because only debuggers can take the mental work burden off our brains (nevermind the fact that IDA Pro is designed with circumstances that have nothing to do with game ROM hacking in mind, nor will it ever be).
Title: Re: Making hacking easier
Post by: FAST6191 on April 08, 2016, 05:37:47 pm
Most debuggers that allow you to step through code will allow you to view the registers as much as you like.

I am not sure how useful a decimal readout would be though, maybe for cheats (which tend to have such things already) but if you have made it to the point in life where you are doing something useful with a debugger you probably know hexadecimal well enough. That said it would probably be but a few lines of code and not take up much UI space so eh.

A logging emulator that gives me a good idea of the standard game loops, vblank loops and such could be helpful at times. I can not see it being of all that much use outside of the people reverse engineering an entire engine though. My just needing to hack in a variable width font or something is less likely to benefit from this. Possibly more on that in a bit.

An actually executed instructions function to help narrow down data or nothing could be of some use too. I will avoid saying too much more as I deal mainly in later systems there "ram execution" is arguably the norm or at least nothing to note and share with the board as an oddity. Self modifying code we tend not to see (programmers largely agreed it was not a great plan) but emulation on systems (and I see everything from partial emulation to game specific emulation to could easily inject any old game you like in there) and interpreted languages would be troubled too I reckon and those are popular.

Compression seems to be less of an issue than it once was for the newer consoles which tend to use standard types (we call it things like GBA BIOS compression even). The compression is seldom the hard part anyway, it seems to be more the writing of a decompression tool or a compression tool (even a compression dodging tool that just sets uncompressed flags in the case of something like LZ) is the bigger ball ache. That said I think my approaches are strange here anyway and the main use for an emulator for me when testing compression related things is giving me a decompressed/decrypted file to look at as the result or testing it once I tried to put it back in. Again though I am not faced with too many truly custom compressions on the newer consoles and for the most part it is just a difference in flag type ("ooh this one uses 3 bits for the copy value and 5 for the read back distance compared to the normal 4,4 split" being the extent of the excitement there).

"in that this would help the hacker to invoke the debugger when triggering circumstances (such as map enters/exits and NPC triggers)"
Since we left single screen games and got scrolling then map exits are not really a hardware concept. If you then still have to debug the game to figure out what to look at for said concepts then a standard breakpoint would do, maybe one you tie to a certain value/range of OAM if you prefer to reference it to something on the screen, or better yet if you have to press a button to trigger the exit in the game then a standard button press breakpoint could do things here.
That said I forget what it is called but fceux offers a kind of new executed code so you start the session, do everything but what you want to look at, note it and then do the thing where the emulator then says "this is new" and you have your code to look at.
Title: Re: Making hacking easier
Post by: justin3009 on April 08, 2016, 05:53:22 pm
I'm not sure as far as debugging on other systems but the SNES has it really nice with Geiger's SNES9X (Even though it's not 100% perfect) with the Trace Logging system it has.  That is seriously a god send when it comes to debugging.  More debuggers should incorporate things like that in my opinion, make it a lot easier to figure the code out.
Title: Re: Making hacking easier
Post by: SleepyFist on April 08, 2016, 05:59:11 pm
Stable/Updated/Better Utilities would be great, many are in a state of decaying compatability.
Title: Re: Making hacking easier
Post by: RyanfaeScotland on April 08, 2016, 06:10:29 pm
Learn more. The more you learn the easier and faster it will become.
Title: Re: Making hacking easier
Post by: zonk47 on April 08, 2016, 06:15:09 pm
Quote from: FAST6191
Compression seems to be less of an issue than it once was for the newer consoles which tend to use standard types (we call it things like GBA BIOS compression even). The compression is seldom the hard part anyway, it seems to be more the writing of a decompression tool or a compression tool (even a compression dodging tool that just sets uncompressed flags in the case of something like LZ) is the bigger ball ache.

This is inconsistent with everything everyone but you has said about the topic. When people stop their hacking, it's usually because they can't deal with the compression.

Quote from: FAST6191
Since we left single screen games and got scrolling then map exits are not really a hardware concept.

I didn't say they were. Map exits happen when the sprite moves beyond the logical boundaries of the map, or onto a door tile. This happens, usually, in response to the player's motion of the player to move into that space. Once the player moves into that space, the game takes over until the next map is loaded. This about always involves decompression, and is moreover the only time except at the intro/ending sequences when it actually happens. So if say, you had a control panel in which to check a button for break-on-press, you could mosey your character over to the map's edge, or next to an NPC, then pause the emulator; check the button; and press the button. Then you'd be able to trace all the code dealing with the transition between maps.

This is actually completely necessary, because the code for input handling is generally interrupt based, particularly on handhelds where it is expedient to relieve pressure on the battery by suspending execution until an interrupt forces it to resume.

Learn more. The more you learn the easier and faster it will become.

Not likely. There's too much to be learned and not enough people willing to become specialized experts.

I do think we have to do this stuff ourselves, particularly because ROM hacking divides the community. In the beginning emulator authors had to work hand in hand with hackers... this is no longer the case. One of the most notable events that I can recall in recent emulator history was the departure of one of the lead programmers of PCSX2 several years ago. The departure was abrupt and without qualm: the dev said he had been offered a job that would receive his complete commitment from that day forward. Considering how passionate people in this community tend to be about their work, it must have been quite a sum. Over the past decade emulator authoring has moved from pariah status to sought-after specialization: emulator authors, for modern systems at least, are thought of as being experts in the modeling of highly abstract systems and self-starters besides, strong qualities for any technical project lead. At least a hundred (probably more) theses have been presented around emulator design. Association with pirates and/or people who work with (presumably) pirated material is a professional liability. Even those who are not pursued for their skill (or else dismissive of the pursuit) seem to be doing fine on Google Play, except when an association with piracy is made. Emulator authors have little financial incentive to make our job easier, so we will have to make the improvements ourselves. (disclaimer: I'd find it awesome if some of these ideas were integrated into working emulators by their original authors... I just don't think it likely).

April 09, 2016, 12:09:18 am - (Auto Merged - Double Posts are not allowed before 7 days.)
I'm not sure as far as debugging on other systems but the SNES has it really nice with Geiger's SNES9X (Even though it's not 100% perfect) with the Trace Logging system it has.  That is seriously a god send when it comes to debugging.  More debuggers should incorporate things like that in my opinion, make it a lot easier to figure the code out.

Geiger's emulator is a start. Needs work, but a start nonetheless.

@FAST6191: it seems to me that the following approach would make your work easier:
- you could run the decomp code in a CPU core to conduct the decompression of the material to RAM, then HDD automatically. This way, you'd only need one program for all decompression needs. As for the recompression, you could replace the decomp routine with a generic (such as LZW or Huffman) and compress it with that.
Title: Re: Making hacking easier
Post by: Rotwang on April 09, 2016, 03:17:49 am
We do need to stop relying on tools like IDA Pro though, because only debuggers can take the mental work burden off our brains (nevermind the fact that IDA Pro is designed with circumstances that have nothing to do with game ROM hacking in mind, nor will it ever be).

IDA Pro is tailor made for ROM hacking.  It has support specifically for Gameboy and SNES ROMs and will map out all the registers for you, address the ROM as though it were loaded on hardware and has this kickass tree view that makes ASM tweaks child's play. What the fuck is wrong with your brain that you created a thread called "Making hacking easier" and one of your suggestions is that people STOP using IDA Pro?


Not likely. There's too much to be learned and not enough people willing to become specialized experts.


And now you say that there's no point in learning how to hack because there is too much to learn... You're only kidding yourself at this point. If that's really your goal you might as well have called this thread "Making hacking braindead" which is completely unrealistic. You can't turn this hobby into something that it isn't. You get out of it what you put into it, and that's true of any hobby. If not learning anything is your strategy then all you can really do is sit around twiddling your thumbs for someone to make your magic bullet decompressor tool for you, which I guess is technically easier than learning to do your own hacks. The problem is much more complicated than you are making it seem, and I get the idea that you really don't know what you're talking about. You can't really expect the entire hobby to work like it's some sort of commercial make-your-own-game toolkit.
Title: Re: Making hacking easier
Post by: Turambar on April 09, 2016, 03:50:36 am
A feature I've always wanted is the ability to see what the program counter was on the previous instruction or even the last handful of instructions. Sometimes a breakpoint happens right after a jump and it would be really useful to see where it jumped from.
Title: Re: Making hacking easier
Post by: tryphon on April 09, 2016, 03:59:10 am
A feature I've always wanted is the ability to see what the program counter was on the previous instruction or even the last handful of instructions. Sometimes a breakpoint happens right after a jump and it would be really useful to see where it jumped from.

quoted for truth :)
Title: Re: Making hacking easier
Post by: Rotwang on April 09, 2016, 04:01:27 am
A feature I've always wanted is the ability to see what the program counter was on the previous instruction or even the last handful of instructions. Sometimes a breakpoint happens right after a jump and it would be really useful to see where it jumped from.

Logs are really useful for shit like this.
Title: Re: Making hacking easier
Post by: Turambar on April 09, 2016, 04:11:24 am
You could get it in a log, but then you'd have to properly time and make the log file and then dig through it. I want to see it immediately in the debugger, that is much more convenient.
Title: Re: Making hacking easier
Post by: STARWIN on April 09, 2016, 10:31:56 am
- there should be a debugger view option which displays the last values of the registers (only those changed by the instruction) at the moment of execution alongside the code.

trace log? or trace loggy additions for the disasm screen?

Also, for the bit manipulations such as shifting and rolling, it would help if the actual decimal arithmetic equivalents were also displayed.

what? like multiplying by two or dividing by two? for bit arrays definitely useless. rolling sounds like algo wizardy, it tends to be equally incomprehensible regardless.

- as someone mentioned in the "How to increase interest in hacking" thread, there is need for the ability to make the main routine auto-trace its loops.

how is this any different from setting breakpoint in the loop and using a trace log? "auto"? do you even know what is part of main routine and what not?

- the code view window should only display bytes already executed as ASM, and mark areas read into RAM appropriately, so as to "map out" the ROM as the game is played. While there are limited cases of self-modifying code/RAM execution, I think work-arounds could be made for these. The former stipulation is vital, as the status quo is confusing as hell to everyone and makes hacking a pain generally.

what are the benefits of mapping out a ROM?

Also I think there needs to be a "trace on input" function, in that this would help the hacker to invoke the debugger when triggering circumstances (such as map enters/exits and NPC triggers) that require decompression of data. We could crack compression easier this way, and find addresses for things like HP/shop data/exp with less trouble.

is this the same as setting a breakpoint at input routine that breaks when a button is pressed and then turning logging on manually?

And I think we need to make these changes ourselves, because I don't think we can necessarily count on emulator authors to do them for us.

good luck, but won't you turn into an emulator author if you do that?

A feature I've always wanted is the ability to see what the program counter was on the previous instruction or even the last handful of instructions. Sometimes a breakpoint happens right after a jump and it would be really useful to see where it jumped from.

I suppose "last PC + last n branches taken" could save a bit of time. When within a spaghetti routine at least. Between routines is easier because of step out.
Title: Re: Making hacking easier
Post by: zonk47 on April 09, 2016, 01:20:53 pm
I must admit... I'm quite taken aback... it looks like nobody here means to help me achieve these goals, so in self-respect to myself I am frankly hesitant to try advancing them any further. Is your pride really so great, that you are unable to make these small sacrifices for the good of the community?
Title: Re: Making hacking easier
Post by: Isao Kronos on April 09, 2016, 02:12:58 pm
you talk about "the good of the community" but some things I've seen you say don't match up with that
Title: Re: Making hacking easier
Post by: Rotwang on April 09, 2016, 02:13:13 pm
I must admit... I'm quite taken aback... it looks like nobody here means to help me achieve these goals, so in self-respect to myself I am frankly hesitant to try advancing them any further. Is your pride really so great, that you are unable to make these small sacrifices for the good of the community?

Okay, I will stop using IDA Pro and stop learning things. For the good of the community. You have convinced me.

What sacrifices are you even talking about? The time spent on making a magical tool that hacks all the ROMs for you and can abstract things like "map exits" on its own? Good luck with that. The fact that you want a program to tell you what math is really being accomplished by bit shifting betrays your inexperience. That's something that's covered quite early on in ASM documentation.

And what's with this "your pride is so great" bullshit? What pride? Stop trying to cover up the fact that this is a utility request thread in disguise as some kind of community altruism. Or is it a utility request thread? With some of your other ideas I don't know what goal you are even trying to accomplish here, I guess for someone or something to do all your thinking for you?


April 09, 2016, 02:29:59 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
You could get it in a log, but then you'd have to properly time and make the log file and then dig through it. I want to see it immediately in the debugger, that is much more convenient.

It's not like you only get one shot at it. Set a breakpoint to your instruction, record a log, and then the only "digging through" you have to do is looking at the end of your log.
Title: Re: Making hacking easier
Post by: zonk47 on April 09, 2016, 03:13:47 pm
Rotwang you are totally misunderstanding me. I talked about using an AI to assist in language translation, but I don't believe it would be the right road for hacking. Such a pipe has two ends, as they say, and the tools we make to help us could also be used against us.

I would just like to have greater clarity as to what's going on in the machine. I've got a couple VB8-based emulator projects in my IDE right now, but I could only do a really intuitive job of it by putting in custom code for each of the 256 (or more) operands in these cores.

As for the program counter history that's a five minute fix, for the VB-based emus anyway. This is not a utility request thread, it is a -collaboration- request thread.

Quote from: STARWIN
what are the benefits of mapping out a ROM?

It helps you locate obscure code.
Title: Re: Making hacking easier
Post by: tryphon on April 09, 2016, 03:45:31 pm
One thing I'd like to get would be a FREE IDA, a little less complex and rom-hacking oriented.

BTW, do anyone know which algorithm IDA uses to draw its tree view ?
Title: Re: Making hacking easier
Post by: STARWIN on April 09, 2016, 04:09:33 pm
Well the IDA version 5 seems to be freeware at their site. No idea what functionality it has as I haven't tried it.

What is this tree view exactly and what makes it so useful?
Title: Re: Making hacking easier
Post by: tryphon on April 09, 2016, 04:14:13 pm
IIRC free version only support very few processors, and not the 68000.

The graph view is this :

(https://www.hex-rays.com/products/ida/pix/5_plain_graph_view.gif)

That said, I personnally don't use it.
Title: Re: Making hacking easier
Post by: zonk47 on April 09, 2016, 04:21:46 pm
Well the IDA version 5 seems to be freeware at their site. No idea what functionality it has as I haven't tried it.

What is this tree view exactly and what makes it so useful?

It's use of the file explorer control to disassemble on demand. It makes a tree node for every instruction in a routine. You can click on a procedure in the routine, and it will make a branch below that routine with a node for each of its instructions. The downside is that it naturally requires a CPU core (or a lookup table anyway) with which to conduct the disassembly and give verbal feedback. It's a new approach so the only such table is available is for the Z80 and it doesn't do indexed mode or bit mode yet (so the disassembly tends to become less accurate).
Title: Re: Making hacking easier
Post by: Rotwang on April 09, 2016, 04:32:28 pm
It supports the 6502 and 65816 and will label all the SNES I/O, VRAM APU and DMA registers for you if you feed it a SNES ROM. It also supports the 68k but I haven't tested it yet.
Title: Re: Making hacking easier
Post by: STARWIN on April 09, 2016, 04:46:28 pm
mm.. I don't know about that graph view, seems a bit messy to read.

the way kingcom did it with pcsx2 (and how some other apps do it) seems nicer to me:

(http://aerie.wingdreams.net/wp-content/uploads/2014/08/pcsx2deb.png)

though an even better thing for readability would be decompiled pseudocode IMO. hopper seems to have something like that:

(http://www.hopperapp.com/Screenshots/capture5.jpg)

essentially the concepts of structural programming are good for removing spaghetti. i might try to code a some kind of decompiler just as a toy project for r3000 later, because it might speed up some of my projects.. or just for fun if not. no doubt a goal where perfection is impossible.
Title: Re: Making hacking easier
Post by: zonk47 on April 09, 2016, 04:48:36 pm
Link to treeview disassembler thread: http://www.romhacking.net/forum/index.php/topic,21605.0.html

It also uses (rudimentary) decompilation.
Title: Re: Making hacking easier
Post by: FAST6191 on April 09, 2016, 06:20:16 pm
This is inconsistent with everything everyone but you has said about the topic. When people stop their hacking, it's usually because they can't deal with the compression.

Numbers wise the biggest project enders seem to be people finding that the project will happen because they are absolutely committed to spending an entire weekend, fully enthusiastic throughout it, maybe with a bit of help from someone that knows what goes to watch over things -- a wonderful plan if you are doing DIY but less useful for ROM hacking. Though that might be a tiny bit far into cynicism, even for me, so I will cut that one off there.

Actually thinking about it I have to wonder how many things are compression. Much like hex editors seems to be given some false status as magic tools I often see "I can't find it/decode it, must be compressed" when there is no real indication of compression and there are probably a few other conventional decoding methods worth looking at.

Doubtless there are times when a simple project has turned annoying and fiddly, or a bit of quick help ended up somewhat more involved, by virtue of compression. I would also wonder if compression is an ancient bogeyman like NES mapper changes, music hacking or something else that increased computing power and technique refinement has brought into the realm of mortals or those with only the occasional weekend to spare rather than a full time months long reverse engineering effort.
Title: Re: Making hacking easier
Post by: zonk47 on April 09, 2016, 06:41:35 pm
Quote
Actually thinking about it I have to wonder how many things are compression. Much like hex editors seems to be given some false status as magic tools I often see "I can't find it/decode it, must be compressed" when there is no real indication of compression and there are probably a few other conventional decoding methods worth looking at.

Things compressed in games:
- maps: these are always compressed in commercial games unless coded into some kind of "block format". Even then, possibly compressed. Use of RLE is very common for map compression.
- most dialogue text. Text compresses really well with Huffman, DEFLATE, LZW
- all graphics not directly visible in a standard tile editor
- music samples and sound effects (and often the music itself. This is why you can't just dump the tune from ROM)
- plausibly, AI scripts.
Title: Re: Making hacking easier
Post by: FAST6191 on April 09, 2016, 07:05:51 pm
We must hack very different games, though the bulk of what I do is probably for the GBA and DS so that is a distinct possibility.

I would give reasonable odds on levels not being compressed for a lot of things.

I am not surprised to see compressed text, I am certainly not surprised to see uncompressed text.

I have not tried my hand at enough 16 bit and older era games to say much on the graphics there. Though having a look at various sprite sheets and those putting them together there are enough constructed graphics to really trouble people that just want the end result of sprite sheets (give or take palettes) to be there for them to play with. Move to the later consoles and the amount of custom tile sizes and decode formats that even without compression detection built into programs (most start with 10, or on the DS 10, 11 or 40 and then follow known patterns from there) you would be looking at those.

I would say music is not compressed, beyond anything more realistically called a codec, in the vast vast majority of cases. Simple throughput/turnaround reasons all but guaranteeing that. Dumping tunes is then considered hard because the music was more sequence/tracker style built to be read by sometimes rather specific hardware, and possibly also with its own CPU to govern it, than some kind of issues with a compressed file.

AI scripts? Unless we are talking about some card game where they have a database of plays and similar such things then most AI I see is usually very much in the binary.
Title: Re: Making hacking easier
Post by: zonk47 on April 09, 2016, 07:45:19 pm
Back on topic, it seems like there is agreement on the need for a PC status readout. This could be in two places: a tall box off to the side of the code view which shows the last 10 or so values of the PC counter with most recent at bottom, and in the code view window itself. But if we're in agreement on this, we need to get the emulator authors' attention by going to their boards and making the requests, holding those discussions etc etc.

As for Rotwang, sir your inexperience has shown in that you made a generalization about the mathematics related to rolling when there are actually a multitude of different standardized uses for it. Being unable to see the bit status of the operands makes it very difficult to deduce the purpose of the roll without pulling up a calculator and typing the numbers in before and after. In the SaGa disassembly there are segments of 10 straight instructions or more involving nothing but binary arithmetic. SaGa also has this habit of doing 3 or more jumps in a row without even doing a comparison in between... not the easiest thing to follow in a gigantic, VRAM hogging graph like those produced by IDA Pro. IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't. As already noted, it's not an emulator and only a debugger built into one can find everything. (I guess you could use it alongside a debugger but again, the inane stuff).

Quote from: tryphon
BTW, do anyone know which algorithm IDA uses to draw its tree view ?

Seems like a (bad) tracking algorithm like that used by SoM enemies. The kind that chases you up the stairs by running into the wall below them. I suppose the boxes themselves must have some sort of virtual boundary in the periphery, which the (hypothetical) arrows run into, and it continues on like that until a straight path is found.
Title: Re: Making hacking easier
Post by: Klarth on April 09, 2016, 08:00:22 pm
I don't see any specific recommendations on how to make hacking easier.

zonk47 appears to have difficulty using debuggers and assembly. Yes, it is difficult and there is no way around it but to invest time. If you want to learn assembly/hardware for your system, then coding demos is a good investment of time. If you want to learn compression algorithms, then code compression algorithms.

Quote from: zonk47
VRAM hogging graph like those produced by IDA Pro. IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't.
That is completely untrue. IDA Pro is intended for reverse engineering. That's the entire purpose. It's particularly useful for malware analysis. If you have the source code already, then you can use a real source-level debugger. Also, the graph does not take any substantial amount of VRAM to display compared to other UI elements.
Title: Re: Making hacking easier
Post by: zonk47 on April 09, 2016, 08:28:04 pm
That is completely untrue. IDA Pro is intended for reverse engineering. That's the entire purpose. It's particularly useful for malware analysis. If you have the source code already, then you can use a real source-level debugger. Also, the graph does not take any substantial amount of VRAM to display compared to other UI elements.

It's a canvas and 2-d acceleration is all the rage (and there are drop shadows, no less) so I find that unlikely.
Title: Re: Making hacking easier
Post by: Klarth on April 09, 2016, 08:46:38 pm
It's a canvas and 2-d acceleration is all the rage (and there are drop shadows, no less) so I find that unlikely.
Maybe back in 1995.
Title: Re: Making hacking easier
Post by: tvtoon on April 09, 2016, 09:07:05 pm
We should only bother making hacking harder, as it is already too easy! :D

Anyway, something that hasn't been done in a while, like documenting about common developers practices and kits used in games, could greatly help the community. Also trying to figure ways to detect stuff inside binary images, in a general way. Sum: improving the "science" of romhacking. :)
Title: Re: Making hacking easier
Post by: Rotwang on April 09, 2016, 09:26:37 pm
As for Rotwang, sir your inexperience has shown in that you made a generalization about the mathematics related to rolling when there are actually a multitude of different standardized uses for it. Being unable to see the bit status of the operands makes it very difficult to deduce the purpose of the roll without pulling up a calculator and typing the numbers in before and after. In the SaGa disassembly there are segments of 10 straight instructions or more involving nothing but binary arithmetic. SaGa also has this habit of doing 3 or more jumps in a row without even doing a comparison in between... not the easiest thing to follow in a gigantic, VRAM hogging graph like those produced by IDA Pro. IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't. As already noted, it's not an emulator and only a debugger built into one can find everything. (I guess you could use it alongside a debugger but again, the inane stuff).

I know exactly how shifting and rolling works. What crappy debugger are you using that you cannot see the bits that are being modified? I can convert binary to hex in my head and I'm sure that there others on this forum who can do that, and even if you can't, what shame is there in using a calculator in Programmer mode? It literally doesn't matter if IDA Pro is meant for engineers, and I care even less if your computer is too old and crappy to handle its average UI, because it does a wicked job of reverse engineering a ROM, and contrary to your purported "goal" it has made ROM hacking significantly easier for me. And I never said it was an emulator so I'm not even sure why you are even arguing that. I use it in conjunction with a debugger and a hex editor. What is even your complaint? That you sometimes have to use more than a single program at a time? That you have to count how many times in a row you see LSR and do 3rd grade math in your head? It's like you have this weird idiosyncrasy that people shouldn't have to use more than just one debugger or have to use their brain or learn things and it is completely baffling.
Title: Re: Making hacking easier
Post by: FAST6191 on April 09, 2016, 09:48:32 pm
For the DS that was and remains one of the main approaches used, though you might also want to look up some of the stuff Atrius did for his GBA Golden Sun editors.
Header detection, magic stamp analysis, fingerprinting of structures within it, things like 8h usually being the file size.... are all things I saw programs, did, taught and tried to note.
Unless you are wanting to go some kind of more automated pattern analysis -- if you have a 16 bit colour then most systems will tend to discard a bit to give 5 bits per colour channel, thus you are exceedingly unlikely to have F as a result in your graphics, said graphics are unlikely to be random noise so runs of stuff might also be worth a look, even with 256 colours most devs still used far fewer or at very best used some kind of shading, an emulator auto driven palette search/logging could be fun to see, the most common location for the GBA game cartridge was 08000000 through 09FFFFFF, though as things were less than 16 megs then it was 08 all the way and thus if you had a lot of 08 with 3 bytes in between then you likely have a pointer map/table/field/whatever so an automated map of the results of that, some kind of pointer analysis/limited decompilation/proto api looking for instructions that tickle certain parts of hardware and then almost immediately do something (again on the GBA a slight variation on that is used to track down the binary, possibly also parts of the sappy sound format many GBA games use), we know what valid opcodes are so some kind of statistical analysis where invalid ones are not so common, possibly the same as before but also weighted so we are not looking at the once in 10 programs type opcode is downplayed in favour of "add r1, r2",  better yet you rather have a compare without a jump and so on. Much of this would probably be more signal than noise but I do see things like it in malware analysis and obfuscation is the order of the day there. I do not see it making life much easier for anybody, certainly not newcomers, and instead be more likely to give interesting threads for already versed hackers to pull on.

Before I head too far into that world an idea I have had kicking around for a while is I want to see/do some full comparative ROM set analysis. I say the DS SDAT sound format is common and ignoring region dupes the list of things that do not use it is small, with the whole thing to look at I would know for certain and some decent stats on directory names and more besides. Vendor spam on Friday had a 6TB hard drive for not an awful lot and that would probably do nicely for what I need here. I reckon some interesting things could shake out there, indeed we see a very limited version of this in MAME where you have 6 games ever made for a given board and an interconnected analysis of the lot.

There is a slight hesitation that some might read too much into things, but at the same time I am all for common knowledge that is actually wrong to be blown away by meta analysis or some kind of controlled test.

On dev kits I am not sure how useful it would be for many of the older consoles. I have not looked much at what there is for the SNES but the megadrive one on this site is not terribly useful. Most such things I see tend to be more http://www.pagetable.com/?p=28 than truly useful.
I would also be wary of looking too much into them lest we end up doing something like the original xbox homebrew scene where they used the official SDK for most homebrew and thus could only have cloaked distribution of binaries. Nothing is likely to happen but why poison the well for no great gain.
Title: Re: Making hacking easier
Post by: Gemini on April 09, 2016, 09:56:25 pm
Well the IDA version 5 seems to be freeware at their site. No idea what functionality it has as I haven't tried it.
It only has a loader called PC, which can deal with most PC processors (except x64). You can still write your own processor handler, people wrote a SNES disassembler for it before the IDA people actually came up with their own integrated solution (neither is exactly great).

Quote
What is this tree view exactly and what makes it so useful?
It's pretty much a mode that visualizes code flow in order to give a better view of what the original code might have looked like. It can be helpful sometimes, while in most cases it's more of a nuisance. It all really depends on how you proceed in gather information on specific pieces of code. Text view still has arrows and stuff to tell you where the code is going, more or less like Kingcom's solutions.

IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't.
It is intended for any type of disassembling really. The scope is much broader than that.
Title: Re: Making hacking easier
Post by: zonk47 on April 10, 2016, 01:42:46 am
Gemini did you really pay $1200 for that thing?

I'm editing vb8086's debugger to include some new features, particularly those discussed in this thread. No ETA on that because nobody seems willing to help, although I did ask, didn't I? Nor is it unlikely that certain people who when asked to help refused will benefit from it... but what should be expected from a bunch of global warming deniers, really? Well, that's not fair to all of you, I guess people are busy after all. Busy enough to waste time writing crap for half an hour to try to get off the hook for helping out, although I guess some people do get a perverse thrill over behaving like solid walls.
Title: Re: Making hacking easier
Post by: Gemini on April 10, 2016, 07:25:51 am
Gemini did you really pay $1200 for that thing?
The full license is over 3.000 USD actually. Also no, a private donor usually fetches me their personal copy.
Title: Re: Making hacking easier
Post by: BlackDog61 on April 10, 2016, 04:34:29 pm
I must admit... I'm quite taken aback... it looks like nobody here means to help me achieve these goals, so in self-respect to myself I am frankly hesitant to try advancing them any further. Is your pride really so great, that you are unable to make these small sacrifices for the good of the community?
Zonk47,
It's funny. You've not even considered the possibility that we simply disagree with your approach?
So far, the proposal you've made doesn't make me think it will really help. It will take along time to reach a result, if we're still talking about automatic translation. Sorry, but that's how I feel.

I am interested in discussing other options, though.
Title: Re: Making hacking easier
Post by: KaioShin on April 10, 2016, 04:40:37 pm
zonk, people would be much more sympathetic to your ideas if you'd stop making completely unsubstantiated claims with no basis in reality. You simply don't know what you are talking about most of the time. Which would be fine if you'd accept that and asked instead of proclaiming your made up versions of reality simply as fact. In this thread and the other once there are dozens of things you said that are quite simply hogwash. You do not know how to use debugging tools properly. What you want are workarounds for your gaps in understanding and experience on how to debug. What's needed against that isn't different tools, it's better documentation and tutorials.

And going ad hominem ("global warming deniers") isn't going to help your case either.
Title: Re: Making hacking easier
Post by: Bregalad on April 10, 2016, 04:44:21 pm
What can we do to make the process of hacking easier and faster?
Hacking games that use generic format for whathever they use is easier. For instance, many GBA shares a generic music engine, and as such hacking music or sound in GBA games is very easy - or in other words after having gotten the principle, doing it on other games is exactly the same.

Hacking is easier when whathever you're working on has already been documented. It's also easier if you know which RAM and ROM locations is used and which is left unused for your own usage.

Finally, hacking is a lot easier when games leaves lots of ROM space free available. Changing things without expending is hard and very constraining.

I hope this helped.
Title: Re: Making hacking easier
Post by: KC on April 10, 2016, 05:14:05 pm
A PC history is not practical for many systems due to the massive performance costs. For recompiled code it could cut the speed into a fraction of the original speed, even if it's encoded directly into the code.

And I'm probably biased, but I feel PPSSPP's and PCSX2's debuggers do a pretty goob job improving the user experience.
Title: Re: Making hacking easier
Post by: zonk47 on April 10, 2016, 06:42:17 pm
If I do automatic trans, I'll probably do it on my own. I have a lot of specialized expertise for that kind of thing that I don't expect you to have. Right now I'm just trying to improve feedback in emulator debuggers, to make it easier to follow what's going on.
Title: Re: Making hacking easier
Post by: Bahamut ZERO on April 10, 2016, 06:51:10 pm
I imagine if YYCHR could load more types of savestates (like GBA, GBC, NES, Genesis), then a fair number of people would find at least the graphical aspects of hacking easier.  ;)
Title: Re: Making hacking easier
Post by: Klarth on April 10, 2016, 07:48:29 pm
A PC history is not practical for many systems due to the massive performance costs. For recompiled code it could cut the speed into a fraction of the original speed, even if it's encoded directly into the code.
To expand on this for those interested. Systems with higher clock speeds typically use dynamic recompilation to improve performance. Tracing debuggers (like PCSXTrace) will force you to use the static interpreter which reduces performance. For code tracers on the PSX with a 33MHz clock speed, this can be done by formatting the instruction, register values, etc via a sprintf call 33 million times a second. This is actually viable if you only need a few frames worth of data as your FPS will be < 5. You can achieve realtime operation by allowing the user to specify narrow code ranges to trace or by logging the execution of each instruction only once.

When you move up to PS2 at ~300MHz, these restrictions become even more important. An unfiltered, full trace 300 million times a second will now take many seconds to process a frame of execution. Thus techniques to limit the amount of code being traced become even more important. If you need instructions to be logged multiple times but don't know the range, tracing in two steps is required. The first "silent" trace will mark common instructions to not be logged. A second trace will then log the code of interest. This technique relies on the 90/10 rule where 90% of the time, only 10% of the code is running. Performance-oriented logging measures also become more important. Instead of text-formatting every instruction, execution can be faster by maintaining binary data structures that are text-formatted as a post-processing step.

The big takeaway for those who do not program utilities is that tracing becomes less useful for code isolation in higher clock rate systems due to performance and the massive logs created (expect many GB as compared to several MB for a SNES trace). Debugging via breakpoints becomes a necessary skill to isolate the code of interest. After the relevant code is located, understanding the code can be done through the debugger, a restricted trace log, or a disassembly via IDA Pro.
Title: Re: Making hacking easier
Post by: zonk47 on April 10, 2016, 09:27:44 pm
I'd think that slowdown during a log session/disassembly is actually desirable: gives you more ability to get a finer cut of the code.
Title: Re: Making hacking easier
Post by: elmer on April 10, 2016, 09:34:24 pm
I must admit... I'm quite taken aback... it looks like nobody here means to help me achieve these goals, so in self-respect to myself I am frankly hesitant to try advancing them any further. Is your pride really so great, that you are unable to make these small sacrifices for the good of the community?

WTF???

I've got to say ... the things that you're saying, and the way that you're saying them aren't exactly likely to endear you to any of the folks who might be in a position to help you.


And I think we need to make these changes ourselves, because I don't think we can necessarily count on emulator authors to do them for us.

If I do automatic trans, I'll probably do it on my own.

At the end of the day ... that's exactly how things get done in the real world.

If you see a need for something specific, then go and implement it yourself.

People are more likely to help out if you can show that you've already put in a lot of effort on your own part to improve things, and that means more than just talking up your own capabilities.

A bunch of emulators are open-source ... so just pick one that you like and start "fixing" things. If you do a good job, then perhaps your "improvements" will get rolled back into the mainstream distribution and everyone will win.

That's what I've done with Mednafen ... although Rypheca has stubbornly refused to take the changes (yet).  ;)

In the meantime ... those changes have dramatically improved my experience in hacking the translations that I've been working on, and have been well-received by the folks that hang-out on the forum where I've posted the binaries (and source code).

May I suggest to you that following a path like that could get you further than just asking folks to gather around the campfire and pledge to contribute their time to creating your own personal utopian ideal of automatic translation tools.
Title: Re: Making hacking easier
Post by: GHANMI on April 11, 2016, 12:24:22 am
I'd think that slowdown during a log session/disassembly is actually desirable: gives you more ability to get a finer cut of the code.

There's something called frame advance and trace logging.
Look it up. Or better yet, try it out. I recommend FCEUX.
I hope you won't just go "no this is unacceptably complicated" like in the topic about copy pasting a font's graphics in a tile editor that prompted you to make this thread (or maybe stuff about how FCEUX is coded in C rather than BASIC and hence it obstructs people from using it somehow) and try to be humble enough to learn how to do something new.
It'd be much more constructive and do you more good to be the changing force rather than waiting out the rules to bend for your convenience.

I imagine if YYCHR could load more types of savestates (like GBA, GBC, NES, Genesis), then a fair number of people would find at least the graphical aspects of hacking easier.  ;)

Since the savestates are a glorified RAM dump plus other register/CPU data and emulator fluff, it's a matter of making something recognizing where the RAM data begins in each savestate for each emulator, and where the palette in said ROM is stored.
I don't get why there's so many palette formats used (act, pal, yy-chr, etc..) but maybe starting with support for all these formats should be a good starting point.
Title: Re: Making hacking easier
Post by: Spooniest on April 11, 2016, 02:03:34 am
It would make it a lot easier if people weren't dysfunctionally rude, but there's little that can be done about that.

I'm not kidding or digging at anyone; this is simply a truism that I think kind of resounds through any creative endeavor.
Title: Re: Making hacking easier
Post by: zonk47 on April 11, 2016, 02:37:54 am
Mednafen... isn't that that commandline-only multi-emulator? I tried it briefly then quickly shunted it for Bizhawk. Total pain in the ass. Maybe I'd like your improvements though. One issue I have with a lot of emulators is retrace tearing in window mode. V-sync doesn't help, and neither does triple buffering. Probably has something to do with the 75hz refresh I've set for my monitor, but I'm not about to give that up.
Title: Re: Making hacking easier
Post by: RyanfaeScotland on April 11, 2016, 02:44:42 am
I'd think that slowdown during a log session/disassembly is actually desirable: gives you more ability to get a finer cut of the code.

I had this discussion with Kaneda a few weeks back but interesting to mention here too. Is frame rate that important when tracing? Unless you are doing some time critical code then I'd suspect you'd be stepping through as mentioned anyway so the 2 (trace vs framerate) can be considered mutually exclusive.
Title: Re: Making hacking easier
Post by: mrrichard999 on April 11, 2016, 02:45:34 am
Support emulator development and encourage devs to include powerful debugging tools in their emulators. With great tools comes great hacks.
Title: Re: Making hacking easier
Post by: RyanfaeScotland on April 11, 2016, 03:18:07 am
Support emulator development and encourage devs to include powerful debugging tools in their emulators. With great tools comes great hacks.

and great responsibility?
Title: Re: Making hacking easier
Post by: elmer on April 11, 2016, 12:05:45 pm
Mednafen... isn't that that commandline-only multi-emulator? I tried it briefly then quickly shunted it for Bizhawk. Total pain in the ass.

Everyone has their own "comfort zone".

I just tried BizHawk for the 1st time, and it's not for me ... too many windows, and the debugger doesn't show all the information that I want. But it does have some nice features, and I can see it being improved into something nicer.

Mednafen's problem (IMHO) is mainly in the debugging display layout and unreadable-fonts used, i.e. basic UI-design issues ... so that was what I "fixed".

Command-line-only doesn't bother me, nor does using hotkeys instead of menus (although it does involve a learning-curve).


Quote
Probably has something to do with the 75hz refresh I've set for my monitor, but I'm not about to give that up.

Does it really surprise you when you're displaying emulated-console screens that refreshes at 60Hz with a 75Hz monitor refresh???

If you're using an LCD display, then I just don't understand why you'd particularly want to set a 75Hz monitor refresh ... I guess that I'm missing something.
Title: Re: Making hacking easier
Post by: zonk47 on April 11, 2016, 03:09:39 pm
It's not LCD... it's CRT I think (later model, small back). At 75hz the image is considerably more crisp. Not all the emulators have this issue so it's definitely a technical issue for the ones that do.
Title: Re: Making hacking easier
Post by: zhade on April 11, 2016, 03:55:04 pm
One of the things I don't like about romhacking, as someone who is still kind of new to assembly / hex / binary operations and who probably isnt aware of the best tools out there and all.. is how much I have to repeat the same operations over and over again.. For example, sure I can find the binary value of "A5" and the the result of AND'ing it with "43" all in my head.. but honestly, I think it is quicker for me to use the calculator instead and more importantly, I really don't trust myself enough at doing it and prefer to use the calculator to be sure I don't end up pulling my hair off later if something don't make sense and waste time searching at the wrong places for what went wrong just to find out I made a simple calculation error when I could have just take 10 seconds to make the calculation using the calculator.

There is also the need to use multiple tools which are not "compatible" directly with one-another like for example I could be using geiger's debugger, have it break when some address is read, then when it breaks, since it only shows the current line of code and lets say I need to know what comes before that like of code, I'll have to go look in the disassembly, which means switching to notepad++, open the right bank file, copy the address from geiger's debug console (by double clicking the address from the output box and pressing CTRL+C) then press CTRL+F in notepad to search, paste the address from the debugger, change it a bit so it has the right address format for the disassembly (which could be say, from "$01/C01A"  to "01C01A:"). Then let's say the code makes me think the address read could be some graphics, I could switch to tile layer pro, copy the address I had set as a breakpoint from the debugger's breakpoint window, then use "goto address" (or whatever its called) in tlp, paste it, modify it from D31234 to 131234, find out I was wrong and it seems to be some data that is not graphics after all, switch to an hex editor to use a table to see if it might be text which again requires modifying a bit of the address..

Sure, none of this is "hard" and this thread's title is "Making hacking easier" but I think zonk47 was thinking of stuff like that when he asked "What can we do to make the process of hacking easier and faster?" and since I was actually in the process of making exactly that, things to automatize repetitive tasks, and thought I am surely not the first to have thought of that, I was eager to see what some you guys came up with.. but sadly zonk47's bad attitude towards beign wrong / corrected really seems to have wasted a big part of what could have been a really interesting read and a source of useful infos about tools and tricks to make the whole process less tedious which less experienced romhackers (or anyone into really) could have benefited from.

So, hoping the discussion can continue on the right track, here is what I have to propose as a way to save time, avoid errors and cope with multiple programs with picky address formats and the like:

I came across that beautiful thing called "Auto Hot Key", it has been around for like.. a decade maybe ? But I just never heard about it anywhere and since nobody talked about it here I just had to spread the word in case some of you guys didnt know about it.
Its an app that runs ".ahk" scripts which can do many useful things but what I use it for mostly is to make repetitive tasks automatic. With it you can assign "hotkeys" that will basically change the behavior of some keyboard/mouse/joystick inputs to something else.
For example, the first script I made was for visual studio (you can make hotkeys work only with some programs) I didnt like to have to right click on function calls and select "go to declaration" to navigate to the function's declaration, because even tho it takes only 2-3 seconds, Its 2-3 seconds wasted every time and I use this all the time.. Sure I could just go into visual studio's settings and set some key shortcut to make it faster like CTRL+ALT+Q or whatever, which is what I did at first, but since I was used to do it with the mouse, it felt wrong to use the keyboard shortcut and I had to left-click on the function first to put the carret on it for it to work anyway so I still used the right-click menu, and there is no way to set hotkeys to the mouse in visual studio..

Then I found out about autoHotKey, and right away, made a small script that basically just makes it so when I press the middle mouse button, instead it does as if I pressed the left mouse button followed by CTRL+ALT+Q. so now its done in a single click and im a happy bunny.

I got so used to it that I caught myself middle-mouse-clicking on addresses following jumps/branches when looking at disassembled snes asm.. so I thought.. why not make it actually work ? Easy, make the middle mouse button act as "LEFTCLICK / LEFTCLICK / CTRL+F / END / : / ENTER / ESCAPE" which selects the word under the mouse with the 2mouse clicks, search for it, add ":" to the end of the search string (because each disassembled line starts with the address as such C0/0000: ) then start the search with ENTER and close the search box with ESC. and voila, all it takes is a single click now.

That alone isnt very impressive (or so much useful for romhacking) but what makes me talk about this program is the way it makes it very easy to control the behaviour of applications and the way it is possible to enhance any debugger/tool with it. You can easily retreive the text from a chosen control from most applications as well as set it so for example I made a script that gets whats in geiger's debugger's console, takes the address of the last line, open the coresponding bank file in notepad++, then jump to it and highlight it, which makes it so I can follow the code execution using step into/over directly from the disassembly.

The best thing about it all is that it wouldn't be hard to adapt the script to make it work with bsnes+ as well and make the disassembly open into another text editor than notepad++ or make a middle-clicked address open in tile layer pro at the right offset.

it shouldnt be too hard to make a script that would detect whats under the cursor and if it is an hex value, show a tooltip with the decimal and/or binary value of it..

So even if a debugger is not open-source and its development stopped, there is still some things that can be done to make it work like you would like it to and then you can even reuse that feature you added to another debugger later so in a way its even better than having access to the source code !

oh and the scripts can be compiled to .exe so you can share it with the world afterwards... ok.. enough.. Sorry for the huge post I could talk about this all day XD
Title: Re: Making hacking easier
Post by: STARWIN on April 11, 2016, 04:31:22 pm
Geiger's does have a disassembly button that you can use to look at the surrounding code. It can also trace. I suppose having a disassembly open and searching it is roughly equally simple though. I'm not seeing any optimizations specific for that.

Overall going backwards in time has so far been easy for me. Having a save state slightly before the moment of interest, and then setting breakpoints earlier in the routine or doing a step out and setting a break before the call is basically it. The only difficult case would be spaghetti where the execution goes all over the place in a large routine (or lack of routine structure..), but haven't seen such yet.
Title: Re: Making hacking easier
Post by: zonk47 on April 11, 2016, 05:53:01 pm
My motive for editing vb8086 is to offer a capable debugging emulator for the PC-98 in English. Which is why I'm rather surprised that the only help being offered is advice (which really isn't helpful at all). What I need is division of the work. I think we can all agree that most debugging emulators do not offer enough information/feedback as to the immediate processes (or when they do, they only offer this information in terms of a single instruction at a time). That is what's really needed to make the process of hacking faster and smoother. I think the real reason you guys don't want to help is because you don't want it to be easy, because you enjoy the challenge of pushing your ability and wrestling with the experience of your own limits (and, I assume, it makes you feel good to see "lesser" minds fail). Coming from a cog sci background, I can assure you that's all gravy: if you compete better with another in terms of managing figures in a running formula/algorithm, its because your brain simply has more "buffer space" and therefore, more short term data retention (you might say, your CPU has more registers ;). But one brain design doesn't make one person better or superior to another.

@zhade: there is a difference between circumstantial coping and actual evolution. An evolution of tools would make the art of hacking more appealing and accessible... coping techniques like hot key settings really just make the process more obtuse to the outsider, who has to be briefed on yet more cryptic, arcane terminology and method (and as a result, is more likely to want someone else to teach them. However they will be unwilling/unable to pay for this tuition (if it is even offered) and no one will pay it for them, so they will never get the education). Nonetheless I think you did drive home the point that making hacking "easier" means making it faster to do. But again, I don't think the community really wants that because it would make hacking seem more like work (remember these are professional programmers who are used to having everything spelled out for them and bored of it) than a game.
Title: Re: Making hacking easier
Post by: Bahamut ZERO on April 11, 2016, 06:52:59 pm
Quote
I'm rather surprised that the only help being offered is advice (which really isn't helpful at all). What I need is division of the work.

I'm surprised you're still half expecting people to do half the work for you while shooting down the advice those same people offer in one fell swoop.

You oughta be glad they're even giving you advice with the way you  keep responding like a dildo.

Also, I think Zhade had some good, valid advice on making things a bit less tedious through outside means of the debuggers themselves. Saying that people are going to need paid tutors or higher education (or whatever the hell it was you were getting at) to set up some hotkeys is retarded. You essentially told him "fuck your advice, no one wants to use hotkeys WHY'S NO ONE BUILT HALF A DEBUGGER FOR ME" but with a paragraph instead of a sentence.
Title: Re: Making hacking easier
Post by: Klarth on April 11, 2016, 08:05:20 pm
My motive for editing vb8086 is to offer a capable debugging emulator for the PC-98 in English.
Use currently existing debuggers to see what features you like. Then code it. Nobody is going to do it for you. Imagine if your neighbor asked you to help him paint a fence and then sits down while you paint 90% of it. You're wanting to be the lazy neighbor.

I think the real reason you guys don't want to help is because you don't want it to be easy, because you enjoy the challenge of pushing your ability and wrestling with the experience of your own limits (and, I assume, it makes you feel good to see "lesser" minds fail). Coming from a cog sci background, I can assure you that's all gravy: if you compete better with another in terms of managing figures in a running formula/algorithm, its because your brain simply has more "buffer space" and therefore, more short term data retention (you might say, your CPU has more registers ;).
No, this is because the work is long and difficult, especially if you are a working in an area with poor documentation. Your lack of experience and drive to learn/experiment shows in this series of posts and not many people would want to work "with" you. Many knowledgeable people, myself included, have not done a good job in improving documentation to reduce the learning curve. At the end of the day, it is a hobby and writing documentation is not a goal for most people. I have a half-dozen other non-romhacking related hobby projects I could be working on instead of writing documentation.

What I need is division of the work.
Divide the work among yourself and then do it. Then let yourself or the community evaluate whether it's worthwhile.

But again, I don't think the community really wants that because it would make hacking seem more like work (remember these are professional programmers who are used to having everything spelled out for them and bored of it) than a game.
You have a poor concept of what professional programming is about. Most of us are either hobby-level programmers or were involved with romhacking well before getting a professional programming job.
Title: Re: Making hacking easier
Post by: magictrufflez on April 11, 2016, 08:44:19 pm
As someone who has zero skills related to hacking, I think somehow facilitating the learning process would probably be the best way forward, but I'm not exactly sure how you would do that with a hobby that takes place almost 100% over the internet.  I still would encourage every single person not skilled at hacking but still interested in doing hacks to start off with the pretty amazing selection of game-specific utilities on this site.  I honestly think if people spent more time practicing with those and starting small before getting discouraged by the amount of effort a complete-game hack takes, there would probably be a few more members of the community.

That said, most skills take practice and hard work, and you can find structured classes for help training in most of them.  You can find classes on programming too, but I guess I don't know how comparable romhacking and the stuff you find in proper programming classes are.  For all I know, the classroom stuff isn't always applicable to the actual practice of romhacking.

Maybe if someone gets some spare time they could create some kind of learning document for aspiring hackers to learn the basics?  Provided there isn't one out there already--I admittedly wouldn't know because I know I don't have time to learn.

It would probably be a lot of work, so it would have to be a labor of love (much like everything else on this site).  In my experience though, that's the only real way to streamline a skill-learning process.
Title: Re: Making hacking easier
Post by: FAST6191 on April 11, 2016, 08:48:20 pm
I think the real reason you guys don't want to help is because you don't want it to be easy, because you enjoy the challenge of pushing your ability and wrestling with the experience of your own limits (and, I assume, it makes you feel good to see "lesser" minds fail).

I have looked into this sort of thing for a variety of fields*, mainly as secret sauce knowledge and security by obscurity is something I particularly dislike.

*one of the more well studied possibly being the history of bump keys in lockpicking; Danish locksmiths supposedly knew of it for decades before it got popular elsewhere.

I have seen attempts at keeping some kind of ROM hacking knowledge an exclusive skill, usually in cheats but occasionally in things like sprite ripping, I believe I mentioned in a previous post in one of these topics. It tends to be seen in game specific circles and I am comfortable in saying anybody trying that here would get laughed out of the boards. At worst people have to occasionally pull themselves up by their own bootstraps as there are not enough people to hold hands through everything, though it results in fine hackers few would call that situation ideal. If you search hard enough you might find something around here that people did not overview at the time of posting, those would be the ROM hacker equivalents of hackmes or obfuscation challenges/contests ( http://www.ioccc.org/ ).

Anyway I can not speak for others, but I will pretend I can, and say there is a certain amusement to having the mad skills that others might not, however if said skills subsequently become common then it only means a new level has been opened up and said new level probably means a lot more scope for change or projects previously not viable then become viable for ease or time reasons (or if you prefer "the possible is now possible, how can I optimise it a bit more"). This will probably continue until such a time as we have natural language programming and some flavour of AI, and then only limited by imagination.

Maybe I am the exception though and I will have to return you to the pack of professional 6502, z80, 68K, 65c816 and ARMv3 programmers that inhabit the boards.

Back on topic you speak of code section identification and yeah having something tell me what are the main loops and subroutines and whatnot would be nice, however the methods you seem to want to explore are going to leave very fuzzy edges and thus are borderline useless compared to the accuracy of a conventional breakpoint. Similarly I am not sure how such a thing breaks down the most barriers, unless you thing said identification is going to act as a prebuilt tracing session/make a kind of quasi filesystem/data lookup table, in which case you are dreaming. There might be something to having some more soft analysis (relative search is hardly universal but it is a fundamental technique).

Title: Re: Making hacking easier
Post by: Rotwang on April 11, 2016, 09:01:29 pm
As someone who has zero skills related to hacking, I think somehow facilitating the learning process would probably be the best way forward, but I'm not exactly sure how you would do that with a hobby that takes place almost 100% over the internet.  I still would encourage every single person not skilled at hacking but still interested in doing hacks to start off with the pretty amazing selection of game-specific utilities on this site.  I honestly think if people spent more time practicing with those and starting small before getting discouraged by the amount of effort a complete-game hack takes, there would probably be a few more members of the community.

That said, most skills take practice and hard work, and you can find structured classes for help training in most of them.  You can find classes on programming too, but I guess I don't know how comparable romhacking and the stuff you find in proper programming classes are.  For all I know, the classroom stuff isn't always applicable to the actual practice of romhacking.

Maybe if someone gets some spare time they could create some kind of learning document for aspiring hackers to learn the basics?  Provided there isn't one out there already--I admittedly wouldn't know because I know I don't have time to learn.

It would probably be a lot of work, so it would have to be a labor of love (much like everything else on this site).  In my experience though, that's the only real way to streamline a skill-learning process.

A few years ago I tried writing a tutorial on grasping Hexadecimal pertaining to ROM hacking from the perspective of a 100% inexperienced nooblet, covering real-world questions like "what the fuck am I looking at?" and "why the fuck would anyone do it this way?" It's here: http://www.baddesthacks.net/?p=1118

Good documentation and lessons on the fundamentals combined with the reader's drive to learn and master a skill is the only realistic answer to this thread's great question, but Zonk is against this in favor of a one size fits all magic ROM hack button that will never exist.

Now if you'll excuse me I have a global warming deniers rally to attend.  ::)
Title: Re: Making hacking easier
Post by: GHANMI on April 11, 2016, 10:41:02 pm
http://www.baddesthacks.net/?p=1118

I know I have weird tastes, but a hex editor in binary with table support for characters that are not standard 8 bit would be really interesting for cases like Battle of Olympus (5 bits/letter), but more commonly JP games with 10/12/32 bit/letter.
We have tile editors allowing for sub 8x8 tiles (thankfully), why not hex/text too?
If I ever begun coding my own hex editor (hopefully soon, once I'm done with C++ and Qt) I'm planning this sort of thing (maybe hira/kata tbl switching too).

One of the things I don't like about romhacking, as someone who is still kind of new to assembly / hex / binary operations and who probably isnt aware of the best tools out there and all.. is how much I have to repeat the same operations over and over again.. For example, sure I can find the binary value of "A5" and the the result of AND'ing it with "43" all in my head..

The way I like to remember how it works, wouldn't it be better to look at how AND works like a coffee filter?
All the bits that would match a zero bit on the other end would all get zeroed out.
And bits that would match a one bit slip to the other side...

For example with SMB3, when the value from $0578 (setting Mario's form but also other things) gets sent to $00ED (Mario's actual form), it gets AND'd with #$0F (0000 1111) and the value has its leftmost 4 bits zeroed out (filtered out) after this?
In fact, that line's purpose in SMB3 is to filter out values out of an acceptable range.

I'm sure for other logical operations one can make up their own little memorization trick :P
I reckon looking at the problem from a binary standpoint would be tiresome, but it's one of the things that come with this stuff. Maybe if one is used to making cheat codes using bit flags (like status effects in RPGs, or e-Switchs in SMA4) it will be more natural and some notable hex values (01, 02, 04, 08, 10, 20, 40, 80) will stand out, and eventually everything will seem easier as you go.

Auto Hot Key

Apologies for not replying earlier, but this is one really excellent suggestion and I'm pleased everytime Auto Hot Key is mentioned.
First heard of it with people discussing how to deal with a difficult Flash platformer (it's "Pause Ahead" btw, very nice game) by automating certain hard jumps. It's really useful for lots of things, and with it being open-source, it could be useful.

Title: Re: Making hacking easier
Post by: elmer on April 11, 2016, 11:28:43 pm
A few years ago I tried writing a tutorial on grasping Hexadecimal pertaining to ROM hacking from the perspective of a 100% inexperienced nooblet, covering real-world questions like "what the fuck am I looking at?" and "why the fuck would anyone do it this way?" It's here: http://www.baddesthacks.net/?p=1118

I still recommend something like Rodnay Zaks "Programming the Z80" as a book for learning the fundamentals, including binary and hex.

I recently learned that it's actually downloadable now ...

http://www.z80.info/zip/programming_the_z80_3rd_edition.pdf

When you're done with the first 3 chapters of that, you'll have a good set of basic information as to how computers work, and what all this weird stuff means. From my POV, if you can get your head around that, then everything else becomes much, much easier.

Then, if you start by experimenting on a simple machine, like the GameBoy, you'll have a chance to really learn some solid fundamentals before you try to dive into 20-30 years of successive development and specialization that have lead to the complexity of modern machines.

Of course ... that's just my opinion!
Title: Re: Making hacking easier
Post by: Rotwang on April 12, 2016, 12:34:20 am
Yeah, something like that, but with caveats specific to ROM hacking. A lot of these manuals talk about terminating the program, when ROMs tend to only end when the power is cut. That sort of thing.
Title: Re: Making hacking easier
Post by: zonk47 on April 12, 2016, 02:02:12 am
Quote from: GHANMI
I know I have weird tastes, but a hex editor in binary with table support for characters that are not standard 8 bit would be really interesting for cases like Battle of Olympus (5 bits/letter), but more commonly JP games with 10/12/32 bit/letter.

You propose a hex editor which uses a virtual address space to map 5-bit sequences (fivelets?) across a range of bytes. It sounds hard to decode though. Would be better to do it in sets of five bytes, with the fifth byte containing the 5th bits for the nibble pairs in each. (or is this what you were considering?)
Title: Re: Making hacking easier
Post by: GHANMI on April 12, 2016, 02:39:56 am
You propose a hex editor which uses a virtual address space to map 5-bit sequences (fivelets?) across a range of bytes. It sounds hard to decode though. Would be better to do it in sets of five bytes, with the fifth byte containing the 5th bits for the nibble pairs in each. (or is this what you were considering?)

Dunno why but the way you say it sounds overly complicated.
Anyways.

Text data usually encodes data with one byte per character.
One byte has only 256 possible values, so Japanese games often (not always) use 2 bytes per character.
Some even use three bytes.

2 bytes means 65535 possible values.
Most retro Japanese games have around 650 kanji characters.
Recent ones have around 2800 kanji characters. The ones who put all possible characters have around 8000.

So 65535 is a bit of an overkill.
But it's far easier to just use 2 whole bytes for the job, and so most devs do just that.

But devs in the nineties who had big troubles fitting the text in a tiny ROM, disagreed about that.
So instead of using whole bytes (each byte being 8 bits), they'd encode each character on less than 8 bit multiples.
That's a simple compression.

Lagrange Point had less than 0x80 characters.
And those characters were two identical sets (like upper case and lower case).
So they used 6 bits per character.

Battle of Olympus only needed 26 English letters and a few control codes.
So they used 5 bits per character.

As Rotwang said in his tutorial linked here, that turd of a game for NES based on the Simpsons
used 4 bits (half a byte, also called "nibble") per character for the most needed letters,
and 8 bits (one byte) for the other letters.
This idea was improved and called the Huffman compression and is widely used but the character length in bits is variable and thus wouldn't interest us.

There's games using even 12 bits per character (Zelda Kagami no Triforce and Tengai Makyou Zero for example).

Well.
Some hex editors offer options to modify the "unit" from a byte (8 bit) to a nibble (4 bit) or a half word (16 bits/2 bytes) or a word (32 bits/4 bytes). But all of these are multiples of 8.
If these options were freely configurable so that a table using less than 8 bit values per character (be it 5, 6, 8, 10, 12..), it would help for a few games (I bet the main usefulness would be for stuff like the slightly more common 3-byte characters though).

Title: Re: Making hacking easier
Post by: zonk47 on April 12, 2016, 03:00:22 am
If you're just gonna encode these characters as continuous bits streams, then the streams must always be multiples of 8, in terms of bytes, to be even.

for 5 characters you don't hit a multiple of 8 until 40 bits (every 8 characters)
for 6 characters not until 48 (8 characters)
for 7 characters not until 56 (8 characters)

So below 8 bits, its always an 8 character sequence.... and I think this continues on... it's always
eight characters. But after 10/80, the length of the sequence (in bytes) must be the same as the bit length to always be even.

(13 x 8 = 104, 17 x 8 = 136)

In other words, any sequence so coded/compressed must be a multiple of the binary size x 8 bytewise. If you have a five bit sequence, its must be sized 5 bytes, 10 bytes, 15 bytes etc; for a 6 bit sequence, 6 bytes, 12 bytes, 18 bytes, etc.; and so on. Any file produced by such an editor will naturally have size that is a multiple of its bit/character range.

I'm pretty sure that Dragon Warrior 2 uses 5 bits for its tiles... I do remember having enormous difficulty following the DWEdit's viewer code for its maps.
Title: Re: Making hacking easier
Post by: GHANMI on April 12, 2016, 03:21:54 am
If you're just gonna encode these characters as continuous bits streams, then the streams must always be multiples of 8, in terms of bytes, to be even.

Err...
I'm not the one who encoded them that way, the game programmers did (and made it work so it's obviously not impossible).
Also, it's entirely possible to make the hex editor parse units of less than 8 bits to interpret as characters.
There's that nice thing called bit shifting, and logical operations...

Just one example of a possible implementation.
Supposing the current setting is 7 bits / character.

Hex editor reads the first byte.
We need first 7 bits.

So, we have one leftover bit.
We copy him, and him alone, with an "AND #$01" (0x01 = 0000 0001) to a value called DATA_LEFTOVER (which uses a whole byte)
We have a variable HOWMANY_LEFTOVER telling us how many leftover bits there are. We put 1 in that variable
DATA_LEFTOVER is shifted to the left "7 minus HOWMANY_LEFTOVER" times.

For the first byte, we copy it to our first character,
shift its bits to the right as much times as HOWMANY_LEFTOVER (so, only once),
and do an "AND#$7F" (0x7F = 0111 1111).
We get our first character.

Now, hex editor reads first 7 bits minus HOWMANY_LEFTOVER.
So only 6 bits.
Now those 6 bits are copied to our second character after some bit shifting to the right, then OR'd with DATA_LEFTOVER (which has the remaining bit from that character that was in the first byte).

And so on.
It's very doable without major changes to how the hex editor works.
Title: Re: Making hacking easier
Post by: zonk47 on April 12, 2016, 03:33:06 am
The size of the stream produced nonetheless ought to be as many bytes as there are bits per character. For 13 bit character, 13 bytes etc.

My method would be to make an array of as many bits in a minimum sequence (say, 48 for 6 bit characters/6 byte sequence) and use multiplication to find the first bit of the character, then just start setting 1s and 0s. This way you have your data model for the writing. To do the writing, set the bits based on the values in the array using a per-byte/per-bit loop.

Your way is probably more space efficient, though it's very difficult to follow. You'll have to do a comprehensive write up.

The vb8086 debugger is coming along well; however, I'm having difficulty understanding the extension mechanism (that is, opcodes beyond $FF that are accessible via the GRP instructions). How do these things work, in terms of bytecode?
Title: Re: Making hacking easier
Post by: GHANMI on April 12, 2016, 10:48:49 am
The size of the stream produced nonetheless ought to be as many bytes as there are bits per character. For 13 bit character, 13 bytes etc.

Eh? No.
Each 7 (or whatever) bit character is stored in a byte for convenience.
Bit shifting solves all problems.
Title: Re: Making hacking easier
Post by: magictrufflez on April 12, 2016, 12:18:19 pm
A few years ago I tried writing a tutorial on grasping Hexadecimal pertaining to ROM hacking from the perspective of a 100% inexperienced nooblet, covering real-world questions like "what the fuck am I looking at?" and "why the fuck would anyone do it this way?" It's here: http://www.baddesthacks.net/?p=1118

Good documentation and lessons on the fundamentals combined with the reader's drive to learn and master a skill is the only realistic answer to this thread's great question.

I totally agree 100% on this.I think the community is pretty good about helping people when they need specific help, but I think it might be asking too much for folks who are doing this as a hobby to step in and take the time to teach people on a consistent basis.  Or at least on a basis that would be useful for new hackers--that's why I think the type of doc that Rotwang made is really really valuable.

If there are more kind of tutorials and stuff, I think just getting a good compilation of them (and maybe even reviews so folks don't end up starting with bad ones) is probably the best way to ease the growing pains.  Is there a place like this we could maybe redirect people to if they want learning materials?  Or how much work do you guys think it would take to put something accessible together?
Title: Re: Making hacking easier
Post by: BlackDog61 on April 12, 2016, 03:01:57 pm
The size of the stream produced nonetheless ought to be as many bytes as there are bits per character. For 13 bit character, 13 bytes etc.

My method would be to make an array of as many bits in a minimum sequence (say, 48 for 6 bit characters/6 byte sequence) and use multiplication to find the first bit of the character, then just start setting 1s and 0s. This way you have your data model for the writing. To do the writing, set the bits based on the values in the array using a per-byte/per-bit loop.

Your way is probably more space efficient, though it's very difficult to follow. You'll have to do a comprehensive write up.

The vb8086 debugger is coming along well; however, I'm having difficulty understanding the extension mechanism (that is, opcodes beyond $FF that are accessible via the GRP instructions). How do these things work, in terms of bytecode?
I agree with GHANMI. The bit-to-byte matching proposed here is totally unnecessary. Bit shifting + bit masking handles this nicely. You can even map to bitfield structures if you really want that (there are compiler compatibility limitations for those, so you might want to stick to shifting).
... Unless you're writing this with VB in mind and VB has some language limitations with regards to bits handling? (I don't know VB that much.)
Title: Re: Making hacking easier
Post by: Bahamut ZERO on April 12, 2016, 03:28:37 pm
A few years ago I tried writing a tutorial on grasping Hexadecimal pertaining to ROM hacking from the perspective of a 100% inexperienced nooblet, covering real-world questions like "what the fuck am I looking at?" and "why the fuck would anyone do it this way?" It's here: http://www.baddesthacks.net/?p=1118

Good documentation and lessons on the fundamentals combined with the reader's drive to learn and master a skill is the only realistic answer to this thread's great question, but Zonk is against this in favor of a one size fits all magic ROM hack button that will never exist.

Now if you'll excuse me I have a global warming deniers rally to attend.  ::)

Those two tutorials you wrote are both hilarious and informative. I always wondered what people were talking about when they mentioned "nybbles."

And the method you mentioned for creating table files was a great tip! I'll keep it in mind the next time I'm text-hacking a game and can't track down it's font graphics (my usual method of finding values to make a table file with). Damn fine reads!
Title: Re: Making hacking easier
Post by: Rotwang on April 12, 2016, 07:56:57 pm
Thanks, good to know these are useful. I will consider continuing the series.
Title: Re: Making hacking easier
Post by: RyanfaeScotland on April 13, 2016, 03:58:37 am
Those two tutorials you wrote are both hilarious and informative...

There is a fairly common used phrase along the lines of 'Explain it to me like I'm a 5 year old' which is probably why the tutorials at BaddestHacks are so good, because the average contributor there has the mentality of a 5 year old. ZIIINNNNGGGGGGG!  :laugh: :thumbsup:
Title: Re: Making hacking easier
Post by: Rotwang on April 13, 2016, 07:46:54 pm
There is a fairly common used phrase along the lines of 'Explain it to me like I'm a 5 year old' which is probably why the tutorials at BaddestHacks are so good, because the average contributor there has the mentality of a 5 year old. ZIIINNNNGGGGGGG!  :laugh: :thumbsup:

Yes, but let the record show that you registered there last week  :laugh:

I honestly love writing documents like this because it forces me to verify everything I think I already know about the subject and in the end I wind up learning some things myself or correcting any misunderstandings I might have had.
Title: Re: Making hacking easier
Post by: zhade on April 14, 2016, 12:33:43 pm
I honestly love writing documents like this because it forces me to verify everything I think I already know about the subject and in the end I wind up learning some things myself or correcting any misunderstandings I might have had.

Yeah, I've heard the saying that you don't really understand something completely unless you can explain it to a 5 year old. That must be a really good exercise that makes sure you understand each concept fully. I think the same goes for the readers too, even the more advanced might have some misunderstandings that wouldn't be corrected if they were not taken for "dummies" :P
Title: Re: Making hacking easier
Post by: BlackDog61 on April 14, 2016, 04:34:01 pm
Yeah, I've heard the saying that you don't really understand something completely unless you can explain it to a 5 year old. That must be a really good exercise that makes sure you understand each concept fully. I think the same goes for the readers too, even the more advanced might have some misunderstandings that wouldn't be corrected if they were not taken for "dummies" :P
If even 5-year olds contribute to game translations, I'm all for it!

Any chance of promoting that link to the Getting Started section?
Title: Re: Making hacking easier
Post by: zonk47 on April 14, 2016, 05:14:28 pm
gbatemp has an introduction to Kanji translation. Might want to check that out for ideas.