News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Making hacking easier  (Read 11759 times)

tryphon

  • Hero Member
  • *****
  • Posts: 722
    • View Profile
Re: Making hacking easier
« Reply #20 on: April 09, 2016, 04:14:13 pm »
IIRC free version only support very few processors, and not the 68000.

The graph view is this :



That said, I personnally don't use it.

zonk47

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Making hacking easier
« Reply #21 on: April 09, 2016, 04:21:46 pm »
Well the IDA version 5 seems to be freeware at their site. No idea what functionality it has as I haven't tried it.

What is this tree view exactly and what makes it so useful?

It's use of the file explorer control to disassemble on demand. It makes a tree node for every instruction in a routine. You can click on a procedure in the routine, and it will make a branch below that routine with a node for each of its instructions. The downside is that it naturally requires a CPU core (or a lookup table anyway) with which to conduct the disassembly and give verbal feedback. It's a new approach so the only such table is available is for the Z80 and it doesn't do indexed mode or bit mode yet (so the disassembly tends to become less accurate).
A good slave does not realize he is one; the best slave will not accept that he has become one.

Rotwang

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: Making hacking easier
« Reply #22 on: April 09, 2016, 04:32:28 pm »
It supports the 6502 and 65816 and will label all the SNES I/O, VRAM APU and DMA registers for you if you feed it a SNES ROM. It also supports the 68k but I haven't tested it yet.

STARWIN

  • Sr. Member
  • ****
  • Posts: 445
    • View Profile
Re: Making hacking easier
« Reply #23 on: April 09, 2016, 04:46:28 pm »
mm.. I don't know about that graph view, seems a bit messy to read.

the way kingcom did it with pcsx2 (and how some other apps do it) seems nicer to me:



though an even better thing for readability would be decompiled pseudocode IMO. hopper seems to have something like that:



essentially the concepts of structural programming are good for removing spaghetti. i might try to code a some kind of decompiler just as a toy project for r3000 later, because it might speed up some of my projects.. or just for fun if not. no doubt a goal where perfection is impossible.

zonk47

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Making hacking easier
« Reply #24 on: April 09, 2016, 04:48:36 pm »
Link to treeview disassembler thread: http://www.romhacking.net/forum/index.php/topic,21605.0.html

It also uses (rudimentary) decompilation.
A good slave does not realize he is one; the best slave will not accept that he has become one.

FAST6191

  • Hero Member
  • *****
  • Posts: 2499
    • View Profile
Re: Making hacking easier
« Reply #25 on: April 09, 2016, 06:20:16 pm »
This is inconsistent with everything everyone but you has said about the topic. When people stop their hacking, it's usually because they can't deal with the compression.

Numbers wise the biggest project enders seem to be people finding that the project will happen because they are absolutely committed to spending an entire weekend, fully enthusiastic throughout it, maybe with a bit of help from someone that knows what goes to watch over things -- a wonderful plan if you are doing DIY but less useful for ROM hacking. Though that might be a tiny bit far into cynicism, even for me, so I will cut that one off there.

Actually thinking about it I have to wonder how many things are compression. Much like hex editors seems to be given some false status as magic tools I often see "I can't find it/decode it, must be compressed" when there is no real indication of compression and there are probably a few other conventional decoding methods worth looking at.

Doubtless there are times when a simple project has turned annoying and fiddly, or a bit of quick help ended up somewhat more involved, by virtue of compression. I would also wonder if compression is an ancient bogeyman like NES mapper changes, music hacking or something else that increased computing power and technique refinement has brought into the realm of mortals or those with only the occasional weekend to spare rather than a full time months long reverse engineering effort.

zonk47

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Making hacking easier
« Reply #26 on: April 09, 2016, 06:41:35 pm »
Quote
Actually thinking about it I have to wonder how many things are compression. Much like hex editors seems to be given some false status as magic tools I often see "I can't find it/decode it, must be compressed" when there is no real indication of compression and there are probably a few other conventional decoding methods worth looking at.

Things compressed in games:
- maps: these are always compressed in commercial games unless coded into some kind of "block format". Even then, possibly compressed. Use of RLE is very common for map compression.
- most dialogue text. Text compresses really well with Huffman, DEFLATE, LZW
- all graphics not directly visible in a standard tile editor
- music samples and sound effects (and often the music itself. This is why you can't just dump the tune from ROM)
- plausibly, AI scripts.
A good slave does not realize he is one; the best slave will not accept that he has become one.

FAST6191

  • Hero Member
  • *****
  • Posts: 2499
    • View Profile
Re: Making hacking easier
« Reply #27 on: April 09, 2016, 07:05:51 pm »
We must hack very different games, though the bulk of what I do is probably for the GBA and DS so that is a distinct possibility.

I would give reasonable odds on levels not being compressed for a lot of things.

I am not surprised to see compressed text, I am certainly not surprised to see uncompressed text.

I have not tried my hand at enough 16 bit and older era games to say much on the graphics there. Though having a look at various sprite sheets and those putting them together there are enough constructed graphics to really trouble people that just want the end result of sprite sheets (give or take palettes) to be there for them to play with. Move to the later consoles and the amount of custom tile sizes and decode formats that even without compression detection built into programs (most start with 10, or on the DS 10, 11 or 40 and then follow known patterns from there) you would be looking at those.

I would say music is not compressed, beyond anything more realistically called a codec, in the vast vast majority of cases. Simple throughput/turnaround reasons all but guaranteeing that. Dumping tunes is then considered hard because the music was more sequence/tracker style built to be read by sometimes rather specific hardware, and possibly also with its own CPU to govern it, than some kind of issues with a compressed file.

AI scripts? Unless we are talking about some card game where they have a database of plays and similar such things then most AI I see is usually very much in the binary.

zonk47

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Making hacking easier
« Reply #28 on: April 09, 2016, 07:45:19 pm »
Back on topic, it seems like there is agreement on the need for a PC status readout. This could be in two places: a tall box off to the side of the code view which shows the last 10 or so values of the PC counter with most recent at bottom, and in the code view window itself. But if we're in agreement on this, we need to get the emulator authors' attention by going to their boards and making the requests, holding those discussions etc etc.

As for Rotwang, sir your inexperience has shown in that you made a generalization about the mathematics related to rolling when there are actually a multitude of different standardized uses for it. Being unable to see the bit status of the operands makes it very difficult to deduce the purpose of the roll without pulling up a calculator and typing the numbers in before and after. In the SaGa disassembly there are segments of 10 straight instructions or more involving nothing but binary arithmetic. SaGa also has this habit of doing 3 or more jumps in a row without even doing a comparison in between... not the easiest thing to follow in a gigantic, VRAM hogging graph like those produced by IDA Pro. IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't. As already noted, it's not an emulator and only a debugger built into one can find everything. (I guess you could use it alongside a debugger but again, the inane stuff).

Quote from: tryphon
BTW, do anyone know which algorithm IDA uses to draw its tree view ?

Seems like a (bad) tracking algorithm like that used by SoM enemies. The kind that chases you up the stairs by running into the wall below them. I suppose the boxes themselves must have some sort of virtual boundary in the periphery, which the (hypothetical) arrows run into, and it continues on like that until a straight path is found.
« Last Edit: April 09, 2016, 08:02:33 pm by zonk47 »
A good slave does not realize he is one; the best slave will not accept that he has become one.

Klarth

  • Sr. Member
  • ****
  • Posts: 484
    • View Profile
Re: Making hacking easier
« Reply #29 on: April 09, 2016, 08:00:22 pm »
I don't see any specific recommendations on how to make hacking easier.

zonk47 appears to have difficulty using debuggers and assembly. Yes, it is difficult and there is no way around it but to invest time. If you want to learn assembly/hardware for your system, then coding demos is a good investment of time. If you want to learn compression algorithms, then code compression algorithms.

Quote from: zonk47
VRAM hogging graph like those produced by IDA Pro. IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't.
That is completely untrue. IDA Pro is intended for reverse engineering. That's the entire purpose. It's particularly useful for malware analysis. If you have the source code already, then you can use a real source-level debugger. Also, the graph does not take any substantial amount of VRAM to display compared to other UI elements.

zonk47

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Making hacking easier
« Reply #30 on: April 09, 2016, 08:28:04 pm »
That is completely untrue. IDA Pro is intended for reverse engineering. That's the entire purpose. It's particularly useful for malware analysis. If you have the source code already, then you can use a real source-level debugger. Also, the graph does not take any substantial amount of VRAM to display compared to other UI elements.

It's a canvas and 2-d acceleration is all the rage (and there are drop shadows, no less) so I find that unlikely.
A good slave does not realize he is one; the best slave will not accept that he has become one.

Klarth

  • Sr. Member
  • ****
  • Posts: 484
    • View Profile
Re: Making hacking easier
« Reply #31 on: April 09, 2016, 08:46:38 pm »
It's a canvas and 2-d acceleration is all the rage (and there are drop shadows, no less) so I find that unlikely.
Maybe back in 1995.

tvtoon

  • Sr. Member
  • ****
  • Posts: 351
    • View Profile
Re: Making hacking easier
« Reply #32 on: April 09, 2016, 09:07:05 pm »
We should only bother making hacking harder, as it is already too easy! :D

Anyway, something that hasn't been done in a while, like documenting about common developers practices and kits used in games, could greatly help the community. Also trying to figure ways to detect stuff inside binary images, in a general way. Sum: improving the "science" of romhacking. :)

Rotwang

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: Making hacking easier
« Reply #33 on: April 09, 2016, 09:26:37 pm »
As for Rotwang, sir your inexperience has shown in that you made a generalization about the mathematics related to rolling when there are actually a multitude of different standardized uses for it. Being unable to see the bit status of the operands makes it very difficult to deduce the purpose of the roll without pulling up a calculator and typing the numbers in before and after. In the SaGa disassembly there are segments of 10 straight instructions or more involving nothing but binary arithmetic. SaGa also has this habit of doing 3 or more jumps in a row without even doing a comparison in between... not the easiest thing to follow in a gigantic, VRAM hogging graph like those produced by IDA Pro. IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't. As already noted, it's not an emulator and only a debugger built into one can find everything. (I guess you could use it alongside a debugger but again, the inane stuff).

I know exactly how shifting and rolling works. What crappy debugger are you using that you cannot see the bits that are being modified? I can convert binary to hex in my head and I'm sure that there others on this forum who can do that, and even if you can't, what shame is there in using a calculator in Programmer mode? It literally doesn't matter if IDA Pro is meant for engineers, and I care even less if your computer is too old and crappy to handle its average UI, because it does a wicked job of reverse engineering a ROM, and contrary to your purported "goal" it has made ROM hacking significantly easier for me. And I never said it was an emulator so I'm not even sure why you are even arguing that. I use it in conjunction with a debugger and a hex editor. What is even your complaint? That you sometimes have to use more than a single program at a time? That you have to count how many times in a row you see LSR and do 3rd grade math in your head? It's like you have this weird idiosyncrasy that people shouldn't have to use more than just one debugger or have to use their brain or learn things and it is completely baffling.
« Last Edit: April 09, 2016, 09:38:46 pm by Rotwang »

FAST6191

  • Hero Member
  • *****
  • Posts: 2499
    • View Profile
Re: Making hacking easier
« Reply #34 on: April 09, 2016, 09:48:32 pm »
For the DS that was and remains one of the main approaches used, though you might also want to look up some of the stuff Atrius did for his GBA Golden Sun editors.
Header detection, magic stamp analysis, fingerprinting of structures within it, things like 8h usually being the file size.... are all things I saw programs, did, taught and tried to note.
Unless you are wanting to go some kind of more automated pattern analysis -- if you have a 16 bit colour then most systems will tend to discard a bit to give 5 bits per colour channel, thus you are exceedingly unlikely to have F as a result in your graphics, said graphics are unlikely to be random noise so runs of stuff might also be worth a look, even with 256 colours most devs still used far fewer or at very best used some kind of shading, an emulator auto driven palette search/logging could be fun to see, the most common location for the GBA game cartridge was 08000000 through 09FFFFFF, though as things were less than 16 megs then it was 08 all the way and thus if you had a lot of 08 with 3 bytes in between then you likely have a pointer map/table/field/whatever so an automated map of the results of that, some kind of pointer analysis/limited decompilation/proto api looking for instructions that tickle certain parts of hardware and then almost immediately do something (again on the GBA a slight variation on that is used to track down the binary, possibly also parts of the sappy sound format many GBA games use), we know what valid opcodes are so some kind of statistical analysis where invalid ones are not so common, possibly the same as before but also weighted so we are not looking at the once in 10 programs type opcode is downplayed in favour of "add r1, r2",  better yet you rather have a compare without a jump and so on. Much of this would probably be more signal than noise but I do see things like it in malware analysis and obfuscation is the order of the day there. I do not see it making life much easier for anybody, certainly not newcomers, and instead be more likely to give interesting threads for already versed hackers to pull on.

Before I head too far into that world an idea I have had kicking around for a while is I want to see/do some full comparative ROM set analysis. I say the DS SDAT sound format is common and ignoring region dupes the list of things that do not use it is small, with the whole thing to look at I would know for certain and some decent stats on directory names and more besides. Vendor spam on Friday had a 6TB hard drive for not an awful lot and that would probably do nicely for what I need here. I reckon some interesting things could shake out there, indeed we see a very limited version of this in MAME where you have 6 games ever made for a given board and an interconnected analysis of the lot.

There is a slight hesitation that some might read too much into things, but at the same time I am all for common knowledge that is actually wrong to be blown away by meta analysis or some kind of controlled test.

On dev kits I am not sure how useful it would be for many of the older consoles. I have not looked much at what there is for the SNES but the megadrive one on this site is not terribly useful. Most such things I see tend to be more http://www.pagetable.com/?p=28 than truly useful.
I would also be wary of looking too much into them lest we end up doing something like the original xbox homebrew scene where they used the official SDK for most homebrew and thus could only have cloaked distribution of binaries. Nothing is likely to happen but why poison the well for no great gain.

Gemini

  • Hero Member
  • *****
  • Posts: 2007
  • 時を越えよう、そして彼女の元に戻ろう
    • View Profile
    • Apple of Eden
Re: Making hacking easier
« Reply #35 on: April 09, 2016, 09:56:25 pm »
Well the IDA version 5 seems to be freeware at their site. No idea what functionality it has as I haven't tried it.
It only has a loader called PC, which can deal with most PC processors (except x64). You can still write your own processor handler, people wrote a SNES disassembler for it before the IDA people actually came up with their own integrated solution (neither is exactly great).

Quote
What is this tree view exactly and what makes it so useful?
It's pretty much a mode that visualizes code flow in order to give a better view of what the original code might have looked like. It can be helpful sometimes, while in most cases it's more of a nuisance. It all really depends on how you proceed in gather information on specific pieces of code. Text view still has arrows and stuff to tell you where the code is going, more or less like Kingcom's solutions.

IDA Pro is intended for the microcontroller designers and electrical engineers who buy it, not ROM hackers who, as you mentioned, don't.
It is intended for any type of disassembling really. The scope is much broader than that.
I am the lord, you all know my name, now. I got it all: cash, money, and fame.

zonk47

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Making hacking easier
« Reply #36 on: April 10, 2016, 01:42:46 am »
Gemini did you really pay $1200 for that thing?

I'm editing vb8086's debugger to include some new features, particularly those discussed in this thread. No ETA on that because nobody seems willing to help, although I did ask, didn't I? Nor is it unlikely that certain people who when asked to help refused will benefit from it... but what should be expected from a bunch of global warming deniers, really? Well, that's not fair to all of you, I guess people are busy after all. Busy enough to waste time writing crap for half an hour to try to get off the hook for helping out, although I guess some people do get a perverse thrill over behaving like solid walls.
« Last Edit: April 10, 2016, 03:01:44 am by zonk47 »
A good slave does not realize he is one; the best slave will not accept that he has become one.

Gemini

  • Hero Member
  • *****
  • Posts: 2007
  • 時を越えよう、そして彼女の元に戻ろう
    • View Profile
    • Apple of Eden
Re: Making hacking easier
« Reply #37 on: April 10, 2016, 07:25:51 am »
Gemini did you really pay $1200 for that thing?
The full license is over 3.000 USD actually. Also no, a private donor usually fetches me their personal copy.
I am the lord, you all know my name, now. I got it all: cash, money, and fame.

BlackDog61

  • Hero Member
  • *****
  • Posts: 784
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: Making hacking easier
« Reply #38 on: April 10, 2016, 04:34:29 pm »
I must admit... I'm quite taken aback... it looks like nobody here means to help me achieve these goals, so in self-respect to myself I am frankly hesitant to try advancing them any further. Is your pride really so great, that you are unable to make these small sacrifices for the good of the community?
Zonk47,
It's funny. You've not even considered the possibility that we simply disagree with your approach?
So far, the proposal you've made doesn't make me think it will really help. It will take along time to reach a result, if we're still talking about automatic translation. Sorry, but that's how I feel.

I am interested in discussing other options, though.

KaioShin

  • RHDN Patreon Supporter!
  • Hero Member
  • *****
  • Posts: 5697
    • View Profile
    • The Romhacking Aerie
Re: Making hacking easier
« Reply #39 on: April 10, 2016, 04:40:37 pm »
zonk, people would be much more sympathetic to your ideas if you'd stop making completely unsubstantiated claims with no basis in reality. You simply don't know what you are talking about most of the time. Which would be fine if you'd accept that and asked instead of proclaiming your made up versions of reality simply as fact. In this thread and the other once there are dozens of things you said that are quite simply hogwash. You do not know how to use debugging tools properly. What you want are workarounds for your gaps in understanding and experience on how to debug. What's needed against that isn't different tools, it's better documentation and tutorials.

And going ad hominem ("global warming deniers") isn't going to help your case either.
All my posts are merely personal opinions and not statements of fact, even if they are not explicitly prefixed by "In my opinion", "IMO", "I believe", or similar modifiers. By reading this disclaimer you agree to reply in spirit of these conditions.