News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Imperial Exchange - Generic ROM editing tool  (Read 4384 times)

Wa

  • Jr. Member
  • **
  • Posts: 3
    • View Profile
    • Logic Place
Imperial Exchange - Generic ROM editing tool
« on: July 09, 2013, 08:30:21 am »
Greetings!

I've been working on this project for over two years now (of course most of that time was not spent actually working, and half of it was a complete rewrite), but it's based on my own limited experience with ROM hacking and what kinds of things I needed as I went. However, I don't think it will need fundamental changes, at least, but it could certainly use more features.

Anyway, I'm a member of Team PokéMe so my only hacking (in terms of consoles) has been on the Pokémon mini and Adventure Vision. So nothing particularly advanced or recent. I've had some tips from them, but I would like input from the ROM Hacking community at large as to what sort of things they need from this project.

You may see the code here.

Now to explain it... though the wiki gives some information. The idea for the project is to have a more or less simple descriptor file (a RPL) that describes a ROM (or any binary file) in terms of what its contents are; be they graphics, code, sound, etc. This descriptor format is then used (by imperial.py) to transfer that data between the ROM and your computer in standard formats, so that you can edit them and easily put them back in. Currently it supports graphics and data (text, binary, pointers, level descriptions, whatever). After 1.0 I will be working on an assembler system for it, primarily intended for partial code modifications. I have much of the assembler already written, it's just a matter of updating/finishing and including it. Then sound, though I expect that to take a while.

The system is basically at 1.0, in terms of the features I wanted it to have for this milestone. I'm holding back on the official 1.0 release because there are still some minor things I would like to finish, as indicated by the countdown in the readme, currently.

In terms of the distant future, I do intend to add a de/compression and de/encryption system as well as support for 3D models, but I have not studied models sufficiently, yet. There will be no support for generic compression/encryption, obviously, that's too ludicrous even for me. I also plan to add some extra tools to it that utilize the system in different ways.

What I want to ask is, what am I overlooking? Other than adding modules for completely different systems, do you guys have any suggestions? I do intend to make more modules myself (namely Adventure Vision) but am more focused on the main code and documentation right now. I hope that other people might consider contributing system modules. However, I encourage you to answer me in respect to other systems. For instance, "Your system wouldn't be helpful for X console hacking because it lacks Y."

You are also welcome to ask me questions, I realize this is rather vague.

FAST6191

  • Hero Member
  • *****
  • Posts: 3182
    • View Profile
Re: Imperial Exchange - Generic ROM editing tool
« Reply #1 on: July 09, 2013, 10:09:55 am »
Such an idea has been floated in the past (my last pondering of it -- http://www.romhacking.net/forum/index.php/topic,16070.0.html ) though getting something done is nice too.
The main problem with this sort of project is it tends to end up either
a) Complex enough that becomes a programming language unto itself. Text extraction tools are often a good example here.
b) Inflexible enough that you might as well not bother.
c) An actual programming language though probably not a good one (see something like romulan)
d) Another program with another extension format. As of the last few years and things like tinke or everything including lua or python modules then you face some steep competition here as things like tinke have serious functionality before you even start extending things.

From what you have said thus far you are probably not far enough in ROM hacking to break out of that mould. By all means make a game, system, franchise or format specific tool (we could always use more good ones of any of those) and I absolutely encourage programming it in such a way that you can easily come back and adapt it (nice function names, comments, no hardcoded variables anywhere*, variables maybe even kicked to an external source sheet, variables also means the words you use so people can translate it and so forth)

*no not even that one.
On d) from the list above. This gets interesting as we already have several standard formats. Table files (the three or four formats already out there anyway) and nobody is really going to want to accept another one.
Graphics... I already have http://code.google.com/p/tiledggd/ and it is great, however it is not so hot with some of the more exotic alpha formats seen in 3d textures so there is definitely room for improvement. Similarly some of the quasi compression/referencing tile formats that the GBA and DS support can be troublesome with that (GBA3 Xbpp and GBA2 4bpp being the names of two of them, seen in actual games as well).
Even on assembly we have fairly accepted formats in some cases. On the DS (and I guess perhaps the GBA) there is the NEF format (used by no$gba, crystaltile2 and recognised in many places, a fraction more on it http://gbatemp.net/threads/gbatemp-rom-hacking-documentation-project-rewritten-for-2012.73394/page-4#post-2782288 ) to say nothing of this is when lua often appears.

More generally. "so that you can edit them and easily put them back in" Will this include pointer support at both file level and console level? It is not so useful without it when bookmarks exist in a lot of things and copy and paste exist in hex editors or cut and copy stuff exists to use with the command line. Speaking of pointers you have several different types of them to deal with before you even come to the file formats -- the NES, the SNES, the GBA and the GBA (and DS for that matter) all have somewhat different styles of doing things (see some of the text decoders) and can be encoded oddly.

On compression http://www.romhacking.net/utilities/826/ for the GBA/DS I consider to be reference grade implementations of it. If you wanted to make a generic LZSS compression tool it would probably not even be that hard (you have window length, flag type, flag distance, split between read back and length of read, start from the end or start from the start compressions. That would probably also be the hardest give or take exotic huffman (and not so much uses huffman).

Encryption. If you can not implement a generic XOR (about the main type of encryption seen on consoles) then it does not say great things.

On assembly. I already mentioned the NEF format. It goes a lot deeper though (later posts in http://www.romhacking.net/forum/index.php/topic,16594.0.html just begin to hint at it, http://www.keil.com/support/man/docs/armasm/armasm_Cacdbfji.htm might be warranted after reading it).

"simple descriptor file (a RPL)"
A perfectly valid way of working, however we do have tools for various formats (many general things but more often than not compression) or indeed whole games (Atrius' golden sun GBA tools for the others reading) that will search the game for various formats.

As I am just dropping data and picking holes at this point I suggest you look at 12 different things plus the links I have already dropped.

1) Atlas and the normal text dumping tools. http://www.romhacking.net/utilities/224/

2) The recent discussion on kruptar. http://www.romhacking.net/forum/index.php/topic,16496.0.html

3) Tinke. https://code.google.com/p/tinke/

4) Crystaltile2. http://www.romhacking.net/utilities/818/ I have some very old source code for it as well but that is probably not so useful.

5) Romulan. http://stealth.hapisan.com/ROMulan/ROMulan.html

6) Tahaxan. http://www.romhacking.net/utilities/455/

7) Lowlines' console tool. http://www.romhacking.net/forum/index.php?topic=8407.0

8) Fceux lua. http://www.fceux.com/web/help/Commands.html (also the basis for a lot of other emulators). Whether you want to have your tools reach out and touch emulators (memory grabs, cheat making, palette/VRAM grabs and such)

9) http://transcorp.romhacking.net/scratchpad/Table%20File%20Format.txt

10) Pretty much any of the non pokemon game specific stuff (anything with Mario in, golden sun, advance wars and fire emblem are usually good bets). http://gbatemp.net/threads/mkds-course-modifier.299444/ , http://nsmbhd.net/download/ , http://www.feshrine.net/ (might take a bit of digging), http://forums.warsworldnews.com/ and http://forum.goldensunhacking.net/ being reasonable jumping off points for each of those.

11) VGMtrans. http://filetrip.net/nds-downloads/utilities/download-vgmtrans-92909-f27960.html

12) VGMToolbox. http://sourceforge.net/projects/vgmtoolbox/ and http://hcs64.com/mboard/forum.php?showthread=22580

As you have already found this is a massive project and many around here will have been around to see several wide eyed neophyte ROM hackers crack on with making "the great ROM hacking tool" only to decide they have bitten off more than they can chew. Two years, a github, a nice language like python and most of what you have said say this might get further than a lot of them but I doubt it is enough to overcome a lot of the doubts people around here might have.

Link I have not been able to fit in elsewhere http://nvwr.googlecode.com/svn/trunk/libs/

Wa

  • Jr. Member
  • **
  • Posts: 3
    • View Profile
    • Logic Place
Re: Imperial Exchange - Generic ROM editing tool
« Reply #2 on: July 09, 2013, 10:51:28 am »
Such an idea has been floated in the past
It isn't surprising, I guess. ^^

Quote
b) Inflexible enough that you might as well not bother.
I am adding as much flexibility as I can in RPL itself, and for what cannot be done there, one can make a library in Python. I think this should be sufficient?

Quote
c) An actual programming language though probably not a good one (see something like romulan)
I think of this more like JSON or spreadsheets, personally. I don't want it to be a programming language.

Quote
On d) from the list above. This gets interesting as we already have several standard formats. Table files (the three or four formats already out there anyway) and nobody is really going to want to accept another one.
Well this isn't meant to be like a table file, at least as I understand those. They're for describing text encodings, yes? I do plan to make the map struct support those in the future. I have a few table format specifications downloaded, I just haven't bothered implementing it, yet.

Quote
Graphics... I already have http://code.google.com/p/tiledggd/ and it is great
This looks very cool! I made something like this a long time ago, but not as clean. :3 It's nice to see a tool like this. The graphic struct in RPL supports all of this (I believe, only skimming the tutorial) in a separate way, and allows for some crazier formats, that are probably not used.

Quote
Even on assembly we have fairly accepted formats in some cases.
If I'm reading this correctly, then, well, I assure you I will not be making a custom format for the assembly. I intend to make accept as much as I can by drop-in. Currently, the only definition I've written for it is for a printer, though. ^^; However, when I write the definition for PM it will be written to accept assembly written for PMAS, and so on for each console and its main assembler, for whatever anyone adds. That's the idea at least.

Quote
More generally. "so that you can edit them and easily put them back in" Will this include pointer support at both file level and console level?
I'm not sure what you mean by a pointer at the file level so let me just explain how it works. You can make a graphic like:
Code: [Select]
graphic MyGraphic {
    base: $00401000
    file: MyGraphic.png
    dimensions: [100, 100]
    # Format definition here... since graphic is the generic graphic struct.
}
Then when you export, it will read the binary data from that address in the ROM and write it as a png to MyGraphic.png. Of course, it does the reverse when importing. Now if you want pointers within the ROM, you would do something like this.
Code: [Select]
data MyPointer {
    base: $00502000
    file: MyPointer.rpl
    xaddr: [number, 4]
}

graphic MyGraphic {
    base: @MyPointer.xaddr
   # Same stuff otherwise.
}
Which will allow you to point to a referenced location as the address for the graphic. You can of course stack this as needed and do some neat things with inheritance if you want. If you mean point to a certain place in the export file, you can also do that. Multiple graphics can write to the same file, and you can use the offset key to address where the image is placed in the metaimage. You can also share the data file between multiple data structures. So I think this covers your concerns?

Quote
On compression http://www.romhacking.net/utilities/826/ for the GBA/DS I consider to be reference grade implementations of it. If you wanted to make a generic LZSS compression tool it would probably not even be that hard
Certainly not! I've written de/compression tools before, and have reversed such functions from Windows programs. It's just a matter of doing it, but my focus on the project is elsewhere right now.

Quote
Encryption. If you can not implement a generic XOR (about the main type of encryption seen on consoles) then it does not say great things.
I would hardly call selecting a key for a decryption algorithm generic. ^^; By generic I meant a way to existentially describe the algorithm, like much of the rest of the project tends to do.


I will carefully study all of these links, but the ones I skimmed certainly seem interesting, especially Tinke!

Quote
As you have already found this is a massive project and many around here will have been around to see several wide eyed neophyte ROM hackers crack on with making "the great ROM hacking tool" only to decide they have bitten off more than they can chew. Two years, a github, a nice language like python and most of what you have said say this might get further than a lot of them but I doubt it is enough to overcome a lot of the doubts people around here might have.
I won't contest you. It's all quite true. Though with as much as I've done already, I don't foresee myself stopping just because I think it's too much work. I'm loving every minute of developing this.

Thanks for all the commentary. ^^

FAST6191

  • Hero Member
  • *****
  • Posts: 3182
    • View Profile
Re: Imperial Exchange - Generic ROM editing tool
« Reply #3 on: July 09, 2013, 11:54:09 am »
The NEF/assembly stuff.
It it halfway between labels (proper developer grade assembly stuff will allow you to label memory locations and whatnot to be more human readable and even use them in something almost resembling pointer maths in C -- most generic disassemblers will not do this even where it is possible), label function locations  comments and other such niceties.
If you are not doing this (and outside the DS and maybe IDA it is by no means common) then carry on.

Pointers
You have standard count from 0 stuff (standard pointers)
You have offset pointers (every pointer lines up but it will be 20h out
You have relative stuff (add the value at this location to the value of this location to get where you need to go)

Those are the standard three file level pointers though it can get quirky little things like
http://www.romhacking.net/forum/index.php?topic=12953.0 (if you prefer a slightly more visual version then http://gbatemp.net/uploads/gallery/album_143/gallery_32303_143_6437.png )
And standard maths/operations get used as well from time to time
http://www.romhacking.net/forum/index.php/topic,15605.0

Aspects of pointer maths appear back in too and that can happen on the older consoles as well. Several archive formats I have pulled apart for the DS will just include file sizes rather than direct locations. A few others enjoy having their files aligned to given boundaries (sometimes up to 100h) and it can get weirder from there.

Once you actually take it to the hardware you have various things
The GBA maps the cart to memory at 08000000 through 09FFFFFF and has several mirrors (higher waitstates) which you get to contend with. This is not so bad as the extra byte rounds it up to 8 hex digits and can be NORed out or something*.
*for games above 16 megabytes you will probably want to take 08 or the equivalent for the mirrors off as it 6 hex digits

SNES hirom and SNES lorom support ( http://romhack.wikia.com/wiki/SNES_ROM_layout ) is quite important for many around here and the absence of it from a program is very much a negative.

Banks. With the exception of the GBA (save for GBA flash carts/homebrew and one as yet undumped title) and most of the megadrive* (though there were some later games with this) most carts if they are mapped to memory will only be mapped a bank at a time** and that gets confusing as it varies somewhat between systems (most will opt for a split which appears as ? bytes which will always be there and ? bytes which can change at any time). However as you are dealing with things in a ROM image and not as it appears to the system you get to account for this.

*the megadrive is a dual processor machine though which includes a 68000 as the main chip and a z80 for sound though the z80 can reach out into the main memory as and when it needs to (it appears to it as banks after a fashion).

**your 16 bit processor only being able to address 64K after all. Speaking of things larger than you can reasonably address at byte level

If you are aiming your tool at isos then you most likely get to play with sector addressing, in some ways it is kind of similar to banks which you have probably just dealt with so that is not so bad. Often you will have a filesystem here but not always (the playstation family is noted for using raw reads on occasion).
Most filesystem stuff is not mapped to memory so you will also have to account for that in various ways (mainly that things get loaded into memory and operated upon from there).

To be fair it is trying to support all this that usually drags these sorts of projects down. The trouble is though in many ways if you are not going to rewrite my pointers (and maybe those of everything after it for filesystem/archive stuff) then your program is not as useful as it might be.

On tables
Yeah the basic table is 00=some UTF8 character(s) 01=some UTF8 character(s) and so on which works for many games. However such a thing does not get you very far (it is one of the reasons we have a handful of formats here to say nothing of iffy support between certain programs, granted that is usually just a matter of making sure you have the right newline and maybe having a blank line at the end).
I would probably not suggest going as far as implementing that proposed table format spec I linked but you will ideally want something that nips at Atlas' heels or can produce something it will respect.

Wa

  • Jr. Member
  • **
  • Posts: 3
    • View Profile
    • Logic Place
Re: Imperial Exchange - Generic ROM editing tool
« Reply #4 on: July 09, 2013, 12:36:14 pm »
most generic disassemblers will not do this even where it is possible
It's not an assembler without labels, seriously!

Quote
Pointers
You have standard count from 0 stuff (standard pointers)
You have offset pointers (every pointer lines up but it will be 20h out
You have relative stuff (add the value at this location to the value of this location to get where you need to go)

Those are the standard three file level pointers though it can get quirky little things like ... And standard maths/operations get used as well from time to time

Aspects of pointer maths appear back in too and that can happen on the older consoles as well. Several archive formats I have pulled apart for the DS will just include file sizes rather than direct locations. A few others enjoy having their files aligned to given boundaries (sometimes up to 100h) and it can get weirder from there.
All of this is currently supported. Math could stand to be better supported, though. (EDIT: Much better! Fixed it up.)

Quote
Once you actually take it to the hardware you have various things
The GBA maps the cart to memory at 08000000 through 09FFFFFF and has several mirrors (higher waitstates) which you get to contend with. ...
Any offsetting like that can easily be adjusted with math, if need be. Mirror support is something I should add, though, thanks.

Quote
SNES hirom and SNES lorom support ( http://romhack.wikia.com/wiki/SNES_ROM_layout ) is quite important for many around here and the absence of it from a program is very much a negative.
Sounds like something that would be handled in a SNES lib. Will certainly fully research and implement such things should I make that lib.

Quote
Banks. ...
The system itself does not need to realize the existence of banks for them to be handled properly. This is just an issue of using an offset. I could add a way to make it even more painless, though.

Quote
If you are aiming your tool at isos ..
I don't have plans for it currently, but it's something to think about when everything else (in terms of what types of resources are handled) is complete. It certainly would be nice to have something for making an ISO's contents more accessible. Even so, for system-specific image formats, it will likely be something for that system's library. That would not stop me from writing what I can to make implementing another image's support easier, however, I haven't researched filesystems much (unless you want to count archives...).

Quote
On tables
Yeah the basic table is 00=some UTF8 character(s) 01=some UTF8 character(s) and so on which works for many games. However such a thing does not get you very far (it is one of the reasons we have a handful of formats here to say nothing of iffy support between certain programs, granted that is usually just a matter of making sure you have the right newline and maybe having a blank line at the end).
I would probably not suggest going as far as implementing that proposed table format spec I linked but you will ideally want something that nips at Atlas' heels or can produce something it will respect.
The mapping support is pretty robust, currently. It's called mapping instead of tables cause it handles more than just text, sorry if that's confusing. >< Admittedly I haven't fully tested the extent of its robustness, but it should currently be able to support decoding even varying length character encodings like UTF8 (of course it supports UTF8 natively). However, the thing it lacks is support for loading table file formats. The data struct is currently what's meant to be used for importing scripts, but it also does not support any standard script formats. It only supports bin, txt, and rpl. So these are definitely two things I will need to add. Thanks again.
« Last Edit: July 09, 2013, 04:00:08 pm by Wa »