GBA and DS rom hacking documentation

[FAST6191]

As I made fairly extensive use of RHDN's database of utilities, documents and more to make it/linked in it common courtesy says I should probably link it up here.

Background- I largely rewrote a few earlier documents I made on GBA and DS ROM hacking and just made a preview release of it (it is still far from finished but it hopefully has enough for people to be getting along with in the meantime).

You can grab it from http://trastindustries.com/randomfiles/romhacking2016_copy_1.pdf with this page and https://gbatemp.net/threads/gbatemp-rom-hacking-documentation-project-new-2016-edition-out.73394/ being the main places I note any updates to it.

An HTML version (I had to play to the static nature of PDF and put images where I would usually put videos which is not ideal for some things) and my source document should be along before too long.

[henke37]

Just use the submit document form dude. It's what I did.

[FAST6191]

It is still very much a work in progress so I avoided submitting it. I have no objection to it being submitted but I doubt it says that much more than what is already there/well known/available at this point.

However contents list (file formats mostly not there and part III is still somewhat unfinished)

Spoiler

[title:Contents list]I Introduction 11
II Rom hacking concepts 14
1 Basics 14
1.1 Hexadecimal 14
1.1.1 Representation 16
1.1.2 BCD (Binary coded decimal)   17
1.1.3 Big and little endian    18
1.1.4 Signed values, floating point and fixed point  18
1.2 Hex operations 23
1.2.1 Shift 23
1.2.2 Rotate 24
1.2.3 Flip 24
1.2.4 Boolean logic 25
1.2.5 Hex Mathematics    27
1.3 Patching and patch making    28
1.4 File systems and operations    29
1.4.1 GBA 30
1.4.2 DS 30
1.4.3 GC (gamecube) 31
1.4.4 Wii 32
1.4.5 Xbox 32
1.4.6 Xbox 360 33
1.4.7 PS1 and PS2 34
1.4.8 PS3 34
1.4.9 PSP 34
1.4.10 Saturn 35
1.4.11 Dreamcast 35
1.4.12 Amiga 35
1.4.13 PC and related hardware   35
1.5 Finding the object of your interest   36
1.6 Abstraction 37
1.7 Tools of the trade continued    38
1.7.1 Hex editor 38
1.7.2 Tile editor 50
1.7.3 Spreadsheet and command line   54
1.7.4 Compression 56
1.7.5 Music 57
1.7.6 ASM 58
1.8 Basic file format concepts    62
2 Graphics 64
2.0.1 Aliasing 65
2.0.2 Haloing 66
2.0.3 Bit depth 66
2.1 Palettes and colours 66
2.1.1 GBA colours (15 bit)    66
2.2 Tiles 67
2.2.1 1Bpp 67
2.2.2 4 Bpp 68
2.2.3 8Bpp 68
2.2.4 GBA3 Xbpp 69
2.2.5 GBA2 4BPP 71
2.2.6 Bitmap 74
2.2.7 Known formats 74
2.2.8 Crystaltile2 export and import   74
2.2.9 Avoiding gradients, AA, lossy/noise and such things 79
2.3 Layout, timing, OAM and special effects   79
2.3.1 Introduction to the OAM and BG modes  79
2.3.2 Timing 80
2.3.3 GBA and DS OAM (sprites)   81
2.3.4 GBA and DS BG modes    84
2.3.5 Basic animation 88
2.3.6 Window feature 93
2.3.7 Special features (flipping, affine transformation, alpha and such) 93
2.3.8 Basic DS layout formats and mapping   95
2.3.9 Video memory handling and alignment   98
2.4 3d  98
2.4.1 Basic 3d (bones, coordinates, keyframes)  99
2.4.2 Viewpoints 101
2.4.3 Textures and material colours   102
2.4.4 Models 103
2.4.5 Lighting/shadows    104
2.4.6 3d smoke and fog 105
2.4.7 Animations 106
2.4.8 DS 3D hardware 107
2.4.9 The shift of the 3D to DS 2d   109
2.4.10 NSBMD 109
2.4.11 Non NSBMD 119
2.5 Notes and further reading    120
3 Text 121
3.1 Tables 121
3.1.1 Relative searching    123
3.1.2 Corruption and alteration    130
3.1.3 Memory viewing and corruption   132
5
3.1.4 Frequency analysis    134
3.1.5 Language analysis    136
3.1.6 Pointer and encoding/hex analysis   137
3.1.7 Assembly tracing 137
3.1.8 Font viewing 137
3.1.9 Language comparing    138
3.1.10 Table creation tools    138
3.2 Pointers 141
3.2.1 Special cases and non pointer concepts   142
3.2.2 Example reverse engineering of pointers  143
3.3 Markup, control codes and placeholders   148
3.3.1 Worked example 148
3.4 Fonts 153
3.4.1 NFTR 154
3.4.2 Common hacks 160
3.5 Scripting and layout 163
3.5.1 Layout and limits 171
3.6 Text extraction and insertion    172
3.6.1 Text extraction 172
3.6.2 Text insertion 175
3.7 Language detection in DS games    177
3.8 Translation hacking 178
3.8.1 The types of Japanese characters and how they work 179
3.8.2 Japanese glyphs/characters and observations on the language 182
3.8.3 On language 183
3.9 Japanese text editors and translation tools   184
3.9.1 General Japanese capable text editors   184
3.9.2 Rom hacking tools    184
3.9.3 CAT tools 185
4 Multimedia 186
4.1 Sound 186
4.1.1 SDAT (NDS) 190
4.1.2 Others 198
4.1.3 Tracker formats 199
4.1.4 General rule of thumb for custom audio formats  200
4.1.5 Common DS SDAT audio hacks (undubbing, injection,
tweaks and relinking)    200
4.1.6 GBA audio 220
4.2 Video 224
4.2.1 General video theory    225
4.2.2 Mods/VX/act imagine by Mobiclip  226
4.2.3 RAD/Bink 226
4.2.4 Criware 227
4.3 Cut scenes 227
6
5 Game logic 229
5.1 Levels and Stats 229
5.1.1 Example tools 231
5.1.2 Level editing techniques    232
5.1.3 Stats 243
5.2 Compression 244
5.2.1 Lossy 245
5.2.2 Lossless 245
5.2.3 Basic theory of the actual implementations  246
5.2.4 Compression at hexadecimal level   251
5.3 Cheating 254
5.3.1 General cheat making    255
5.3.2 GBA cheat making    256
5.3.3 DS cheat making 258
5.3.4 Basic making of a cheat    262
5.3.5 Cheat prevention methods and frustrations  265
5.3.6 Instruction editing cheating    269
5.4 Programming concepts 272
5.4.1 Functions and procedural programming  272
5.4.2 IF ELSE 273
5.4.3 Recursion 273
5.4.4 Iteration 273
5.4.5 Loops 274
5.4.6 Turing complete 274
5.4.7 Fundamentals of Assembly    274
5.5 Assembly 278
5.5.1 ARM 279
5.5.2 GBA Assembly specifics    280
5.5.3 DS Assembly specifics    285
5.5.4 The GBA and DS compared   289
5.5.5 On controls 290
5.5.6 Hooking 291
5.5.7 GBA cart as extra memory for DS hacks  292
5.6 Non specific assembly discussion   292
5.6.1 Language mod example    292
5.6.2 Non code in ASM    296
5.6.3 Destructive vs non destructive assembly editing  296
5.6.4 Polymorphic and dynamic code   297
5.6.5 Slowdown and speedup    299
5.6.6 Cryptography (encryption, checksums and signatures) 300
5.6.7 Save editing 306
5.6.8 Interpreted languages    307
5.6.9 Game AI, game logic and game theory   307
5.7 Flash cart and emulator theory    311
5.7.1 GBA 313
5.7.2 DS 315
7
5.8 Rom hacking "protection"    317
III Examples, oddities and techniques319
6 Crystaltile2 general usage guide 319
7 GBA tracing 325
7.0.1 Worked examples 326
8 DS tracing 326
8.1 Cart read command 327
8.1.1 Basic lookup and methods for it   327
8.1.2 Header reverse engineering/generated values  327
9 Reverse engineering various ROM images 327
9.1 Large archive on top of filesystem    328
9.1.1 Tony Hawk 328
9.1.2 Star Wars - The Force Unleashed   328
9.1.3 El Tigre Make my mule    328
9.2 Compression 328
9.3 First Person Game 329
9.4 Platformer 329
9.5 Fighting games 329
9.6 Role playing games 330
9.7 Racing games 330
9.8 Puzzle 330
9.8.1 Mahjong game 331
9.8.2 Tetris 331
9.9 Other genres 331
10 Developer leftovers 331
11 Workarounds 332
12 Moving to a new system 332
13 Developer tricks aka thinking like a game developer 333
13.0.1 Level and mechanism design    334
13.0.2 Sprite and palette reuses    335
13.0.3 Pre rendering 335
13.0.4 Speed blur and fog    335
13.0.5 Loading covers 335
13.0.6 Optimisation of loading    336
13.0.7 3d imagery in general    336
13.0.8 Procedural generation    337
8
13.0.9 Noise on images and soundSome systems even allow for
this 338
13.0.10Using the limits of the system/working to them  338
13.0.11Network coding 339
14 Game design and media 339
15 Python, batch files and programming for rom hacking 340
15.1 ROMulan Data Extractor/Injector    340
15.2 Programming languages 341
15.3 Python 341
16 PC program hacking 342
16.1 Debugging 343
16.2 Decompilation 343
17 Version control and project management344
17.1 Project and team management    344
17.2 Version control 346
18 Interesting links and further reading347
18.1 Links 347
18.2 Further reading 347
IV File formats (specications, methods and known
formats)349
19 General things about the DS 349
20 Generic DS nitro SDK format 349
21 General file reverse engineering 349
21.1 Headers 350
21.2 File sizes 350
21.3 Multiple versions of the game    350
21.4 File names and extensions    350
21.5 Tile viewers 350
21.6 Pointers and such 351
22 Sound 351
22.1 SDAT 351
22.2 SSEQ 352
22.3 STRM 352
22.4 SWAR 352
22.5 SWAR 353
22.6 BANK 353
9
22.7 Other formats 353
23 Graphics 354
23.1 NCER 354
23.2 NANR 355
23.3 NCGR 355
23.4 NSCR 355
23.5 NMCR 356
23.6 NFTR 356
23.7 NSBMD 356
23.8 NSBTX 357
23.9 NSBCA 357
24 Packing format 358
24.1 NARC, ARC and CARC 358
25 Text 358
25.1 BMG 358
V Glossary, index and such 360
26 Glossary 360

[close]

[Klarth]

Pretty impressive what you've gathered/written so far.  I skimmed through and the biggest thing that jumped out is that your margins are too large for my preference, especially the horizontal.

[FAST6191]

Yeah I just left it with the LyX defaults as far as margins, fonts and most of everything else in the layout department went as it usually works for me. I did fire off a version with larger margins and although I will probably have to redo most of the image sizes as they were relative to text width (by the way others reading many are shrunk to try to maintain some readability but they should still be full res so you can zoom in without having to stare at pixels) and some of my page breaks it did look a lot better.

Typos and clarifications aside most of part II is frozen and the rest will hopefully not take that long so I should have something resembling a proper layout for the next version.

[henke37]

An interesting read, but about section 2.3.6. (Windowing), you should discuss the fact that it's legal to change the window settings mid frame. Best example of this IMNHO is all the spotlight scenes from Ghost Trick. But I suppose the keyhole animation from Super Mario World works too.

[FAST6191]

Thanks henke37 I added it to the second preview release I just made, other than the windowing stuff there is nothing much new but it did see me play editor for several hours which should have got most of the typos and suspect grammar/disjointed phrasing and fix a few ordering issues.

The original link should work but http://filetrip.net/nds-downloads/utilities/latest-gba-and-ds-rom-hacking-guide-f30026.html has it if not.

[henke37]

I think you might have misunderstood me a bit. When I said mid frame I didn't mean it is a good idea to change things while rendering a scanline. I meant that it is possible to do a sort of realtime shape drawing by changing the size of the window each scanline during HBlank. Affine transformations can also be changed for each scaneline during HBlank. That's what the classical "mode 7" does.

This image shows the spotlight effect that Ghost Trick likes to use. The game widens the window each scanline and as such gets this trapezoid shape.

[FAST6191]

Yeah I really should not just edit things on the fly and I feel most of the more advanced graphics section probably would be better read as go to http://www.coranac.com/tonc/text/video.htm and http://nocash.emubase.de/gbatek.htm at this point.

More editing ensues.

[FAST6191]

Possibly pushing the "must keep threads updated" requirement but hey.

New version out, mainly to fix the links now no$gba/GBAtek changed domain but I have also played editor a bit over the last couple of years, anything not finished last time has probably not been updated. Likewise several tools have been updated (vgmtrans is now being supported by loveemu and has a couple of versions out, desmume has had some great updates and no$gba also had some updates, though I have changed parts to reflect the debugging version now being freeware) so that is going to be a bit behind the times. The move from Windows to Linux brought with a bit bunch of headaches when it came to getting it so Lyx could export it (not quite as bad as some Microsoft office horror stories we probably all have but similar social circles), I am also expecting a bit of fallout from that.

The filetrip link in the first post should get to it but http://filetrip.net/nds-downloads/utilities/download-gba-and-ds-rom-hacking-guide-2014-preview-1-f32908.html also has it. I did not change the margins in the end, despite having to fiddle with several of the images.

The link to the other domain will probably want to be updated at some point.

[FAST6191]

http://filetrip.net/nds-downloads/utilities/download-gba-and-ds-rom-hacking-guide-2016-preview-1-f33419.html
If filetrip does not work for you then try to be kind on my bandwidth please (download it rather than read it online every time) and you can get it http://trastindustries.com/randomfiles/romhacking2016_preview1.pdf
I am planning to get back into writing this over the coming months so I am not planning to spread this version far, though I am certainly not treating it as some kind of closed preview or something.

Changes are mainly google code links changed to github where applicable ( https://code.google.com/p/support/wiki/ReadOnlyTransition ), some other broken links fixed and some general tweaks and modernising things. Not a lot has been changed or added but I do tweak things from time to time on it and I would like to believe it is better.

Oh and I asked the author of monkey moore (the very nice relative search tool, http://www.romhacking.net/utilities/513/ ) whether there was a new site for it (it was on google code) and he said https://github.com/rjricken/monkey-moore was the new one. A basic search did not bring that up so I will link it there to help with the spreading of it. If someone wants to update that rhdn link with the new site that would be nice as well.

[DEDEDE2]

Thank you. 　I am grateful to you.
I don`t have understood ROM hacking, But I was helped you.

Thank you very much!