News: 11 March 2016 - Forum Rules

Author Topic: Making a N64 String Editor  (Read 8603 times)

Iron Knuckle

  • Jr. Member
  • **
  • Posts: 7
    • View Profile
Making a N64 String Editor
« on: October 18, 2012, 05:16:14 pm »
Hello, I am wanting to design a set of tools with C++ for hacking an N64 game. I don't have any experience with C++, but I do have the will to learn. I thought the best way to start out was with a simple string editor for the game. After I learn more about working with C++, I plan on stepping up the level to texture, model and map editors/viewers/extractors. Though that'll take a while, so I was hoping that someone could walk me through the steps of making a string editor.

First off, I have some questions:

The major tools that I've seen are for the Zelda OOT Debug Rom. The Rom's size is double that of the game that I want to edit. Is the level of compression going to make it harder for me to program tools for it?

The string editor for the Zelda Debug talked something about Rom "Injection". Does this mean it can move all of the offsets without messing up the game? If so, I'd love to know how that works.

Uh, well, my last question is: Can someone help me out here? I'd really love to start on this. Sorry if this was posted on the wrong board.

Zoinkity

  • Hero Member
  • *****
  • Posts: 565
    • View Profile
Re: Making a N64 String Editor
« Reply #1 on: October 18, 2012, 07:13:19 pm »
To be honest, all these questions have game-specific answers.  What title are we talking about?

That particular version of OoT is pretty much uncompressed unless I'm seriously mistaken, so the filesizes are misleading.  Filesize doesn't matter that much anyway.  There's some small Eurocom titles that are absolute beasts thanks to the nutty way they set up their games.

Zelda and the following list of titles use a particular kind of filesystem, and I find it's one of the easier ones to edit.  There's a table that uses codewords (which are the position of the binary if the whole game was decompressed) to load entries that consist of ROM addresses, filesizes, and compression flags. 
ANIMAL FOREST-NAFJ-NAFE.txt      Doubutsu no Mori/Animal Forest (Japan/North America)
THE LEGEND OF ZELDA-CZLE-CZLE1-CZLJ.txt   The Legend of Zelda: Ocarina of Time (Japan/North America v1.0/v1.1)
THE LEGEND OF ZELDA-CZLE2.txt      The Legend of Zelda: Ocarina of Time (North America v1.2)
THE LEGEND OF ZELDA-NZLE15 (debug).txt   The Legend of Zelda: Ocarina of Time Debug ROM (North America)
THE LEGEND OF ZELDA-NZLP-NZLP1.txt   The Legend of Zelda: Ocarina of Time v1.0/v1.1 (Europe)
THE LEGEND OF ZELDA-NZLP15.txt      The Legend of Zelda: Ocarina of Time GameCube (Europe)
MAJORA'S MASK-NDLE (Demo).txt      The Legend of Zelda: Majora's Mask Kiosk Demo (North America)
ZELDA MAJORA'S MASK-NZSE.txt      The Legend of Zelda: Majora's Mask (North America)
ZELDA MAJORA'S MASK-NZSE (GameCube).txt   The Legend of Zelda: Majora's Mask GameCube (North America)
THE MASK OF MUJURA-NZSJ.txt      The Legend of Zelda: Majora's Mask (Japan)
ZELDA MAJORA'S MASK-NZSP.txt      The Legend of Zelda: Majora's Mask v1.0 (Europe)
ZELDA MAJORA'S MASK-NZSP1.txt      The Legend of Zelda: Majora's Mask v1.1 (Europe)
STARFOX64-NFXE.txt         Star Fox 64 v1.0 (North America)
STARFOX64-NFXE1.txt         Star Fox 64 v1.1 (North America)
STARFOX64-NFXJ.txt         Star Fox 64
STARFOX64-NFXP.txt         Lylat Wars (Europe)
STARFOX64-NFXU.txt         Lylat Wars (Australia)

There are other kinds of filesystems in other games, and then there's some that don't use them at all.  Like I said, a lot of this is on a per-game basis.
As for compression, there's only a handful of known formats.  Relatively few use the Nintendo LZ types; the two most common would be gzip and traditional LZSS.  As a note though you'll want to compress files through gzip and not zlib, since gzip gives you a noticably better compression rate.

Iron Knuckle

  • Jr. Member
  • **
  • Posts: 7
    • View Profile
Re: Making a N64 String Editor
« Reply #2 on: October 18, 2012, 11:19:52 pm »
The game is Rayman 2. I don't think it uses a common compression. I've downloaded the MiniUSF files for it before, and here's what the ReadMe said:

Quote
BLEH! This was a very complicated game to hack! Probably the most difficult one
I will ever do. Well, maybe. ;)
The first problem I had ripping this was locating the track select routine. It
could not be found by cheat search, which is my standard method for finding the
routines. I had to start from the alCSPPlay call (MANY thanks to someone42 for
the more-complete libultra sigs -- this rip couldn't be done without them!) and
work my way back from there. I back-traced several pointer writes, probably
about 4 or 5, until I was finally able to locate the pointer table being used.
When I found that, I was then able to find the track select routine which
indexed it. And WOW! It contained 1006 pointers. After locating the track
select routine, I had to step out of _4_ subroutines before the music player
call was made. After finding out which of the ~150 JALs was the MusicStart
routine, I had to go back through all of the others with NOPs to find the ones
which weren't needed. Unlucky me, NONE of them were needed, so that whole
process wasted a good hour and a half.
After getting that far, I began to test all of the audio indices, and wouldn't
you know it, it was a random mess of sequenced music and PCM sound effects!
(This is the reason I could not find a music variable with a simple cheat
search.) :(
Eventually, I was able to compile a list of all music indices, after playing
all 1006 in a row. I then wrote a small program that used this list to
rearrange the pointer table so all the music pointers where right at the
beginning of the table. Finally, I was able to do a standard rip with usfPJ64!

I don't know if this helps me in figuring out the compression style, because I don't know it it's the compression of the game that made it hard to rip the USF files or if it was how the music was stored. Anyway, it sounds like it'll be a hard game to crack.

Zoinkity

  • Hero Member
  • *****
  • Posts: 565
    • View Profile
Re: Making a N64 String Editor
« Reply #3 on: October 19, 2012, 02:06:54 pm »
Sound is its own beast, and the USF format is actually a super-minimized ROM file and savestate so the game plays its own sound for you o.0.  To be fair though it has 137 midi sound files (actual midi, not derivative), on top of the sound effects files so it's not like they didn't have their work cut out for them. 

Most of the game is uncompressed, if not all of it.  The issue is that it uses a series of offset tables and hardcoded pointers to load them.  The system's pretty complicated, and there's certainly enough tables.  The beginning of the ROM is simple enough, prior to that nonsense.

0x0   bin   bootstrap.bin
0x1000   bin   system.bin
0x1DCC0   bin   game-1.bin
0xC5BF0   bin   game-2.bin
0xD0A20   bin   fontbase.dl
0xD0B40   bin   B1_soundtable-1.ctl
0xD75B0   bin   B1_soundtable-1.wbk
0x1EF398
# This handles the other B1/S1 soundtables.
0x1EF3A0   bin   soundtable.seq
0x20E62C   bin   B1_soundtable-2.ctl
# This table links midi format files which can be extracted by removing their header and saving as .mid
0x22D50C   bin   S1_miditable.bin
0x2E6425
0x2E6428   bin   B1_soundtable-2.wbk
# Table header: 0xC
# 0x0   4   (float) multiplier; 1/value for actual
# 0x4   2   #entries
# 0x6   2   
# 0x8   2   ->80021E50
# 0xA   2   ->800E73BC
# Entries:   0x8 each
# 0x0   4   base table offset
# 0x4   4   size of segment loaded
# Offset +0x624 isa  table of i4 images.
# Offset +0x199C is a table of c16 images.
# Offset +0x1CB4 is more complicated; headers 0x24 each for indexed images.
0x7882A0   bin   images.tbl
0x7E5E5E   bin   images.bin
0x9E505E   bin   palettes-3.bin
0x9E577E   bin   images-3.bin

0x1050B60   bin   text.tbl

0x10883F0   bin   text.bin

# Offset table (via 80001A70); first word is number of entries
# Entry format: 0x8 each
# 0x0   4   base offset to data
# 0x4   4   #entries in data (<<3 for size)
0x1A98DB0   bin   
0x1D2E8C0

The text table is even annoying, broken into several segments.  The start of that table has a 2-byte ID/flag register followed by a 2-byte string length.  Somewhere in there should be a pointer or offset table as well, but it isn't blatantly obvious.  In fact, I'm a bit sketchy as to where it really starts; most of these tables are referenced via tables, so a simple address half search isn't going to work well.

Text itself is nifty, since it uses very simple escape codes.  /W#:, /H#:, and /Z#: specifies position, /C centers, everything else is displayed.

Great game, Rayman 2, and it really was and is underappreciated.  One of the largest titles on the system too, so your work is cut out for you.  No compression though!

Iron Knuckle

  • Jr. Member
  • **
  • Posts: 7
    • View Profile
Re: Making a N64 String Editor
« Reply #4 on: October 19, 2012, 03:55:01 pm »
...Did you find that all out by yourself :o

Alright, I'm referring to the Zelda Debug again, sorry: Someone made an extractor where it took all of the tables (I guess tables, I'm not very familiar with how the N64 Roms work) and separated them into different files. Would that make it easier to edit the game? If it's uncompressed, I should be able to make a working extractor... right?

Zoinkity

  • Hero Member
  • *****
  • Posts: 565
    • View Profile
Re: Making a N64 String Editor
« Reply #5 on: October 19, 2012, 05:12:28 pm »
Didn't really look in-depth with Rayman, but yep, that should be accurate.  Just to warn you though it is a totally different kettle of fish than Zelda in many ways.  It even uses a different microcode.

Let's talk Zelda though.
Zelda debug has a filetable spanning from 0x12F70 - 0x19030.  The different versions are at slightly different addresses.  Compressed files also have a clear indicator, b'Yaz0', which is a stream-oriented LZSS variant.
The filetable follows this format:
Code: [Select]
0x0 4 start codeword
0x4 4 end codeword
0x8 4 start ROM address
0xC 4 if compressed, end ROM address

  This works in a couple ways.  If the end ROM address is set, the file is known to be compressed.  There are a couple games that use this filesystem but how they determine filesizes differs.  Some use the difference in the codewords, others the difference in sequential entries, and some will only use the decompressed size and sequentially read 1-2 kb chunks of the comrpessed files until they match expected.
  The 'codewords' are used in functions to look up the files.  They also happen to match the ROM addresses each file would be located at if the entire contents of ROM were decompressed.  Some games also factor in memory allocated after the file into this value, but not sure if Zelda does specificly.
  All of the game's files appear in this table, including the table itself. 

  If you look at the OoT debug ROM's table, you'll notice nothing is compressed.  Also, the debug ROM can't run on actual N64 hardware because, to date, nobody has ever made a cart (WROM or EEPROM) that is larger than 512Mb.


Dealing with individual files is much easier in the end, but different games are structured radically different.  Some include full filetables with original filenames; some use multiple forms of compression; some have compressed filetables, and in some cases this compression can differ from the ones used in-game; many use multiple tables or include additional tables for different kinds of resources; one uses a giant PKZip folder; some have no table whatsoever, relying on the code to explicitly call each resource.
For a reference on different kinds of filetables, here's a link to a bunch of miniextractors I've used to make Midwaydec's filetables.  Certainly not comprehensive, but should help a little to understand how different games are structured.
http://www.mediafire.com/?flfeygyprab5253

Also, both SubDrag and I have made decompressors for N64 titles and the source is available for each.  His is far better coded honestly.  His is available here:
http://goldeneyevault.com/viewfile.php?id=213

Midwaydec is poorly-coded in python and sporadically updated, available here:
http://www.mediafire.com/download.php?zije4b8bbc15wvb

Iron Knuckle

  • Jr. Member
  • **
  • Posts: 7
    • View Profile
Re: Making a N64 String Editor
« Reply #6 on: October 20, 2012, 03:06:52 pm »
So when I loaded the Rayman 2 Rom in that extractor program and it gave me all of those addresses, those are the different tables in the Rom? Wow, that is a lot of them. I highly doubt it, but are the tables for separating, for example, the stats of a Robo-Pirate, the stats of Rayman, all of the vertexes of a certain map or model, and stuff like that? I'm sorry for all of the questions, I've just always wanted to learn how Rayman 2's Rom worked so I can start making programs to hack it.

Zoinkity

  • Hero Member
  • *****
  • Posts: 565
    • View Profile
Re: Making a N64 String Editor
« Reply #7 on: October 20, 2012, 04:03:31 pm »
The soundtools are only loading soundbanks.  The rest of the data is unaccounted for. The issue is that nobody's made up a complete ROM map for the game yet, of the form:
Quote
0x0   bin   bootstrap.bin
0x1000   bin   system.bin
0x1DCC0   bin   game-1.bin
0xC5BF0   bin   game-2.bin
...
So in fact you don't have a complete set of tables, and no extractor, as of yet, will automatically work that out.  Sadly, you'll probably have to disassemble the ROM to build that list yourself.

There's obviously the text table, and that really will let you split all the strings and (theoretically) reorgainize/resize them.  There's also the image tables, and you should easily be able to parse out the simple non-indexed images without much effort.  The indexed ones use a more complicated table, but again somewhat trivial.

Without knowing how the game is set up stats may be hardcoded in ASM or could be in a table.  I'd lean toward the first over the second, but without disassembly it's hard to know for certain.

Then you get into something interesting.
If you notice, there's no actual display lists besides a token base for the font.  That's because this title, like Eurocom titles, generates its display lists at runtime from instruction lists.  Naturally, that also means they aren't using the typical N64 vertex tables either.  Usually they'd be set up like:
Code: [Select]
0x0 2 xpos
0x2 2 ypos
0x4 2 zpos
0x6 2 RESERVED
0x8 2 s (horizontal image mapping position)
0xA 2 t (vertical " " ")
0xC 4 vertex color or normal depending on mode
Without knowing where the build instructions are it's impossible to know what the data format's going to look like.  For instance, Eurocom's microcode used floating-point numbers for vertices, omitted vertex color except as a special case, had a table of shorts for image mapping, and a bunch of strange overrides to play tricks with palette swapping.  It was a real pain to turn it into 'normal' N64 display lists.
So, this could either make it easier or much harder to eventually rip/view/replace models.  Either way, it's what they used and you're stuck with it.


You'll need to do the disassembly work anyway since replacing tables will require changing the addresses they're called from in ASM.  If you move something or expand something, everything that shifts will need to be readdressed.  That's an easy thing to automate though.


Anyway, this game's never been poked much so you'll have to disassemble a fair amount of it yourself.  It's not going to do the work for you ;*)  The real trick is to use a debugger and breakpoint on each table access.  Watch how they call each other, what the offsets are for each table's subsequent calls, and from the data they use determine their uses.
It's a large but well orgainized game and shouldn't give you too many problems. 

Iron Knuckle

  • Jr. Member
  • **
  • Posts: 7
    • View Profile
Re: Making a N64 String Editor
« Reply #8 on: October 21, 2012, 04:57:45 pm »
So, what exactly do you mean by disassembling? I've used a R4300i disassembler before (Renegade 64).

I've learned some pretty cool stuff taking a look at the text table. There is some stuff that leads me to think that there's a debug menu. There are what looks like a list of "Plans", each having a unique name and a Plan #, some having #a and #b. They might be cut-scenes, because some were named closely to the post-scene of the first Grogloth battle, like "Jumps", "Lands", "Surprised", "Everything Falls", "Rayman Approaches", and "Falls".
Also, at the very end it says something about a mini-game called Menezis. I looked it up and it's only played on the newer versions, like DreamCast and PC. Interesting. Maybe it was integrated into the N64 version but never used. I wonder if there would be a way to activate it?

Zoinkity

  • Hero Member
  • *****
  • Posts: 565
    • View Profile
Re: Making a N64 String Editor
« Reply #9 on: October 21, 2012, 09:03:55 pm »
Could be.  Could be a red herring too, rolled over from general source code.  You'll have to poke at it and find out.  Easiest way would be to look for any uses of those strings; look for something using the indices for the instructions, for instance, then backtrace the functions.

Iron Knuckle

  • Jr. Member
  • **
  • Posts: 7
    • View Profile
Re: Making a N64 String Editor
« Reply #10 on: October 24, 2012, 01:50:17 pm »
EDIT: Ah, I think you're right. In the text table there is also "Setup Keyboard" and "Setup Joystick".

Alright, so back to making the string editor. Would it be best to make the string editor extract the text table into a single file, edit it, then inject it into the rom?
« Last Edit: October 24, 2012, 02:08:05 pm by Iron Knuckle »

Zoinkity

  • Hero Member
  • *****
  • Posts: 565
    • View Profile
Re: Making a N64 String Editor
« Reply #11 on: October 24, 2012, 07:30:23 pm »
Yep, at least in my experience.
Just keep in mind if the injected file's size is larger than it was to begin with you'll have to change offsets for later files.  (Sizes too if they're explicit)

For future reference:
There's two lines of thought on this, one being "edit the stuff and tumble it back it" or "put new stuff at the end of the ROM and relink".  Given the ROM can't grow larger without becoming console-incompatible (and most emus too) you'll have to, but with other games you get that choice.

Iron Knuckle

  • Jr. Member
  • **
  • Posts: 7
    • View Profile
Re: Making a N64 String Editor
« Reply #12 on: October 27, 2012, 02:02:50 pm »
Are all of the FF values at the very end of the rom the remaining space? If so, I put it all through a calculator and there is 360.724 Kilobytes of FF values at the end.

Now, I just need to learn how to make the program in C++... Do you know of any open source string editor that I could look over?

Zoinkity

  • Hero Member
  • *****
  • Posts: 565
    • View Profile
Re: Making a N64 String Editor
« Reply #13 on: October 27, 2012, 05:47:51 pm »
It's all unused space.
Chip sizes are all multiples of 2, so there's usually a little (or sometimes a lot) of unused space at the end.  0xFF is the original bit settings, and when written then unset them.  A couple companies would pad their ROMs to the actual size with 0 or a pattern, but most don't.

Don't know of one in C++, and my Animal Forest and GE text converters in C wouldn't help any due to the peculiar way they stored their data.  (Plus, it's badly coded)  I could probably rewrite the GE on in python and it would be 1/10th the size.
Might want to look at any string editor, regardless of system.