[N64] A question about locating textures within a rom

[orbitaldecay]

Hello All,

I want to write a texture importer / exporter for Mario Kart 64. I would like to find a way to systematically locate textures within the rom. This question might be more geared toward an emulator writer.

I am using Nemu64. My understanding is that the address of a texture in RDRAM has to somehow be DMAed into DMEM within the RSP (either the display list can be constructed by the CPU and DMAed over or constructed within RSP, not sure which). From there the RDP takes the texture address and copies the texture to TMEM (please correct me if my understanding is wrong).

My first thought was to place a write breakpoint on the SP_DRAM_ADDR_REG (0xA4040004) and trace all of the DMAs to DMEM under the assumption that I could find a pointer to the texture in RDRAM in the data that is being DMAed. Unfortunately this has not been fruitful. Is this a theoretically sound approach?

I am trying to avoid the Tile Molester approach as I don't know if there are any compressed images in the rom (and it would take entirely too long to locate most of the relevant textures this way). Is there anyone who has written a texture importer for an N64 game who could chime in?

Thanks!

[Zoinkity]

Not that simple.  Not only are the vast majority of textures compressed they're organized using a series of tables.  Also, the images themselves are raw binaries, so you'll have to interpret the display lists linked to each image table to know exactly how to read them (unless you just fudge it).

Basically, in order to insert anything you have to be able to move everything.  You also need to preserve the offsets from certain key table addresses in order for the game to be able to look anything up again.

Here's a proof-of-concept file replacement script to insert the unique Japanese textures into a USA rom:
http://pastebin.com/zvxX2SYF
-used to make this patch:
http://www.mediafire.com/download/fe1x4t49c9r2fn0/MarioKart64-Jimgs.zip

Besides the table offsets, you also need to know the structure of certain tables to tamper with them and the namespaces they're at (though this does overlap).  In the ROM they're at J-0x122DC0, Ja-0x121BF0, E-0x122390, P-0x122570, and Pa-0x122490.

Code Select Expand

0x0 4 start display lists
0x4 4 end display lists
0x8 4 start vertices
0xC 4 end vertices
0x10 4 start image table
0x14 4 end image table
0x18 4 start DL namespace [0F000000]
0x1C 4 #vertices
0x20 4 offset from vtx ROM address to geometry script
0x24 4 size of scripting
0x28 4 namespace [09000000]
0x2C 4 RESERVED

Image table is broken into two parts.  The list of images comes first and is NULL-terminated.  Note images are shared between tracks, so any change to an image requires changing its entry in every table.  Images are not in order.
The list of virtual addresses follow the images are offsets in the display list binary to each display list.  Only alter them if the DL is being altered. 
Image entries are very simple:

Code Select Expand

0x0 4 offset in image bank to data
0x4 4 compressed size
0x8 4 decompressed size
0xC 4 RESERVED
Note that the compressed size is the actual binary size and this should be--and will be--padded.

The offsets to images are relative to the start of the game's image bank.  This differs by game, and would be J-0x642940, Ja-0x64D7B0, E-0x641F70, P-0x642170, Pa-642090.
If you wish to resize or change image types, or need to automate sizing and typing for image extraction, you'll have to parse the display list binary found in the table.  Offsets to nested display lists are given at the end of the image table.

That's not all the files in the game of course.  There's also tables of uncompressed data at 0xDE7F4, compressed data at 0xDEAC0, the TKMK files, and some hardcoded things.  Also, certain resources have hardcoded references besides those that appear in the tables, like Latiku.  Refer to the python source for a list of all known references that need to be looked at for file replacement.
In general, just look at the python code.

As an example, the "64" logo can be found at 0F000B18 in all games.  It's the first image to vary.

Code Select Expand

64.64x32.c16
J 0x643458 0F000B18 000002BF 00001000 00000000
E 0x642A88 0F000B18 00000567 00001000 00000000


[orbitaldecay]

@Zoinkity: Wow! Thanks a lot for your comprehensive reply. This will probably save me a week's worth of effort (at least). What strategy did you use to locate these tables? Just tracing through code? Is it sensible to use my aforementioned strategy to locate decompressed textures in RAM and set breakpoints to find the decompression routines and work backwards from there? (I'm spending this evening trying to track down the decompression routines). More to the point, you have given me the fish, and now I am seeking the rod.

I realize the answer might be as simple as just, "Trace through code till you find it" 

Edit: Just to add a little more clarification, I've had a lot of experience with locating in-game variables. My end goal is to write a kart editor and I've managed to locate most of the tables within the rom that specify various kart properties (e.g. mass, acceleration, turning speed, etc.) But what I do not have experience with is the graphical side of things. I would like to develop a method to locate textures for other games as well that I work on in the future, similar to the way that I have developed a method for locating in-game variables.

Hopefully I am making sense. Again, thanks for your time.

[Zoinkity]

Basically just killed the audio thread and made a list of all the stuff loaded from ROM when a track started.  Once you have a few base addresses it's as simple as searching for them in ROM.  In most other cases though I just make a full filelist of everything and look for address use.  Oftentimes it's important to not only have the images but display lists that use them, so in the end it cuts down a lot of effort.

To be honest, finding textures in ROM is a somewhat game-specific enterprise.

Only compression type they use is in MK64 MIO, which is ringless LZSS with a small extension to allow for longer runs on matches.  There's about a dozen different compressors and decompressors for it.

If an image is compressed it would (99% of the time) set of a breakpoint on write.  However, uncompressed images could be DMA'd straight from ROM to the final location, and later titles could use the RSP copy microcode to DMA from a decompression buffer to final position.

[orbitaldecay]

I see. Excellent. Well, thank you again for your help. I will keep the community abreast of progress on the importer / exporter and I'll be sure to give you proper credit in the final release
August 23, 2015, 01:16:53 AM - (Auto Merged - Double Posts are not allowed before 7 days.)
For anyone who is interested / following this thread. Found a good thread on Origami64 on MK64 hacking, too. Also, some links on MIO0. The decompression routine is located at 0x800400D0 in RAM. The decompression is simple, I've written some C code to do it.

I'm current attempting to track down the texture dimensions. They make their way onto the display list somehow. Any pointers on how this happens would be great, otherwise I'll post back when I figure it out.

[orbitaldecay]

Haven't forgotten about this thread

There is a separate kart data bank located at $145470 in the US rom that contains all of the kart textures and palettes. I'm working on documenting all of the tables that reference it as I work on writing code to relocate it so I can modify the kart textures. Any interested parties can find the documentation so far at http://pastebin.com/08LRvtEf. Please note that it's a work in progress, so it will be changing frequently in the coming weeks. Comments, suggestions, etc. are welcome and appreciated.

[Zoinkity]

They use the tables at 0xDEAC0 for images and 0xDE7F4 for palettes, no?

Not complete, but better than most:
http://pastebin.com/R2LwwiJS

Already pasted at the editor project, but here's how to parse the geometry bytecode:
http://pastebin.com/QMfNHWA2

Still need to implement clamping:
http://i.imgur.com/51Q5ren.png

[orbitaldecay]

That's most of it. There's more palettes at E2340 (actually the primary kart palettes are there). I think there might be some other unused stuff floating around in the bank. Screenshot looks great! Thanks for the link

[Zoinkity]

Tracks are loaded in "clumps" consisting of DLbranches loading individual segments of track.  The idea is as you drive around it will load in only the things you're coming up on instead of trying to work out how much of everything you can see.

The point here is that the primary DL has a table of these and the offset to the table for each track is hardcoded in ASM.  For instance, the offset for Koopa Troopa Beach can be found at (0xFCD70, 0xFCD84).
You can find them easily looking for JALs to 80290DAC.  A0 is the offset.

Those are really part of larger, track-specific functions that also load the sprite tables, etc.  Subtract 0x801969F0 from each pointer to get the ROM address.

Code Select Expand

80292464 Mario_Raceway
80292920 Choco_Mountain
80292D98 Bowsers_Castle
80292F80 Banshee_Boardwalk
802933D8 Yoshi_Valley
802934A0 Frappe_Snowland (0x110)
802935F8 Koopa_Troopa_Beach
80293840 Royal_Raceway
80293A90 Luigi_Raceway
80293F3C Moo_Moo_Farm
80294410 Toads_Turnpike
80294730 Kalimari_Desert
802949B0 Sherbet_Land
80294A7C Rainbow_Road
80294B2C Wario_Stadium
80294FDC Block_Fort
80295094 Skyscraper
8029522C Double_Deck
80295310 DKs_Jungle_Parkway (0x1A0)
802954E8 Big_Donut

More awesomeness, still with no clamping:
http://i.imgur.com/uAV3A93.png
https://imgur.com/23NHqD5.png
http://i.imgur.com/NQE9P0q.png
http://i.imgur.com/HjJM5Ww.png