logo
 drop

Main

Community

Submissions

Help

Author Topic: (Genesis) Find the compressed data and figure out the compression scheme  (Read 9941 times)

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
Hello,

I'm new here and I'm not good at ROM Hacking at all.  I did NES ROM Hacking with a tutorial, but it was really boring, I'm not a fan of the NES.
I would like to hack the graphics of a Genesis ROM, Dragon Ball: Buu Yuu Retsuden.  For now, I would like to change the hair color from gold to black.
However, I can't find any graphics in the ROM with Tile Layer Pro - And it's the same for all my others ROM -, I guess they are compressed.
Here's what I'm doing for a test, with TLP, but it changes nothing:
http://download252.mediafire.com/yxj...z4w1dm/tlp.wmv

Somebody told me he was able to help my a bit with this.  Because I can't uncompress, I can't do anything at all (And I can't see any text with Translhexion.  I can't translate.).  He told me if I can locate the compressed data and figure out the compression scheme (He told it's hard for a begginer.  The hardest part.), he could write a decompressor (rip directly from the rom) and compressor (and rom inserter) for me.


So please, how do we locate the compress data, and the compression scheme?  What do I have to do first?  Which tools do I need?


Thanks for your help.  :)

FaustWolf

  • Sr. Member
  • ****
  • Posts: 259
    • View Profile
Looks like your video URL got truncated :(


Two questions:

1. Do you just want to do a palette modification, or do you intend to go further in the future and actually change the shape of the sprites/insert different sprites, etc? Sometimes image data could be compressed but the palette data might be stored separately and uncompressed. You might be able to find and isolate palette data in an uncompressed savestate or RAM dump and then search for it in the ROM, but I'm not sure whether the utilities needed to make that process user-friendly exist for Genesis games.

2. Just a ballpark guess, but to be absolutely sure, have you tried looking for the graphics with TLP's codec set to 4BPP Linear, 2 Dimensional mode? (Or does TLP even have that setting -- not entirely sure, since I normally use TileMolester for quick graphics checks).


Generally speaking, the most common approach to figuring out decompression routines is to use an emulator's debugging and tracing features to study whatever functions are being called when graphics load. I'll leave it up to others to describe how that's done exactly because I'm sure I'd get it wrong at this point.

Sometimes it's perfectly possible to do a "brute force compare" between compressed and decompressed graphics, and literally figure out how the decompression routine is transforming the data at the byte level. Theoretically a very thorough documentation of the routine in layman's terms could be passed to a programmer and the programmer could interpret and create suitable code to reproduce the documented process. However, success will depend hugely on luck -- simple compression routines based on some variation of RLE, dictionaries, or even "sliding windows" can be rather enjoyable to figure out by comparison, but when you get into more complex compression algorithms the frustration meter rises quickly, and perhaps to the point of impossibility. Most modders will probably discourage you from brute force comparison for this reason, but I've always been rather tickled by such exercises myself.

Before addressing general methods for "brute force compares" it will be necessary to have an uncompressed copy and a compressed copy of a spritesheet from the game on-hand. That, in turn, depends on whether there's currently a convenient way of obtaining VRAM (visual RAM) from Genesis savestates, or doing straight RAM/VRAM dumps from a Genesis emulator. I'm not sure if that's currently possible; if not, some quality time with a debugger will be necessary.

EDIT: In Romhacking.net's utilities archive you can find Fuzzbuzz's DebuGens and Korama's GSavestate, which look most impressive for the needs of brute force comparison! A utility like Genstracer might be recommended to you if you wanted to pursue the more usual practice of getting what you need in a debugger, though I wonder if DebuGens would suffice as well for that purpose. It appears to be quite full featured!
« Last Edit: May 25, 2010, 02:19:41 pm by FaustWolf »
Verve Fanworks: We shall shew thee here the force of our arms!

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
Thanks for the reply.

Here's the link of my video:
http://download252.mediafire.com/1ggnnoya5tyg/vtmikz4w1dm/tlp.wmv

So:
1.  For now, I'll be happy with the palette modification, like changing hair color, or clothes color, but my real goal would be extremely hard, I would like if possible, to add the caracthers of Goku with regular hair, and the normal form of the others saiyans too.  I would like to make the Saiyan version playing and the black hair.  Both.  And my ultimate goal would be to add a few caracthers, like Buu.  Don't know if this is possible, but there's sprites of Buu in Super Butouden, I guess that I wouldn't be able to copy-paste the sprites however.  I'll make Buu to have the moveset that another caracther is using too, to avoid extremely complicated manipulations.  And later on, maybe I'll try with others caracthers. :)

2.  TLP coded?  What is it? ^^" (Sorry about that).  Well, I checked the Palette-Format-SNES/GBC, and the View-Format-Genesis, but that's all I can do.


Thanks again

FaustWolf

  • Sr. Member
  • ****
  • Posts: 259
    • View Profile
Quote
2.  TLP coded?  What is it? ^^" (Sorry about that).  Well, I checked the Palette-Format-SNES/GBC, and the View-Format-Genesis, but that's all I can do.
I'd highly recommend TileMolester, if only because that's what I'm most used to. Set it to 4BPP-Linear, 1 Dimensional mode (these options are under "View > Codec" and "View > Mode") and then check the area of your ROM starting at address 0x2F360 ("Navigate > Go To..."). Looks like there might be just be some uncompressed image data there, but I'm not sure if it's enough to be one or more characters. How many player characters does the game have? In any case, to see what this is exactly, I would recommend doing a "Palette > Import From...Another File...(Genesis savestate format)." Whatever image data is sitting there has been ripped up into little chunks ("tiles") that are difficult to make out at first, but cycling through the palette data in a savestate taken from the game might shed a little light on what this is. If it is image data, the tiles appear to be 8x8 pixel blocks, so collapsing the view to one column might help you make sense out of it.

Apparently most of your image data is compressed (surprise, surprise) with the exception of the possible graphics mentioned above, some font sheets, and miscellaneous labels, which can be viewed with the right Codec settings and careful attention to the data as you scroll through.

Looks like DebuGens (linked in the previous post above) can dump palette data, so what you'll want to do is toss the extracted palette of interest into a hex editor, toss your ROM into the hex editor as well, and then use the palette data as your search criteria. You can then edit the palette to your liking provided you find it uncompressed in the ROM -- be sure to look for documentation on how Genesis palettes typically work and how they express colors in hexadecimal, since I'm not sure offhand. As far as hex editors go, Cygnus is recommended by me, but Hex Workshop is also commonly recommended -- I think Cygnus should still have a free version and Hex Workshop probably might have a 30 day trial.
« Last Edit: May 26, 2010, 03:03:16 am by FaustWolf »
Verve Fanworks: We shall shew thee here the force of our arms!

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
I tried Tile Molester, it's definitively better than TLP!
There's exactly 11 playables caracthers in the game.
I can't see any of your tiles, even with the Search.  I used the japanese ROM, but I can't see theses (I prefer the French ROM, because I can read, and the japanese ROM is going slow on any emulator, don't know why).

Quote
Looks like DebuGens (linked in the previous post above) can dump palette data, so what you'll want to do is toss the extracted palette of interest into a hex editor, toss your ROM into the hex editor as well, and then use the palette data as your search criteria. You can then edit the palette to your liking provided you find it uncompressed in the ROM.

How do I dump the palette?  I tried using CPU-Dumping-Genesis-Graphics-Dump TLP format Palletes and I get a TLP file, I open it with Cygnus, and I get a very small Hexadecimal part (Like 15-20 lines).

Quote
toss the extracted palette of interest into a hex editor, toss your ROM into the hex editor as well, and then use the palette data as your search criteria.


I tried opening the TLP file, then the French ROM, but I can't open 2 at the same time.  In this case, the ROM is opened and the TLP file  is closed.




If this is actually too hard for me now you think, it would be good to just learn how to translate please.  Because I open the ROM in a Hex Editor, but I don't get any text.

Thanks

tummai

  • Full Member
  • ***
  • Posts: 204
  • Location: Japan
    • View Profile
    • Tummai Games
How do I dump the palette?  I tried using CPU-Dumping-Genesis-Graphics-Dump TLP format Palletes and I get a TLP file, I open it with Cygnus, and I get a very small Hexadecimal part (Like 15-20 lines).

That sounds about right.  At any one time, the Genesis VDP can have 4 16-color palettes loaded and ready to use.  That's 64 colors which would result in a small dump file.

I don't know what TLP palette format is, but when a game writes palettes to CRAM (Color RAM - the area in VDP memory that holds the palette information) each color gets written as a word in this format: 0000 BBB0 GGG0 RRR0.  (B=Blue, G=Green, R=Red).  So colors look like $EEE, $246, $80A.  In a hex editor they'd be split up into bytes: 0E EE 02 46 08 0A.  Does your TLP-format palette dump look like that?

If not, is there an option in your emulator to do a straight dump of the palettes?  If not, you might want to try another emulator.  Regen (debugger version) makes it really easy.  Just open the VDP debugger and click "Dump CRAM".  You can get Regen here: Regen (get the latest version with D at the end).

Quote
I tried opening the TLP file, then the French ROM, but I can't open 2 at the same time.  In this case, the ROM is opened and the TLP file  is closed.

Try this: Open the palette file and write down a row of 8 or so colors on paper.  Then load the ROM and type them manually in the hex search box.

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
Well, I get this only.  There's no more text or number under:



Quote
Try this: Open the palette file and write down a row of 8 or so colors on paper.  Then load the ROM and type them manually in the hex search box.

So I'm doing this next?

Tauwasser

  • Restricted Access
  • Hero Member
  • *
  • Posts: 1397
  • Fantabulous!!
    • View Profile
    • My blog
There are some numbers right at the top. Go fetch your glasses my son. It starts with Black followed by 88% gray and so on. Presumably those bytes are in the following order: B G R A where A is Transparency.

cYa,

Tauwasser

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
There are some numbers right at the top. Go fetch your glasses my son. It starts with Black followed by 88% gray and so on. Presumably those bytes are in the following order: B G R A where A is Transparency.

cYa,

Tauwasser

Yes, I saw those numbers.   ;D
But there's only transparency?  No colors?

I can try changing theses values?

tummai

  • Full Member
  • ***
  • Posts: 204
  • Location: Japan
    • View Profile
    • Tummai Games
There are some numbers right at the top. Go fetch your glasses my son. It starts with Black followed by 88% gray and so on. Presumably those bytes are in the following order: B G R A where A is Transparency.

cYa,

Tauwasser

Yes, I saw those numbers.   ;D
But there's only transparency?  No colors?

I can try changing theses values?

There are colors.  The first three bytes (54 50 4C) are the header.  See how they spell out "TPL"?  After that there are color definitions.  Each color is 4 bytes, in the order BGRA (if Tauwasser is right about TPL color order).

Assuming he's right, the first color is B=0, G=0, R=0, A=0, which is black.  The second color is B=E0, G=E0, R=E0, A=0 which is gray.  The third color is B=0, G=60, R=0, A=0, which is green, etc.

This palette probably won't help you search for the palette in the ROM though, since the TPL palette format is different from the Genesis's palette format.  You should try to get a CRAM dump in raw Genesis palette format using Regen(D).

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
Good, so with Regen(D), I got the CRAM, while findind what I was looking for, I need now to know how to edit them. ^^

http://img130.imageshack.us/img130/6295/printdbz.jpg

I need to edit the CRAM in a Hex Editor right now?

Sorry for all those idiots questions, I truly suck at ROM Hacking.

Thanks a lot for your patience!

tummai

  • Full Member
  • ***
  • Posts: 204
  • Location: Japan
    • View Profile
    • Tummai Games
The CRAM dump is just a snapshot of what the palettes look like inside the VDP at a given moment in the game.  To change the actual colors in game, we will need to find the palettes in the original ROM and change them there.  The CRAM dump is just a way to help us find the palettes by giving us a hint on what to search for.

Take a peek at the CRAM file in a hex editor.  Each color is represented by 2 bytes.  Each palette is 16 colors (32 bytes).  The CRAM dump will actually have each color stored backwards, so if the hex editor says this:

Code: [Select]
00 00 EE 01 00 00 AE 0A 8E 08 AA 0A 88 08 8E 00
0E 00 0A 00 00 0C 00 08 EE 02 AA 02 88 0E 66 0E

Then your actual colors will be:

Code: [Select]
0000 01EE 0000 0AAE 088E 0AAA 0888 008E
000E 000A 000C 0008 02EE 02AA 0E88 0E66

Notice for each pair of bytes, we just swapped them.

So pick the palette you want to change, swap the bytes to convert to color values and then do a hex search in the original ROM file.  If the palettes are stored in the ROM uncompressed, then you're set.  If not, then on to plan B.

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
Ugh, I reverted the CRAM, which at the beggining was like this..

Ah, anyways; I'm all confused in all my CRAM files that I copied..
All those numbers are bugging me and I'm wondering if I'll be able to...

Bah, this is way too hard.  :(
Sorry but I think I'll give up for now, except if there's an easier way...

What's the Plan B?

tummai

  • Full Member
  • ***
  • Posts: 204
  • Location: Japan
    • View Profile
    • Tummai Games
Don't give up.  You're almost there.  :)

Delete all your CRAM and palette files and then dump the CRAM again.  Then you'll just have one file which is easier to work with.  Load it up in your hex editor and post a screenshot.  I'll walk you through it from there.

Check the hair on the left Goku:


GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
Thank you.

It's the same thing with the French ROM?
Because like I said, I prefer this one.  This wont affect the data?  Will I be able to do this way with the French ROM?  It's a SMD file, the BIN is corrupted.

Allright, I'll show a step-by-step pictures of what I'm doing.

http://img715.imageshack.us/img715/3609/romscreen1.jpg
http://img20.imageshack.us/img20/3527/romscreen2.jpg
http://img696.imageshack.us/img696/6674/romscreen3.jpg
http://img714.imageshack.us/img714/4862/romscreen4.jpg
http://img28.imageshack.us/img28/6137/romscreen5.jpg
http://img193.imageshack.us/img193/6847/romscreen6.jpg

This is exactly what I want to do for now in your picture.  I was able to do this in a image editor for fun, but not in the ROM yet.


Thanks again!

tummai

  • Full Member
  • ***
  • Posts: 204
  • Location: Japan
    • View Profile
    • Tummai Games
Thank you.

It's the same thing with the French ROM?
Because like I said, I prefer this one.  This wont affect the data?  Will I be able to do this way with the French ROM?  It's a SMD file, the BIN is corrupted.

I don't have a copy of the French ROM, but I imagine it would be the same for both versions (if this doesn't work, shoot me a PM and we can discuss me getting a hold of the French ROM).

Ok.  The Genesis allows four 16-color palettes loaded at one time.  Each color is represented by 2 bytes.  2*16 = 32.  Each palette is 32 bytes.  Here is how they are laid out:



Notice I numbered the palettes 1-4, and indicated the corresponding bytes in the hex editor.

1P Goku is probably palette #2, as it has orange and yellow.  Let's try to find palette #2 in the ROM.  First we need to translate those 32 bytes into Genesis color values.  We do this by swapping the bytes in each byte-pair:

Code: [Select]
00 00 0C 00 EE 0C AE 04 6E 00 2C 00 E0 0E E0 0E
00 00 22 02 44 04 66 06 AA 00 CC 00 00 00 EE 00

becomes:

Code: [Select]
0000 000C 0CEE 04AE 006E 002C 0EE0 0EE0
0000 0222 0444 0666 00AA 00CC 0000 00EE

Now we want to find those yellows, because those are the colors used for his hair.  So we open the ROM in our hex editor and hex search for the following string: 00AA00CC000000EE

I get a match.  Do you?

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
Yes, I have one.  :)
At the third line which is the second palette.  It's logical because it's probably the one of Goku.

How did you found that 00AA00CC000000EE is yellow?

tummai

  • Full Member
  • ***
  • Posts: 204
  • Location: Japan
    • View Profile
    • Tummai Games
Yes, I have one.  :)
At the third line which is the second palette.  It's logical because it's probably the one of Goku.

How did you found that 00AA00CC000000EE is yellow?

whoops.. Sorry, I skipped a step. If you look at palette #2 in the picture I posted, the yellows are the last four colors in the palette (actually 2 yellows, a black and another yellow).  So I picked the corresponding color values:

Quote
0000 000C 0CEE 04AE 006E 002C 0EE0 0EE0
0000 0222 0444 0666 00AA 00CC 0000 00EE

Notice the black in the middle of the yellows is 0000, which is what you'd expect for black,

GameX

  • Newbie
  • *
  • Posts: 17
    • View Profile
OK, so I changed the:

Quote
0000 000C 0CEE 04AE 006E 002C 0EE0 0EE0
0000 0222 0444 0666 00AA 00CC 0000 00EE

In this:

Quote
0000 000C 0CEE 04AE 006E 002C 0EE0 0EE0
0000 0222 0444 0666 0000 0000 0000 0000

Now I need to "import" it in the ROM with a certain way?
« Last Edit: June 02, 2010, 07:04:03 pm by GameX »

KingMike

  • Forum Moderator
  • Hero Member
  • *****
  • Posts: 5942
  • *sigh* A changed avatar. Big deal.
    • View Profile
It's a SMD file, the BIN is corrupted.
Maybe the file is renamed.
SMD is the scrambled format, BIN is the format you want to hack.
(the easiest way to tell is to look at offset 100 in a hex editor. If it says SEGA MEGA DRIVE (or SEGA GENESIS), then it's BIN)
Quote
Sir Howard Stringer, chief executive of Sony, on Christmas sales of the PS3:
"It's a little fortuitous that the Wii is running out of hardware."