Mega Man X4 (PSX) Decompression Routine

[DarkSamus993]

I'm spearheading the complete re-rip of http://www.sprites-inc.co.uk/ graphic archives and replacing everything with data rips.
Of course this means dealing with compression at times, and the game I'm currently working with is Mega Man X4 (X5-6 look to use the same compression method just by looking in Tile Molester). The player sprites and weapons seem to be the only things that are compressed fortunately. I know I can dump them with a VRAM viewer, but I'd like to increase my knowlegde and skills by figuring out the decompression routine; I just need a bit of guidance on what I'm doing.

What I've got going so far:
I found a tool called BioFAT that was able to split the game archives into parts. Looking at the palettes stored in the files, I was able to determine what the compressed graphics were, despite not being able to view them.

The compressed MMX4 files:
PL00_U.ARC > PL00_U02.BIN = X (Force Armor)
PL01_U.ARC > PL01_U02.BIN = Zero
PL02_U.ARC > PL02_U02.BIN = X (Ultimate Armor)
The compressed MMX5 files:
ROCK_X5.DAT  >  ROCK_X588.BIN > ROCK_X58803.BIN = X (Force Armor)
ROCK_X5.DAT  >  ROCK_X590.BIN > ROCK_X59003.BIN = Zero
ROCK_X5.DAT  >  ROCK_X592.BIN > ROCK_X59203.BIN = X (Force Armor) - DUPLICATE
ROCK_X5.DAT  >  ROCK_X593.BIN > ROCK_X59303.BIN = X (Ultimate Armor)
ROCK_X5.DAT  >  ROCK_X594.BIN > ROCK_X59403.BIN = X (Falcon Armor)
ROCK_X5.DAT  >  ROCK_X595.BIN > ROCK_X59503.BIN = X (Gaea Armor)
The compressed MMX6 files:
ROCK_X6.DAT > ROCK_X685.BIN > ROCK_X68503.BIN = Zero
ROCK_X6.DAT > ROCK_X686.BIN > ROCK_X68603.BIN = X (Ultimate Armor)
ROCK_X6.DAT > ROCK_X687.BIN > ROCK_X68703.BIN = X (Falcon Armor)
ROCK_X6.DAT > ROCK_X688.BIN > ROCK_X68803.BIN = X (Shadow Armor)
ROCK_X6.DAT > ROCK_X689.BIN > ROCK_X68903.BIN = X (Blade Armor)

Here are the files, perhaps someone recognizes the compression method and/or a tool exists already that would work on them.
http://www.mediafire.com/download/cm0wn55j180i411/MMX4-6+Compressed+Graphics.zip

At any rate, I'll probably be stuck using a debugger and figuring out the decompression routine myself.  I've been messing around with PCSX Agemo Debugger and pSX's built in debugger the past couple of days and want to make sure I did everything correctly. I figure I better make sure I even found the correct routine before I start trying to figure out what it all does.

PCSX Agemo Debugger:
1) Use GPU upload breakpoint to find where X's graphics are loaded into VRAM from and then dump the RAM.

GPU Upload from $8016DEA8 , $800 bytes
area X/Y(320, 0) W/H(64,16)

2) Use tracing and assembly logs to find the code being executed.

Tracing: Use the emulator's Save(F1),load(F3) + debugger's "total inst."
   a) Emulator F1 then F3 (set "total inst." to 0).
   b) Break on Memory Write ($8016DEA8 in this case).
   c) Resume.
   d) When break point occured, remember current "total inst." value.
   e) Emulator F3 (set "total inst." to 0 again).
   f) Use "exec" to execute to before.
   g) Then you can enable "asm log" to get all the instructions.

3) Use AdisasM to disassemble the RAM that was previously dumped and search for the final snippet of code that was executed in the assembly log. This snippet should be located in the Decompression Routine, so just copy the entire function.

Code Select Expand

func16ff4:  ; 80016FF4
80016FF4    lhu    t1, $0000(a0)
80016FF8    addiu  a0, a0, $0002
80016FFC    ori    t0, zero, $8000
80017000    addiu  a3, zero, $0010

loop17004:  ; 80017004
80017004    and    v0, t0, t1
80017008    bne    v0, zero, L17024 [$80017024]
8001700C    nop
80017010    lhu    v0, $0000(a0)
80017014    addiu  a0, a0, $0002
80017018    sh     v0, $0000(a1)
8001701C    j      L17094 [$80017094]
80017020    addiu  a1, a1, $0002

L17024:     ; 80017024
80017024    lhu    a2, $0000(a0)
80017028    nop
8001702C    andi   v0, a2, $f800
80017030    beq    v0, zero, L17044 [$80017044]
80017034    addiu  a0, a0, $0002
80017038    srl    v1, v0, $0b
8001703C    j      L1704c [$8001704c]
80017040    andi   a2, a2, $07ff

L17044:     ; 80017044
80017044    lhu    v1, $0000(a0)
80017048    addiu  a0, a0, $0002

L1704c:     ; 8001704C
8001704C    or     v0, v1, a2
80017050    beq    v0, zero, L170a8 [$800170a8]
80017054    nop
80017058    bne    a2, zero, L17078 [$80017078]
8001705C    sll    v0, a2, $01

loop17060:  ; 80017060
80017060    sh     zero, $0000(a1)
80017064    addiu  v1, v1, $ffff (=-$1)
80017068    bne    v1, zero, loop17060 [$80017060]
8001706C    addiu  a1, a1, $0002
80017070    j      L17098 [$80017098]
80017074    addiu  a3, a3, $ffff (=-$1)

L17078:     ; 80017078
80017078    subu   a2, a1, v0

loop1707c:  ; 8001707C
8001707C    lhu    v0, $0000(a2)
80017080    addiu  a2, a2, $0002
80017084    addiu  v1, v1, $ffff (=-$1)
80017088    sh     v0, $0000(a1)
8001708C    bne    v1, zero, loop1707c [$8001707c]
80017090    addiu  a1, a1, $0002

L17094:     ; 80017094
80017094    addiu  a3, a3, $ffff (=-$1)

L17098:     ; 80017098
80017098    bne    a3, zero, loop17004 [$80017004]
8001709C    srl    t0, t0, $01
800170A0    j      func16ff4 [$80016ff4]
800170A4    nop

L170a8:     ; 800170A8
800170A8    jr     ra
800170AC    nop

pSX:
5) To help confirm this is correct, use pSX's built in debugger and set a memory write breakpoint on $8016DEA8. Open the Memory and Dissasembly windows, and use the Step Into (F7) command. Disassembly shows the game going through the code we pulled and Memory shows data being written into RAM.

Here's the resulting logs, assembly, and RAM dump that were produced while doing all this:
http://www.mediafire.com/download/q52a42iy2dcfuo8/MMX4+-+RAM+Assembly+Logs.zip

Have I done everything correctly?

[Gemini]

That looks indeed like a decompression routine, specifically a 16 bit LZSS with a custom case where it fills the dec buffer with zeroes.

[VicVergil]

It's a bit off-topic but could you share any notes you have regarding the pointer system for this game?

There's these packed (apparently not compressed though) archives PL00_U.ARC PL01_U.ARC and PL02_U.ARC and they hold all graphics but also text and sound. The US version apparently still has the JP audio files in the archive, and inserting the JP variant of said files in an US iso restores all cut voice acting, which means it might be as simple as repointing stuff in the US archive file.

[DarkSamus993]

@Gemini
Cool, now I can move onto figuring out what it does exactly.

@GHANMI
I haven't really done anything with pointer tables as I am more interested in just dumping all the graphics/palettes (eventually audio stuff too probably, but the main focus is the graphics/palettes right now). If the audio files are still present, then I would imagine just repointing things would work.

[DarkSamus993]

Progress update:
Still debugging the routine, because it's still not functioning 100% yet, but I'm getting closer!



EDIT:
I'm stupid and forgot to set mode to 2-dimensional in Tile Molester. It works!



Ignore the Ultimate Armor palette, that's just what I had on hand at the moment.
Now I just need to tell my program to keep decompressing till it reaches the end of the file.

[justin3009]

Fantastic job!  Glad to see this game getting some love

[RaidouJFlo]

Progress update:
Still debugging the routine, because it's still not functioning 100% yet, but I'm getting closer!



EDIT:
I'm stupid and forgot to set mode to 2-dimensional in Tile Molester. It works!



Ignore the Ultimate Armor palette, that's just what I had on hand at the moment.
Now I just need to tell my program to keep decompressing till it reaches the end of the file.

That is insanely cool.

[DarkSamus993]

More progress:
I'll probably be able to release a build pretty soon, so GET READY!
I can also confirm it works for MMX5-6 as well (those games were built off the MMX4 engine after all).

And for a special treat, have a look at two unused frames for the Gaea Armor (MMX5).



The Gaea Armor is unable to use weapons, but lo and behold, some graphics exist (which in this case is Rising Fire from X4, I don't think these frames even got re-used in MMX5) as well as a complete set of palettes for the armor. Also, yes that is the Force Armor left in the data when the Gaea Armor didn't need those frames (lazy Capcom).

And just so Zero doesn't feel left out, and because I want to showcase this thing works for MMX6.

[DarkSamus993]

I apologize for not getting this out sooner, but I needed the time to put together documentation on everything, and being ill took out a large chunk of time that would have otherwise been spent completing this.

Keep in mind this is still a beta build, but it will still get the job done. There are 2 known looping errors that occur if you try and decompress from incorrect offsets, which I'll hopefully be able to patch in the next version. The included offsets in the documentation should keep you safe from the looping bugs.

Source + Binary: http://www.mediafire.com/download/63l65t7y7pph5kj/NovaStrike_v010.zip
Here's some test files if you need them: http://www.mediafire.com/download/2fmajgnb8cid9bc/Test+Files.zip

If anyone has suggestions for improved code, bugs to report, etc. please let me know. This is a command line utility, so no one better post the infamous "it closes immediately on opening" error. Meanwhile, I'll be working on fixing those pesky looping errors.

[Gemini]

If I didn't mess with any of the code flow, this is how the actual decompression would have looked like in CAPCOM's sources: http://pastebin.com/evfVwNB0
That assumes the cpu is little endian, just like the original. If you're really paranoid about endianness, you can change the input-output buffers to whatever abstracted object.

EDIT: optimized code for great justice.

[UltimateXeallar]

I tried running it to see what it did and it gave me this:

Is this normal?

[DackR]

I tried running it to see what it did and it gave me this:

Is this normal?

I'll just leave this here...
https://www.microsoft.com/en-us/download/details.aspx?id=48145