News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Compressed Graphics in Corpse Party: Book of Shadows for PSP  (Read 10313 times)

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Compressed Graphics in Corpse Party: Book of Shadows for PSP
« on: October 08, 2013, 03:36:19 pm »
I'm trying to extract the graphics files from Corpse Party: Book of Shadows (specifically, the character bust graphics), but it's proving to be extremely difficult, especially for someone like myself, with very little knowledge of programming and ripping data from ISOs and such.

I extracted some AFS files from the ISO, and judging from the file names, I believe I found the bust sprites. Unfortunately, they're all listed as being in "KLZ" format, which as far as I can tell, doesn't actually exist. I got some help from someone a little bit more knowledgeable (but not much more) and he looked at the KLZ files in a HEX editor.

He says they look like TIM2 files, but don't match exactly.


"The headers of the KLZ files don't exactly match those of TM2 images. That highlighted span wouldn't typically be there. Maybe 5pb modified the format for their VN engine, which Book of Shadows runs on."

He tried using a couple of programs meant to extract TIM2 files, but he only managed to get garbled pictures like this


Another person I asked had this to say

"I've been playing with the files you uploaded, and I have to say: However these are encoded, it makes sense to me that they probably encoded and compressed the bust sprites somehow. Although, I'd like to say: Looking at how its hex looks, and what I found in Image Viewer (making it look like a TIM2 file and then opening it reveals it's basically a 50-pixel-wide strip of colors and white noise) makes me think it's their variation of compression. That, or it's been straight up fucked up somehow."

The files are like this in both the US and JP ISOs, and I'm kind of at a loss for what to do at this point. Does anyone here, who might actually have some actual experience with this sort of thing, have any ideas?

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #1 on: October 10, 2013, 09:48:54 am »
Okay. "KLZ" isn't good news; it sounds like the files you're looking into might be LZ-compressed.

...well. It is good news in that 1. it tells us the kind of compression you'll need to crack and 2. LZ is dead simple. The Cliff's Notes version is that it outputs "plaintext" bytes until it hits a compression code, which is a length-width pair indicating how far back in the output buffer to look for the next bit and how many bytes to copy from that point. How those compression codes are flagged varies, but I think I might be able to hand you a breakthrough.

You see how there are seven "garbage" bytes before the recognizable "TIM2" there? I know this is my DS hacking experience talking, but it seems strange that they'd align the data on an odd number. I think those first six bytes have to do with the compression properties (the uncompressed size is probably in there somewhere), and the seventh is - drumroll! - a compression code indicator. DS's LZ implementation groups codes (both plaintext and compression) into groups of eight, prefixed by a single byte containing indicator flags. If the flag is clear (0), the corresponding code in the sequence is plaintext; if it's set (1), then it's a compression code. I know this is a PSP game we're talking about, but that doesn't mean it might not use a similar implementation.

To use that snippet as an example, the first indicator is "00", so the next eight bytes are meant to be output as plaintext (yay!). Then we hit "81" - which, in binary, is 10000001 (that's a lot of dalmatians!). We don't treat this as a literal number - it just tells the decompressor when it should try to read a compression code and when it should output plaintext. What it tells the compressor is that the next two bytes (again, assuming this works like the DS's built-in LZ implementation) should be read as a compression code, the six bytes after it are plaintext, and the two bytes after them are another compression code (seeing the pattern?). The next byte after that is "20", or 00100000, which means that (assuming this is little-endian) the next five bytes are plaintext, followed by a two-byte compression code, followed by two more plaintext bytes (otherwise, the order is reversed - two plaintext, a compression code, then five plaintext).

I don't know if the currently-available PSP emulators have enough debugging functionality to be of much help, but if you could actually find the uncompressed file in memory - or even just the first part of it - it'd be tremendously useful. Without the uncompressed file, figuring out the specifics of the game's LZ implementation is a little like fumbling around in the dark (in a haunted school full of murderous ghosts :3).
In the event of a firestorm, the salad bar will remain open.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #2 on: October 10, 2013, 01:47:16 pm »
Okay, a lot the technical stuff in your post flew right over my head, but if I'm understanding right, PPSSPP's got the debug options we need. I played the game until it loaded up a character graphic, and then opened the memory viewer. I'm not really sure what I'm looking for here, so I took screenshots. The first three don't look like they contain any data, but I'm including them for posterity.

Scratchpad
VRAM
User Memory
Defaul Load Address
Kernel Memory

Default Load Address keeps going like that for awhile, but it's all just random numbers and letters to me, so I just took a screenshot of the top. I don't know if any of that stuff has anything to do with graphic data.

KC

  • Full Member
  • ***
  • Posts: 209
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #3 on: October 10, 2013, 01:58:35 pm »
You will want to use the GE debugger of one of the recent WIP builds (download: http://buildbot.orphis.net/ppsspp/). Step draw until the texture you search is shown on the left, then check its address. For example:


You can then dump that from the memory dialog. Right click in the hex view and select the dump option.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #4 on: October 10, 2013, 03:23:11 pm »
Okay, I downloaded a WIP build and used the GE debugger to look up a texture's address, but when I look that address up in the memory viewer I get this



And that doesn't look right at all. I went ahead and dumped it, regardless, and stuck the file on my dropbox, in case it actually does produce something that could be used by someone who has some idea of what they're doing.

I'm really sorry for being so clueless about this stuff.

KC

  • Full Member
  • ***
  • Posts: 209
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #5 on: October 10, 2013, 05:04:11 pm »
That's not a valid memory address there. Make sure to note the right address that it displays.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #6 on: October 10, 2013, 05:35:42 pm »
Oh whoops, I accidentally typed a 0 instead of a 9.


(click for full size image)

That's not really what I expected to see either, though. I went ahead and included the GE Debugger in the screenshot to be sure I'm looking at the right number.

KC

  • Full Member
  • ***
  • Posts: 209
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #7 on: October 10, 2013, 05:44:44 pm »
What you see there is a raw texture without any headers. As you can see in the preview, there is a lot of blank space. That needs to be encoded too, and that's where the FFs come from.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #8 on: October 10, 2013, 06:28:49 pm »
Oh, I see. That makes a whole lot of sense.

I think I dumped this correctly. I put 0x09861900 as the start point and just kind of let the emulator handle the rest, and got this file from it. I don't really have any idea where to go from here, though.

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #9 on: October 10, 2013, 06:35:44 pm »
Well. It'd help if you could find the absolute start of the uncompressed data - the "TIM2" and onward you can see in the compressed file. It won't match exactly - again, it's compressed - but if you can find the uncompressed data and its compressed counterpart, it should be possible (trivial, if I've got this figured properly) to write up an extractor.

Matter of fact, if you know where to find some specifics on the TIM2 format, it'd be possible to write up a program that could rip 'em directly to PNG, GIF, or whatever other format you choose.
In the event of a firestorm, the salad bar will remain open.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #10 on: October 10, 2013, 07:34:36 pm »
I think I've found some matching data, comparing it to the compressed version of what I believe is the same image.

The compressed image is on the left, the data from the emulator is on the right. I highlighted the values that make me think I got a match.

(click for full size)

I had to scroll down a few lines in the compressed data to reach this point, but this is the first actual data (ie, not FF FF FF) I found in the uncompressed version. I was using the file AYU_AA01.KLZ to compare. The same file is also present in the first Corpse Party (which had simple graphics to rip), so I'm reasonably confident it's the exact same texture as the one I was looking at on the emulator, as long as they didn't change their file names around between games. And here's the dump, if you need it.

Matter of fact, if you know where to find some specifics on the TIM2 format, it'd be possible to write up a program that could rip 'em directly to PNG, GIF, or whatever other format you choose.

The person who initially identified these files as TIM2 pointed me here and here (The second one is Google translated from Russian, it might not be 100% accurate) to get more info on the format.

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #11 on: October 10, 2013, 08:06:57 pm »
Okay. Is there anything before the dump? If there is a "TIM2" in memory, then that's where the dump needs to start. I need to be able to compare the compressed data with the uncompressed data, and that means I need to be able to start at the absolute beginning.
In the event of a firestorm, the salad bar will remain open.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #12 on: October 10, 2013, 08:35:39 pm »
Found it way up near the top



And here's a dump that starts from that line, 088F12E0

edit: I put that dump into a HEX editor that actually has a search function, and TIM2 appears in two other places as well.



« Last Edit: October 10, 2013, 08:59:25 pm by DarkHamsterlord »

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #13 on: October 10, 2013, 09:26:31 pm »
Gentlemen. I believe we've found our Rosetta Stone.

That's the compressed data and its corresponding uncompressed data right there in those last two screenshots. Good going! :3

Looks like I was right, for the most part...but I'm still having trouble working out the control codes. "00 05" seems pretty straightforward (the first byte is the distance back; the second byte is the number of bytes to copy, minus 3; in other words, this translates to a string of 8 "00" bytes, as you can see in the output), but I'm not sure how "01 60" translates into what I'm seeing.

Try finding another matched TIM2 pair with another image. Johnny 5 needs input...
In the event of a firestorm, the salad bar will remain open.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #14 on: October 10, 2013, 09:48:54 pm »
Is this what you're asking for?





These should match up with SEI_AB02.KLZ. And here's the memory dump those screenshots came from.

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #15 on: October 10, 2013, 09:59:01 pm »
Hrrrrrrm. The data's the exact same; not much help. Try looking until you find one that matches up with that very first file you showed a snippet of.
In the event of a firestorm, the salad bar will remain open.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #16 on: October 10, 2013, 10:28:53 pm »
It doesn't match the first shot I posted, since that was sent to me by someone else and I'm not sure what file he was looking at, but I managed to get one with different data. It looks like all the character bust graphics have the same data at the top (which seems like something that makes sense) so I went into the CG viewer and grabbed data from that.





Here's the full dump, and the compressed file: EV0_0A1.KLZ

KC

  • Full Member
  • ***
  • Posts: 209
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #17 on: October 11, 2013, 04:16:17 am »
edit: I put that dump into a HEX editor that actually has a search function, and TIM2 appears in two other places as well

You can also search in the memory viewer by pressing Ctrl+S. If the ascii columns are selected, you can search for ascii data, if the hex columns are selected, you can search for a sequence of bytes (ie FF FF FF FF 10 etc). Press Ctrl+C to continue after a match.

Easiest thing to do would probably to find the actual decompression function. You can use breakpoints in the disassembly for that.

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #18 on: October 11, 2013, 05:16:43 am »
Try this.

https://dl.dropboxusercontent.com/u/74940256/cplz.zip

I can't do much more until I can find a TIM2 format specification, but this will extract the *.klz files into *.tim2 files. Technically, it's a command line-based Python script, but I've made it as newbie-friendly as possible - I've got it compiled to a Windows EXE, so you shouldn't have to install anything extra (just make sure that python27.dll and library.zip are in the same directory), and all you have to do is drag-and-drop the files you want to extract on top of it.

Yes, in case it's not obvious yet, I figured out the LZ compression scheme from the files you gave me. Neat, huh? :3
« Last Edit: October 11, 2013, 05:23:56 am by Ryusui »
In the event of a firestorm, the salad bar will remain open.

DarkHamsterlord

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: Compressed Graphics in Corpse Party: Book of Shadows for PSP
« Reply #19 on: October 11, 2013, 01:21:10 pm »
Thank you so much for the help. But unfortunately, the program you wrote only seems to work on the thumbnail files (I may have accidentally sent you HEX data for a thumbnail at one point, I'm not sure.)



The thumbnail image comes out perfect, but when I tried it with a larger one...



Here's the converted file that didn't come out right, in case you need to look at it in a HEX viewer. If there's any more information you need me to grab from the game, let me know and I'll get it.

You can also search in the memory viewer by pressing Ctrl+S. If the ascii columns are selected, you can search for ascii data, if the hex columns are selected, you can search for a sequence of bytes (ie FF FF FF FF 10 etc). Press Ctrl+C to continue after a match.

You know, I tried CTRL+F and got a pop up box that said "Not implemented" and just assumed they hadn't implemented a search yet.