Any idea what this SLZ algorithm is?

I'm trying to hack a Star Ocean game (Star Ocean First Departure, PSP) but everything is in these SLZ files. All Google can come up with is this article:
http://www.gamedev.net/reference/articles/article294.asp

Basically, according to the algorithm, to decompress you read one byte (the tag) which tells you whether or not the next 8 "chunks" are compressed (a set bit means compressed). For example, if a tag FE was read, it'd be 1111 1110, meaning that the next 7 chunks are compressed and then the 8th is not (so just read it that one byte as it is). For compressed chunks, two bytes are read which determine the offset (in the history) and length as follows:

O11 O10 O9 O8 L3 L2 L1 L0     O7 O6 O5 O4 O3 O2 O1 O0

which allows a maximum history size of 4096 bytes.

Another version, which allows a history of 2048 bytes, is specified as follows:
O10 O9 O8 L5 L3 L2 L1 L0     O7 O6 O5 O4 O3 O2 O1 O0


Now, in these SLZ files I found, it at first seemed that it followed the algorithm in that article, but instead a set bit would mean uncompressed. I found several files that verified this, and there was one that had FF followed by ABCDEFGH and then another FF followed by IJKLMNOP and so on.

Here's one small sample SLZ file I found that precedes an AT3 sound file in the file pack (one huge binary file). I played the sound files and they are all BGM files, and all of them have the exact same SLZ file preceding them except that the At05 you see below is At06, At07, etc.

Code: [Select]

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

15D5B810  53 4C 5A 02 44 00 00 00 F4 00 00 00 00 00 00 00  SLZ.D...ô.......
15D5B820  FF FF FF F4 00 41 74 30 35 F7 16 00 64 20 F1 80  ÿÿÿô.At05÷..d ñ€
15D5B830  04 04 00 FF 05 00 00 00 10 00 00 D6 07 00 01 FF  ...ÿ.......Ö...ÿ
15D5B840  00 F1 06 E0 06 E0 06 E0 06 E0 80 06 E0 06 E0 06  .ñ.à.à.à.à€.à.à.
15D5B850  E0 06 E0 06 E0 06 E0 0C 70 F0 79 00 FF FE 0A 40  à.à.à.à.pðy.ÿþ.@
15D5B860  A0 80 00                                          €.

The first 16 bytes consist of the header. The header consists of SLZ followed by some byte which I have not found the meaninh to yet, an int for the data size (in little-endian), another int for the uncompressed size, and another 4 bytes which always seem to be zero (I haven't paid much attention to the header yet).

Anyways, on to the data and the problem: First, there is an FF, saying that the next 8 bytes are not compressed, and then an F7 (11110111) which says the first 4 bytes are uncompressed, and then there is a offset-length pair, and then 3 more uncompressed bytes (so 9 in total). After 9 bytes, we get to another FF, so it seems like everything is following the algorithm.

However, 8 bytes later, there is 07 (00000111), saying the next 5 chunks are compressed. This is where everything starts to not follow the algorithm. According to the algorithm, if you read first null byte (for compressed data), then you quit the routine (what exactly is meant by "the routine" is not explained). Going with this, everything seems okay (assuming the offset-length pairs are valid) until I hit "E0 06" - a length of 0 (okay it's little-endian but back at the first compressed chunk, there'd be 80 F1)?? And then these pairs with E0's repeat until the end of the file, way past what the last tag specified.

So now I am confused. The file seems to follow the algorithm with the FF tags and the F7 tag (the 9 bytes before another FF), but then after that, everything is messed up and now I doubt that this file follows the SLZ in that article.

I didn't think so at first. No way some Japanese developers would use some article on gamdev right? I looked everywhere but couldn't find anything else. These SLZ files were used for the PSX Star Ocean games too (or so I heard), which were made around the time the article was first published. I also found the algorithm being mentioned in some text about open source Amiga code. From a Google search for the author's name, I learned that he now is a successful game developer who has worked on all sorts of games. Now it seems that it is very well possible that the developers used the article, but from what I've looked at, it doesn't

Maybe I am understanding the algorithm wrong? That C code he provides is so old I gave up on trying to even compile it. It's even harder to understand (I am not into low level programming at all), so all I have is the description (which just describes decompressing for some reason).

I have been going at this for weeks (whenever I felt like it) and I'm about to give up. Has anyone ever heard of any other "SLZ" algorithm? Even Google Japan returns nothing...

I would say that it looks like the first bit can never be compressed. When it's 0 it probably branches into some other compression like huffman or RLE. Although the data gets pretty repetitive.

[FF] FF FF F4 00 41 74 30 35    (8 uncompressed)
[F7] 16 00 64 20 (F1 80) 04 04 00  (4 uncompressed, 1 compressed, 3 uncompressed)
[FF] 05 00 00 00 10 00 00 D6  (8 uncompressed)
[07] 00 01 FF 00 F1 06 E0 06 E0 06 E0 06 E0 80 06 E0 06 E0 06 E0 06 E0 06 E0 06 E0 0C 70 F0 79 00 FF FE 0A 40


http://www.jalix.org/ressources/miscellaneous/~vrac/faqsys/docs/slz_art.txt


There is a routine there. Search for UnPackSLZ.

Looks like the "SLZ" author didn't know about LZSS.

I would say that it looks like the first bit can never be compressed. When it's 0 it probably branches into some other compression like huffman or RLE. Although the data gets pretty repetitive.

[FF] FF FF F4 00 41 74 30 35    (8 uncompressed)
[F7] 16 00 64 20 (F1 80) 04 04 00  (4 uncompressed, 1 compressed, 3 uncompressed)
[FF] 05 00 00 00 10 00 00 D6  (8 uncompressed)
[07] 00 01 FF 00 F1 06 E0 06 E0 06 E0 06 E0 80 06 E0 06 E0 06 E0 06 E0 06 E0 06 E0 0C 70 F0 79 00 FF FE 0A 40


http://www.jalix.org/ressources/miscellaneous/~vrac/faqsys/docs/slz_art.txt


There is a routine there. Search for UnPackSLZ.

Yeah, that's the same article. The file doesn't follow that though.

I never thought of it using some other compression when it's 0, thanks a lot. I'll read up on these other algorithms.

Looks like the "SLZ" author didn't know about LZSS.

Haha I guess he didn't. Looks like it's the exact same.

There's a good chance it's the same compression that was used in triAce's PSX games. SLZ type 0 is uncompressed, type 1 is normal lzss and 2 is mixed lzss/rle. Star Ocean 3 also used a type 3.

Here's an old program I made for decompressings types 0-2: http://rapidshare.com/files/180667057/slz.rar.html (slz data.bin 0x0 output.bin)

The fourth word in the header is rarely used. Valkyrie Profile used it to specify the position of another compressed file.

There's a good chance it's the same compression that was used in triAce's PSX games. SLZ type 0 is uncompressed, type 1 is normal lzss and 2 is mixed lzss/rle. Star Ocean 3 also used a type 3.

Here's an old program I made for decompressings types 0-2: http://rapidshare.com/files/180667057/slz.rar.html (slz data.bin 0x0 output.bin)

The fourth word in the header is rarely used. Valkyrie Profile used it to specify the position of another compressed file.

Thank you very much! Your program works great (from what I have tested), and now I know what that byte after 'SLZ' is.

Would it be possible to get the source code for that though (by PM)? I'd like to know exactly how the type 2 compression works. If you're uncomfortable with giving it out though, I understand.

Thanks again for the info and your program That saved me a lot of work.

There's a good chance it's the same compression that was used in triAce's PSX games. SLZ type 0 is uncompressed, type 1 is normal lzss and 2 is mixed lzss/rle. Star Ocean 3 also used a type 3.

Here's an old program I made for decompressings types 0-2: http://rapidshare.com/files/180667057/slz.rar.html (slz data.bin 0x0 output.bin)

The fourth word in the header is rarely used. Valkyrie Profile used it to specify the position of another compressed file.

Thank you very much! Your program works great (from what I have tested), and now I know what that byte after 'SLZ' is.

Would it be possible to get the source code for that though (by PM)? I'd like to know exactly how the type 2 compression works. If you're uncomfortable with giving it out though, I understand.

Thanks again for the info and your program That saved me a lot of work.

Awesome. Should we add this to the database KC?

Hm... I'd prefer to add recompression first. I already wrote code for that years ago, but I have to refine it and put everything together. No idea when that will happen, though...

we can get by without the code, but some basic algorithm for decompression of method 2 would be nice.

i think i can implement lzss on my own just basing on your tip. well, i'm still learning about data mangling in c++, so i take it as a learning experience.

KC could you please upload your program for the decompression again the link does not work.
I would love to try this on star ocean 4 and see what comes up I have looked into these archives and they appear to be the same compression.

Edited by KingMike:
The old image link seems to have expired. Because the hoster replaced with a huge spam image, I removed it. Just incase the poster wishes to reupload or something, here's the original link:
http://xs537.xs.to/xs537/09133/socean380.png.xs.jpg

ok, i found the source code for slz unpacking type 0 1 and 2.

there is this tool called quickbms, where you can write a script describing the data structure and the tool will take care of it. it has built in tri ace unpacker and various other too.

http://aluigi.altervista.org/papers.htm#quickbms

you can grab the source along with the binary here : http://aluigi.altervista.org/papers/quickbms.zip. file you want to check out is unz.h (look for "triace"). i'm trying to figure it out now, but it's pretty confusing. of course i've tested it on some slz packed files and it works properly.

edit : i've rewritten it into a separate app to get a better understanding of it, but it seems to fail on 0xd1d byte of my sample file, while the original code works. argh

if i find the bug and clean up the code i'll probably post it here.

edit: found the bug now it seems to work properly. i'll try posting it in a minute
February 12, 2010, 05:11:00 AM - (Auto Merged - Double Posts are not allowed before 7 days.)ok here goes.

http://pastebin.ca/1794442

before you bash me for my , uhh, "coding style" - i know it's far from stellar

also, some slz types might not specify unpack size in header (i've only tested type 2 so far from valkyrie profile 2).

also i had an code indenter running wild, and it looks a bit messy.

i've added a big amount of comments in there to get a better understanding of what's going on - i haven't done hex calculations in a while, so that definitely helped me :]

There's a good chance it's the same compression that was used in triAce's PSX games. SLZ type 0 is uncompressed, type 1 is normal lzss and 2 is mixed lzss/rle. Star Ocean 3 also used a type 3.

Here's an old program I made for decompressings types 0-2: http://rapidshare.com/files/180667057/slz.rar.html (slz data.bin 0x0 output.bin)

The fourth word in the header is rarely used. Valkyrie Profile used it to specify the position of another compressed file.

Any chance you could reupload your program KC (or anyone else) ?
Has it been added to the database (I haven't found it but just asking anyway)

I'm actually trying to extract movie files from Valkyrie Profile 2, I've already unpacked files from the archive, but all I get is .unk and .pk1 files, which are obviously (SLZ) encrypted ...

SLZ decompressor with source code added to the database, supporting all modes (0-1-2-3):
http://www.bwass.org/bucket/SLZ_decompressor-CUE.rar

You can use "triAce-PS2" to extract all files from the ISO image:
http://www.sendspace.com/file/d8oank

To extract all files to a folder (command line only):
triAce-PS2 ISOname folder

Thanks a lot for replying "from nowhere" (I see this is your first post) !

I did try your SLZ decompressor on VP2 unpacked files and only a few .slz files are properly decoded, actually only 0002.slz and 0006.slz (other slz make your program crash). These are decoded into obscure .raw files... But what I am supposed to do with those ? I checked them with an hex-editor and they seem to be ELF files... I may be dumb but I'm completely lost there...

I only want to find the cutscenes which -I guess- are located in .pk1 or .bin files. How could I process/decrypt those with your proggie ?
I'll search further anyway and once again thanks a lot for replying and contributing.

I did try your SLZ decompressor on VP2 unpacked files and only a few .slz files are properly decoded, actually only 0002.slz and 0006.slz (other slz make your program crash). These are decoded into obscure .raw files... But what I am supposed to do with those ? I checked them with an hex-editor and they seem to be ELF files... I may be dumb but I'm completely lost there...

- If those are the only ones decoded, maybe the others are archive, which screw up the uncompression program (just guessing)
- Elf files are PS executable, think of it as .exe of your computer, and the one who's launching the whole game in PS2 games are the SLUS/SLPS (etc) file in the main folder of the game.

I only want to find the cutscenes which -I guess- are located in .pk1 or .bin files. How could I process/decrypt those with your proggie ?
I'll search further anyway and once again thanks a lot for replying and contributing.

.pk1   .... maybe PacKage 1, meaning another archive, it's not just about uncompression methods.