Trying to make a table for Suikoden I and II (PSP)

[DarknessSavior]

As far as Java goes, the only stuff I know how to do is really simple stuff. Concatenation, simple arithmetic, and System.out.println stuff. We just started. Yes, it's for college, but I don't have to take it. I chose to take it, hoping that since it's "Intro to Programming" that it would help me out in the realm of ASM (that way I could at least start thinking like a programmer).

So I don't see myself writing some sort of extraction program in Java just yet. Or even by the end of the semester (I'm not 100% of the topics we're covering, but we're only doing the first six chapters out of this giant Java textbook, "Absolute Java").

~DS

[BRPXQZME]

Gonna go out on a limb and recommend you just learn Python. I ain’t ever used it myself, but my experience from watching high-schoolers pick it up is that it makes stuff like this a cinch. Well-documented. zlib is part of the standard package, so you don’t even need to do anything funky to get it set up.

[Gemini]

I wonder if java has a good native support for GZip. That would make the un/packer a lot faster to code, especially if it can work with buffers (:9).

[BRPXQZME]

Does it have native support? Yes. Is it good? ...depends on what you need. Some have felt the need to re-implement the functionality, for example.

[golden]

Gonna go out on a limb and recommend you just learn Python. I ain’t ever used it myself, but my experience from watching high-schoolers pick it up is that it makes stuff like this a cinch. Well-documented. zlib is part of the standard package, so you don’t even need to do anything funky to get it set up.

I recommend it as well, I'm using it for my projects' tools. If you want to, DarknessSavior, you can check out this post which describes how something similar could be accomplished in Python.

[Gemini]

Does it have native support? Yes.

I knew about this one. IIRC, they've used pure C code for that with DLLs or whatever other systems use.

Is it good? ...depends on what you need. Some have felt the need to re-implement the functionality, for example.

:/ I'm not completely sure what's wrong with the default implementation, but as long as it decompresses from buffers it should be fine, especially for this kind of rearchivation programs.

[Kitsune Sniper]

:/ I'm not completely sure what's wrong with the default implementation, but as long as it decompresses from buffers it should be fine, especially for this kind of rearchivation programs.

It says so on the page.

"For example, we needed to full access to zlib to add the packet compression support to pure Java SSH system, but they are useless for our requirements." There's a bug in Java's zlib implementation that won't let them do this, so they made their own version. :p

[Gemini]

I read that part. Still, it doesn't explain if it's important or not for code using files rather than online streams.

[Rudy]

are you still working on it?

[Vegetaman]

I'd suggest C#, as it makes this kind of manipulation ridiculously easy (at least, over C++). However, be very careful, as some commands (like filestream) can wind up being notoriously slow...

[FaustWolf]

DS, I'm beginning to think your hay stack might not be so large after all. If you go through the process of converting your PSP file into an .ISO file with the appropriate utility (maybe not for discussion here), you should find that the system of GZIP-compressed files inside gsd1.bin and gsd2.bin share a lot of names with the files in the PSX versions of these games. At least, this is what I've found with gsd2.bin and the PSX version of Suikoden II.

Unfortunately it's not guaranteed because there are more than a few files in the PSX version that are either missing or named differently in the PSP version, but it's worth a try perhaps. Can you recall the file name of any of the text files in the Japanese language version of Suikoden II-PSX? We can check real quick if fortune smiles upon your project here.

Disregard this if you've tried file name comparison already. 


EDIT: Snooped through a couple of these randomly; results mixed. I didn't find any similarity between VA01.bin (PSX) and VA01.bin (PSP), though I didn't check thoroughly. The files SHUGO.bin (PSX) and HP_SHUGO.bin (PSP) were much more interesting, sharing byte runs of significant length (greater than 0x30 bytes matched in one lucky data search). The PSP files tend to be much larger than their PSX equivalents based on what I've observed so far, so I'm wondering if I've been looking at graphics files that are uncompressed (after GZIP decompression) in the PSP files but still LZSS compressed in the corresponding PSX file. Are the .VA## files music related in the PSX version, by chance? If the soundtrack were remastered, that might explain why the files are radically different, and larger, in the PSP version. Complete speculation on my part though.

In any case, searching for the name of one of the text files you worked with in the Japanese version of Suikoden II in gsd2.bin might be worthwhile if you haven't done this already. Just using WinRAR to apply the GZIP decompression as Rolen47 suggested earlier works like a charm, I think. I remember seeing a recognizable ASCII string (I think it referred to another filename) in the uncompressed VA01.bin (PSP); it struck me as unlikely to appear if the decompression were inaccurate.

[PhOeNiX]

In Java should be relatively simple:
InputStream comprIS=new GZIPInputStream(new FileInputStream("your file name here"));

or if you want to work on buffers

InputStream comprIS=new GZIPInputStream(new ByteArrayInputStream(byte_buffer));

but i'd sudgest using BufferedStream, instead of loading the whole file in memory

InputStream comprIS=new GZIPInputStream(new BufferedInputStream(new FileInputStream("your file name here")));

//your code here

comprIS.close();

[FaustWolf]

With regard to locating the game's text: judging from the English version of Suikoden II (PSX), VA01.bin contains the text for Riou's and Jowy's escape at the beginning of the game. Is that the case for the Japanese version of Suikoden II (PSX) as well? If so, I'd recommend GZIP-uncompressing the file tagged "VA01.bin" in the PSP version of Suikoden II (gsd2.bin) and seeing if your table works there. I didn't see anything immediately recognizable as one of the common Japanese text encoding schemes, but the English version uses a relative alphabet and I think you found that the Japanese PSX version of Suikoden II does as well; probably the case here. It'd be awfully cool if the exact same text encoding scheme were carried over from the PSX version to the PSP version.

I can't guarantee that the text is really in the VA## files, since I don't have access to a Japanese version of Suikoden II to text-compare with, nor a PSP for trial-and-error testing of the actual PSP files. Also, the VA## files in the PSP version and in the PSX version of Suikoden II seem very differently structured. There's many areas in the PSP version of the file that look like they could contain text, so I think the possibility is worth exploring, but even in best case scenario the project might involve more than simply swapping the English-language files from Suikoden II-PSX into Suikoden II-PSP. I suppose we could luck out and find that the PSP version files still have the same text storage structure as the PSX version files, and the differences come from any non-text info that might be stored in the PSP version files.

[DarknessSavior]

I don't quite understand the whole decompression thing.

Also, I only figured out the table for Suikoden I. Table file could totally be different for II.

~DS

[FaustWolf]

Finally got my paws on a Japanese copy of Suikoden II, and there are some shared data in what I would guess are the right places. This is looking a bit more promising now! Take a look around address 0x84F8 of VA01.bin in your Japanese copy of Suikoden II-PSX -- does that appear to hold text according to the table file you have available? If you don't have everything handy at the moment, I'll see if I can figure out the Japanese text encoding scheme Suikoden II was using. If what I'm looking at right now is indeed dialogue, this project looks like it might be doable.


This is what worked for me as far as the PSP file decompression:

1. Convert your .CSO into a .ISO (with the appropriate utility, which isn't hosted here at RH.net to my knowledge, and probably for good reason). Running a search on "cso to iso" should net you what you need. Load your .ISO into a program like ISOBuster and you'll be able to extract individual files. The file I'm looking at is the meat of Suikoden II-PSP, which is "gsd2.bin."

2. gsd2.bin is an archive of GZIP-compressed files. Open it up in a hex editor and you'll find it starts with a table of some kind, probably a series of pointers to all the compressed files. Each file begins with the header Rolen47 mentioned earlier in this thread. You can extract the file via copy/paste in your hex editor, save it, and then decompress it in WinRAR. For example, there's a compressed file tagged VA01.bin that lies at address 0x14000 in gsd2.bin. Copy the data from 0x14000~0x6F000 and save it as its own file. Open up that saved file in WinRAR and extract it, and budda-bing, you might just be looking at the first portion of the game script!

I guess I'll see if I can figure out the text encoding scheme Suikoden II uses real quick. What program do you use for Japanese-language relative searches?

EDIT:
Anyone up for a quick Kanji-identifying exercise? I'm trying to develop a table for Suikoden II (PSX) - Japanese version, and it would help if I could identify the characters shown below in preparation for a relative search. Despite my best efforts I've been unable to hunt these down in the Shift_JIS table:

Just having the first six Japanese characters in the white text display would be awesome. The very first bracket symbol should have a Shift_JIS value of 8175, so I'm looking for the six characters after that.

There's huge runs of bytes shared between the Japanese version of VA01 (PSX) and VA01 (PSP), so I'm growing more confident that the script's very easily accessible. Now I just need to prove it.

[DarknessSavior]

For relative searches, I use monkey-moore. You have to input the Japanese character set by hand in the custom character thing, but it's the best relative searcher out there.

「気がはやいな、あいう。もう着がえたのかい？」

Not entirely sure if that second kanji is 着 or not. But it looks like it.

~DS

[FaustWolf]

Thanks DS! Ugh, looks like this won't be easy. I'm fairly sure I've found the first line of text in the PSX version at address 0x6EA8 in VA01.bin (J-PSX). Two problems, unfortunately: this line of text isn't in the PSP version of the file ( ) and secondly the text encoding scheme looks pretty bizarre ( ).

Check out Jowy's first line here again, clearer pic this time:

In hexadecimal, this reads: DF 01 0C 73 5F 69 47 5A E4 05 00 E5 0E 68 48 02 26 73 49 55 5E 4B 47 E8 E0 00 00 00. The 05 00 should be an opcode that inserts the player's name (I chose "!!!" to keep the Japanese characters distinct from the player name here). DF and E0 should be opening and closing quote brackets respectively. But look at the meat of the dialogue...eugh. It doesn't seem to be encoded in any logical order, and there's just one byte per Japanese character instead of two; is this even possible? With one byte there's only enough room for Hiragana and Katakana, plus a select punctuation marks I think.

Well, actually, I take back the one-byte observation -- looking at this again, the character "い" repeated in Jowy's textbox only matches up if the first Kanji, "気," is interpreted as a two-byte value. Is it possible Hiragana and Katakana are one-byte values, with two-byte values reserved for Kanji and certain text display opcodes?


In any case, the line isn't in the corresponding PSP file, which is disappointing. Turns out the matching data I was looking at earlier in these files doesn't conform to the text encoding scheme either, so what I was examining was probably non-text data and a red herring. I'm not prepared to concede that the text isn't in these files in the PSP version yet, but it would appear the encoding schemes are probably different.               

[Tauwasser]

Well, actually, I take back the one-byte observation -- looking at this again, the character "い" repeated in Jowy's textbox only matches up if the first Kanji, "気," is interpreted as a two-byte value. Is it possible Hiragana and Katakana are one-byte values, with two-byte values reserved for Kanji and certain text display opcodes?

Very much so. However, I have also seen games on recent consoles (GBA) that will have a table of thirty or so kanji addressed in two-byte values and mapped to one-byte values for the text following it. Also, many games will build their table from first appearance of a character in dialog, so it doesn't need to be ordered, though it would be neater. So if you see inconsistencies in character addressing, you might as well check out if there is some loading mechanism that will swap characters. Also, the most-used kanji might have gotten one-byte values by default, even tho the rest got two-byte values.

cYa,

Tauwasser

[FaustWolf]

*insert facepalm here* I should have tried this to start out with, and that would have saved a lot of time...

DS, your text is sitting right there in the PSP files in Unicode buddy! Little Endian style.   There's some opcodes assigned to unused values in the Unicode scheme I think, and these will need to be figured out of course. Check out some data pulled from near the very end of VA01.BIN (PSP):



 
0C 30 17 6C 4C 30 6F 30 84 30 44 30 6A 30 01 30 42 26 31 32 02 30 20 22
82 30 46 30 40 77 4C 30 48 30 5F 30 6E 30 4B 30 44 30 1F FF 0D 30


Someone might want to double-check me on this, but I think it's right. The bytes "42 26 31 32" insert the player's name, and there might be a blank space opcode in there. The bytes "20 22" seem to be a carriage return command.


Unfortunately the file system for gsd2.bin doesn't seem to be readable by ISOBuster; I was hoping you'd be able to use a generic CD reader tool to extract the files, and rebuild gsd2.bin as if it were a generic CD. But since you're only dealing with 26 files, the more important concern will probably be re-inserting English text according to whatever framework and pointer tables the game engine might be using.

It seems there have been a lot of complaints about the English translation for Suikoden II especially? Does this mean you'd prefer to launch a re-translation project rather than use the English text in Suikoden II-PSX?

[Tauwasser]

Someone might want to double-check me on this, but I think it's right. The bytes "42 26 31 32" insert the player's name, and there might be a blank space opcode in there. The bytes "20 22" seem to be a carriage return command.

For a better table see the Unicode Code Charts. Also, see the UniHan database, where you will find most Han (CJKV) ideographs listed with presentational form pictures and all Unicode properties.

Also, your format is UTF-16 LE, sorry I didn't look at it earlier, but you posted a different string above.

cYa,

Tauwasser