Romhacking.net

Romhacking => ROM Hacking Discussion => Topic started by: Xanato on December 20, 2015, 12:24:28 am

Title: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on December 20, 2015, 12:24:28 am
Yep. That infamous problem has pulled the plug on my plans of changing the dialogue of an old RPG in my collection. From what I've read, you need to know how to do programming in order to decompress the text. Unfortunately, that's a skill which I don't really have. Is there an easier way of getting around this or am I out of luck?
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: pablitox on December 20, 2015, 01:02:49 am
It'd be most useful to know the rpg name.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on December 20, 2015, 01:06:38 am
It's The Legend of Dragoon.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: BlackDog61 on December 20, 2015, 01:25:27 am
And what have you investigated that leads to the conclusion of compressed text?
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on December 20, 2015, 11:17:28 am
I've done a lot of relative searches of the BIN file with SearchR X and tried to use Thingy32TW with some table files that I've made. I was able to find text for the additions (fighting skills) in an overlay file, some messages pertaining to Sony Computer Entertainment America, and what appeared to be instructions on which files to load at certain points, but I couldn't find any dialogue from the main game. Using unique or rare words that appear in the script didn't help, either.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: BlackDog61 on December 29, 2015, 05:50:21 am
To help you,we need to hear more details:
- which encodings did you try during your Rsearch? SJIS, UTF8...?
- knowing which game this is could definitely allow someone reading you to try his/her own tricks and point you into the right direction.
- which files have you tried this against?
- have you extracted files from the disk? Do they look like they are archives?
- when you say you've used your own home-made tables, what logic did you use to make them?
- have you tried sjis_dump (or the newer program with some standard tables like UTF8 and UTF16, even if I'd be surprised if an old PSX game used that)? http://www.romhacking.net/forum/index.php/topic,20776.msg291898.htm

FYI: finding some ASCII-encoded "printf" format strings (anddebug strings) is good, but that doesn't give a definite info for the encoding of most Japanese text. I've seen a game with ASCII encoded format strings and yet custom-encoded (=custom table) text.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on December 29, 2015, 06:21:02 pm
- I'm not sure which encoding(s) I've used. I just checked the Help section and Readme file for SearchR X 1.0 and didn't find anything about encodings.
- As stated in my second post, the game is The Legend of Dragoon.
- I've mostly tried relative searches on the entire BIN file. Some were done on an overlay file(s).
- I'd ripped an image file from the first disc of the game and mounted it to a virtual drive on my PC. Yes, the file contains folders with different contents.
- Some of my relative searches brought up bits and pieces of additions (fighting skills) that were all listed around the same spot. After doing some research and finding out that some hex values can represent multiple characters, I attempted to fill in the blanks and found the list becoming more complete as I added to the table file. In addition, messages about Sony Computer Entertainment America in other overlay files were also getting more filled in.
- I haven't even heard of sjis_dump or the newer program, but I'll check it out.

- I've realized that the additions list and company messages are a dead end, but it was my first time decoding hex in a game, so at least I know I'm doing something right.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: yugisokubodai on December 30, 2015, 05:59:09 pm
My experience with the PSX tell me that you need to search in a state file.
It's better than looking in the Rom.
An emulator with Vram viewer like Agemo is useful, too.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on December 30, 2015, 07:45:01 pm
Do you mean a save state file, Yugisokubodai? If so, how would that work?
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: BlackDog61 on December 31, 2015, 03:57:24 am
In general, don't look for strings on the CD image. Extract stuff inside it and then look for strings.
Sjis_dump finds stuff in some files, so I'd bet for SJIS table.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on December 31, 2015, 10:28:30 pm
I've run searches on files within the BIN itself, too. Aside from a few overlay files and SCUS_944.91, nothing familiar comes up. I'll try to use SJIS Dump and see if that uncovers something.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: yugisokubodai on January 01, 2016, 05:27:35 am
Do you mean a save state file, Yugisokubodai? If so, how would that work?

Yes, I mean the save state. You better use PSX1_13's save state than ePsxe. If you're using epsxe's save state, then you must rename the extension to 7z and un-rar it.

Usually, if the text data is compressed, then it will be decompress when displayed on the screen. Take a state when the text is displaying, open your Hex editor and browse to that state file, search for the string.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on January 01, 2016, 10:57:03 pm
Would a relative search utility also work or does it have to be a hex editor? I just used SearchR X to check two quick saves from the game, but I couldn't find the dialogue on the screen.

BlackDog61, what's the command that I should use for SJIS Dump? I couldn't really find any instructions for using it.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: BlackDog61 on January 02, 2016, 04:12:56 am
Would a relative search utility also work or does it have to be a hex editor? I just used SearchR X to check two quick saves from the game, but I couldn't find the dialogue on the screen.

BlackDog61, what's the command that I should use for SJIS Dump? I couldn't really find any instructions for using it.
As I suggest in my review: sjis_dump 4 0
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on January 02, 2016, 10:52:31 am
I have read your review and did not understand the command that was posted. Could you please elaborate on it? This is my first time working with that kind of software, so bear with me.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: BlackDog61 on January 02, 2016, 04:23:22 pm
sjis_dump will try and interpret bytes from the file as SJIS-encoded Japanese text. Not all bytes are even possible SJIS characters, which is what sjis_dump uses to try and detect "real" text. Of course that doesn't work most of the time. :)
The first parameter ("4") tells the program to only dump sequences of 4 valid SJIS characters or more. (It ignores any shorter sequence.) The higher this parameter, the least random data gets interpreted as SJIS. Still, there's goingto be a lot of "not text" dumped.
The second parameter tells whether ASCII is considered valid or not. If you're looking for Japanese text, in general, it's a good idea to not dump ASCII. (There are a lot of bytes in the ASCII range in random data.)
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: yugisokubodai on January 04, 2016, 08:49:21 am
Would a relative search utility also work or does it have to be a hex editor? I just used SearchR X to check two quick saves from the game, but I couldn't find the dialogue on the screen.

BlackDog61, what's the command that I should use for SJIS Dump? I couldn't really find any instructions for using it.

Chance that you're running with a game that uses DTE, MTE or simply, encrypted.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on January 06, 2016, 05:34:57 pm
Okay. How would I deal with that kind of compression/encryption?

Also, I've tried using SJIS Dump on the BIN file, but haven't been able to get it to work. I've gone to the folder that contains the BIN file and entered the following command:

sjis_dump.exe "LoD 1.bin" 4 0

A Windows error message comes up saying that sjis_dump.exe has stopped working. What should I do?
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: KingMike on January 06, 2016, 05:49:29 pm
I have not used this program but it sounds like people are saying DON'T START BY SEARCHING THE ISO.
That could be a reason it's crashing. (if the program is not designed to support full ISO files that are hundreds of megabytes)
Just going from what I read.
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Vehek on January 06, 2016, 07:08:08 pm
You also have too few parameters. You left out the output file name, which goes between the first file name and the threshold. (A probable mistake in the source code stopped it from spotting that error and warning you.)
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on January 06, 2016, 07:51:11 pm
I have done as you suggested and succeeded, Vehek. There's an output file now, but it's filled with countless instances of "Position: (random gibberish)." How do I make sense of this (if at all possible)?
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: BlackDog61 on January 07, 2016, 05:32:28 pm
1 - don't run it on the ISO, run it on the extracted file. (As well spotted & re-stated by KingMike.) Have you done that?
2 - get familiar with Japanese, enough to recognize "garbage" from potentially meaningful Japanese. (Ex: kanji are often intermixed with hiragana/katakana. And punctuation is common in sentences - even if it is different from English punctuation.) Then take the time to browse the entire dumped files and look for "the real thing". (Google translate can help verify stuff makes sense, sometimes but not always.) You could try and look for actually existing game messages you've seen on screen, if you take good care not to mistake one kanji for another, and it you choose a sequence that doesn't involve English characters (just in case - we don't know how they are encoded and you chose to not dump ASCII).

Just for comparison, if relative search doesn't work, then you should be browsing the entire extracted files. :)

EDIT - I forgot to mention that you have to set the encoding to SJIS when you browse the dumped files.(Ex: in Notepad++, just after opening the file and without typing anything in it, menu->encoding->character sets->Japanese->SJIS)
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on January 08, 2016, 11:20:36 pm
Should I put SJIS Dump and the output file in the same folder as the extracted file or does it not matter? If I do have to put them in the same folder as the extracted file(s), will that have a negative effect on the ability of the game to function/be modded on my PC?
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: BlackDog61 on January 20, 2016, 02:07:18 am
You can do as you want. It willonly change the contents of the command you run - since you have to give the path to the file.
It's probably easier for you if you "move the program" into the directory before you execute the command?

None of this affects your ability to edit the game in any way. In fact, having adump of the text doesn't give you ability to edit it, it only helps find which files have text inside. (And that is step 1 to anything else.)
Title: Re: Dealing with Compressed Text in PlayStation RPG
Post by: Xanato on January 23, 2016, 08:03:08 pm
When I asked about the possibility of it negatively impacting the game, I meant more along the lines of having a foreign file mixed among sensitive game data. I was concerned that, should I ever choose to test any modifications by running the game, the game would not function properly due to the presence of such content. I know that changes to a text dump will not be reflected in a game.

What type of folder/file should I be checking for the text? What size would the file most likely be? So far, I know that the STR folder has movies and the XA folder has audio.