Romhacking.net

Romhacking => ROM Hacking Discussion => Topic started by: paul_met on September 11, 2013, 01:26:43 pm

Title: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on September 11, 2013, 01:26:43 pm
Greetings to all.
Many people probably know about this wonderful game Legend of Oasis or the Story of Thor 2, which was released on the console Sega Saturn. I've been wanting to translate this beautiful game in my native language. But the problem is that the fonts of dialogs somehow packed. I was able to find the exact location of the font. It is located in the file CHR.BIN from address 0x4A800.  I find it difficult to determine exactly where the package ends.Please help us to unpack this compressed font. Only this font limits the beginning.
Here is the uncompressed font, taken from the dump of video memory.
(http://i58.fastpic.ru/big/2013/0911/be/f9eba69ab47abf934cedd50e431c6ebe.png)
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on September 21, 2013, 08:59:16 am
No one is interested or can not cope?
If it helps, here's the archive files. There is a binary file with a font that has been extracted from the video memory. Also in the archive there is a palette file for clarity. Font format 4bpp linear + Row interleaved.
Link:http://rghost.ru/48882097 (http://rghost.ru/48882097)
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 02, 2013, 06:36:28 am
I worked with this game some years ago to translate to spanish language using a modified font including all spanish chars, but I never found the font because I'm not a Saturn expert. :banghead:

Now I analized again the game and I was able to decode the data. :beer:
Also, I can encode the data using a fake compression to modify the graphics directly with any of the usual tools. :thumbsup:

But I have some questions: :huh:
- where did you get the offset 0x4A800? I need change some offsets to modify some graphics ("Game Over")
- the game uses a variable font width, where the data are stored?

Some images using a grey palette because I don't know where the real palette is stored and is not needed:
(http://imageshack.us/a/img837/7528/xxzv.png)

(http://imageshack.us/a/img713/2825/e1fc.png)

(http://imageshack.us/a/img716/2691/t1ey.png)

(http://imageshack.us/a/img405/6365/gc0b.png)

(http://imageshack.us/a/img20/6751/1imk.png)

(http://imageshack.us/a/img42/8402/tbbq.png) <--- The extended chars are not usables but all spanish chars are added  ;D

(http://imageshack.us/a/img404/3986/4wug.png)
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 02, 2013, 11:15:26 am
Wow, great job! Can you share your tools for unpacking / packing of resources? I know where to get the palette.
At the expense of Offset "0x4A800", it can be calculated from the size for packed data. But I did not yet know. I figured it by eye. I found a palette for fonts and loaded it into a tile editor. With the right palette to locate the font was not hard. Unfortunately I do not know yet where the offset value of compressed images and font widths.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 03, 2013, 03:38:10 am
I'm finishing a tiny tool to replace the 2 decoded fonts and check that the new CHR.BIN works.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 03, 2013, 11:14:10 am
Very well, I'll wait for the final result. :)
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 03, 2013, 03:15:38 pm
Bad news: the game don't run with the modified file. Maybe the pointers are needed, same as a Saturn expert  :(
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 03, 2013, 04:45:07 pm
Repacked font was larger in volume than the original? The remaining compressed data were displaced?
I still would like to look at the your tool.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 04, 2013, 05:13:08 am
Repacked font was larger in volume than the original? The remaining compressed data were displaced?
I still would like to look at the your tool.
The file can be greater or less than the original, but the game crash. All data are OK, and I can decompress the data without problems, but the game don't accept the new file. I'll try another encoding to see if it works.

I can't help you If this is a pointer problem. A Sega Saturn expert is needed.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 04, 2013, 12:40:26 pm
And if you adjust the size of the new compressed data under the old - the game still will not start?
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 05, 2013, 06:14:16 am
The game only works with the same encoded size. Adding o removing a single encoded byte causes a crash. I can expand the file to make it bigger, adding zeros at end, and the game run fine, so the file can be expanded. The pointers to the data are needed.

Tomorow I upload all, and a brief explanation of the compression. It's a very simple method.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 05, 2013, 07:06:59 am
The game only works with the same encoded size. Adding o removing a single encoded byte causes a crash. I can expand the file to make it bigger, adding zeros at end, and the game run fine, so the file can be expanded. The pointers to the data are needed.
One man had promised me to share information about CPU registers of SH2. I think if I get this information, I can find the necessary poiters.
Yet I found pointers of dialogs text and labels and know the necessary offset.

Tomorow I upload all, and a brief explanation of the compression. It's a very simple method.
Thank you very much.

Some progress. I calculated the pointers, who is responsible for the coordinate of the withdrawal of the font. In our case, it does not give anything special, but it's something. I would like to find more values ​​corresponding to the size of the font.
Spoiler:
(http://i57.fastpic.ru/big/2013/1005/b2/b230c19734d02e25d6acb05776c79bb2.png)
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 06, 2013, 11:29:03 am
Yet I found pointers of dialogs text and labels and know the necessary offset.
Do you have all text codes? I remember various pause codes, center code, some XXNNNN codes, where NNNN is the offset to another message, ...


The compression starts @ CHR.BIN:0x043000, after a big Shift-JIS font not used, until end of file.

You can see various blocks, sector-aligned (0x800 bytes), padded with 0x00:
- 0x043000, 1 file, I have named as 100
- 0x046800, 1 file, I have named as 200
- 0x048000, 1 file, I have named as 300
- 0x049000, 1 file, I have named as 400
- 0x049800, 1 file, I have named as 500
- 0x04A800, 12 files, I have named as 600-611, the first two are the fonts

Each encoded file has a 2-bytes header with the encoded size (low-endian value). The first file has a bad size, so maybe is not needed.  Following the size value are the chunks. The 3 upper bits of the first byte of each chunk defines an "action":
Code: [Select]
(the numbers are in binary)

code 000
  - 000+00000 ..................... end of data (0x00, as usual)
  - 000+nnnnn ..................... copy next 'nnnnn' encoded bytes

code 001
  - 001+mmmmm nnnnnnnn ............ copy next 'mmmmmnnnnnnnn' encoded bytes

code 010
  - 010+0nnnn bbbbbbbb ............ repeat 'nnnn+4' times byte 'bbbbbbbb'
  - 010+1mmmm nnnnnnnn bbbbbbbb ... repeat 'mmmmnnnnnnnn'+4 times byte 'bbbbbbbb'

code 011 (previous code must be '1nn' or another '011')
  - 011+nnnnn ..................... copy 'nnnnn' more decoded bytes

code 1nn
  - 1nn+xxxxx yyyyyyyy ............ copy '1nn' decoded bytes from offset $-'xxxxxyyyyyyyy'

The tools: http://www.mediafire.com/?tgj90qss6srp5z5 (CHR.BIN is not included)
You are free to use, blah blah blah, ...

Run "dec.bat" to decode all files. A log file is created, "decode.log".

"font.exe" replaces the 2 encoded fonts by the 2 decoded fonts, but you need seach and modify the pointers.

Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 06, 2013, 12:35:23 pm
Text dialog pointer has a simple, consisting of 2 bytes and presented in the form of addresses beginning of the line. A variety of labels are offset 0x2da000. So much for the file TH2.LOW. As for 0TH2.BIN - offset is 0x4000.

Thank you for the program, I will look into it and if there are questions, I'll accomplish your goal! ;)

October 07, 2013, 11:05:22 am - (Auto Merged - Double Posts are not allowed before 7 days.)
How about of the real compression of extracted data (not fake)? And if possible, is it possible to adjust the size of the new compressed data to the size of the old?

October 08, 2013, 11:05:52 am - (Auto Merged - Double Posts are not allowed before 7 days.)
I checked - fake compression does not work. I rebuilt the disk image that you can use the new CHR.BIN. But the game freezes after you select "new game". Then I changed the font number 2 to the one that was used in the Japanese version of the game. These fonts are different in size, but with the substituted font game worked perfect. Conclusion - in the case of the font replacement pointer is not needed, since the data is decompressed sequentially. We just need to compress the fonts correctly.
PS: By the way, a font number 1 can be ignored, as it is not used.

Spoiler:
(http://i60.fastpic.ru/big/2013/1008/23/784ec5e232ed7bb6cd0e1b6848435a23.jpg)

And another question - what is meant by the two bytes after the size of the compressed data? For example: "28 00"? Why are they the same for both fonts?
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 08, 2013, 02:50:51 pm
Tha japanese font is smaller as the english font and seems to work.

You can add 1 byte:
- change 0x0004AB9C, from 0x17 to 0x18 (1 more byte)
- change 0x0004B5B2, from 0x79 to 0x78 (copy 0x18 more bytes instead 0x19)
- insert byte 0x61 before 0x0004B5B3 (final font) -> copy 1 more byte (0x18 before + 0x01 now = 0x19, same as the original font)
- remove 1 0x00 from the end of file
The font is the same, but now the game crash.

2800 -> 0010100000000000 binary -> code=001 value=0100000000000=0x800 -> copy next 0x800 decoded bytes
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 08, 2013, 05:37:52 pm
So what do we do now? What are my options? Real compression applied?
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 09, 2013, 04:00:02 am
The decompression seems correct, is a very simple RLE+LZ variation, so the optimal compression, not "real", has the same problem. If you exceed one byte the game crash :(

You need to know how does the game, if there is a limit, if pointers are used, ...
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 09, 2013, 10:58:37 am
You were right. If exceeded the size of the compressed data of the original, the game will not work. Although decompression is performed after all - I learned this by watching the video memory.
I think there is another way - to reduce the size of the font data by removing superfluous. Then compress the data. The size of the new compressed data is guaranteed to get less than the original size.

October 11, 2013, 12:07:34 am - (Auto Merged - Double Posts are not allowed before 7 days.)
By the way, your tool will not decompress all the data in the CHR.BIN. At the beginning of the file is still compressed logo and title screen. Is it possible to make a tool to request the address from which we should start unpacking?
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 11, 2013, 02:58:02 am
I just decoded the final section, where the font is stored, looking for more fonts.
You can decode more sections simply adding: Decode(chr, offset, number); (number is used as file name)

Tomorrow I modify the tool to accept the offset as parameter.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 11, 2013, 10:58:56 am
I just decoded the final section, where the font is stored, looking for more fonts.
You can decode more sections simply adding: Decode(chr, offset, number); (number is used as file name)

Tomorrow I modify the tool to accept the offset as parameter.
All right.
Tell me, you do not want to try to write at least a simple data compressor? Unless of course you have the time for it.
I finally got a file with documentation of Saturn. I will study it. If you're interested, I can share the information.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 12, 2013, 02:57:09 am

Tell me, you do not want to try to write at least a simple data compressor?
No make sense. There is no guarantee that the new compressed files have less size as the original encoded file.

All: http://www.mediafire.com/?92mje9bf6zf2f9g (decode2 is the new tool, source and executable included)

Syntax: decode2 filename offset
"filename" is the file with the encoded data
"offset" is the offset where the encoded data starts
Output: a filename named "xxx.raw" with the decoded data. "xxx" is the 8 digits hexadecimal offset.

"test.bat" is an example to decode/extract some files
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 12, 2013, 03:40:24 am
No make sense. There is no guarantee that the new compressed files have less size as the original encoded file.
I do not agree. Let's understand then. In fact you need to repack just two blocks from the compressed data. This is the font number 2 and the inscription "Game Over". All of these units, we can reduce (from the font throw pacifiers, and the inscription itself make less). Optimal packing new data, we have to get up to 99% smaller.

All: http://www.mediafire.com/?92mje9bf6zf2f9g (decode2 is the new tool, source and executable included)

Syntax: decode2 filename offset
"filename" is the file with the encoded data
"offset" is the offset where the encoded data starts
Output: a filename named "xxx.raw" with the decoded data. "xxx" is the 8 digits hexadecimal offset.

"test.bat" is an example to decode/extract some files
Thank you. I found a trick. The title screen is divided into 2 parts, which are separated from one another byte "0x01". If it is not replaced by "0x00", the program will give an error. Incidentally, the palette is in the same file, and it is also compressed.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 12, 2013, 05:56:44 am
You are wrong. You just want to modify 2 images to translate the game to your language, but what happens if I want to modifiy other graphics (icons, title screen, ...)? I don't want to create a tool to work with only 2 graphics.

Anyway, it's a very simple compression and the method and the decoded source code is available, so anyone can create a compressor.

Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 12, 2013, 08:41:18 am
Your position is clear.
In that case, I would like to learn more on the example of what it means:
Code: [Select]
- 011 + nnnnn ..................... copy 'nnnnn' more decoded bytes"
October 13, 2013, 12:33:32 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
That's interesting. By reducing the size of the font #1 to 4 bytes, managed to increase the space for the font #2 to 3502 bytes. It seems that a limitation is imposed only on the amount of all the blocks of compressed data that come one after another. But not on a separate block.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: CUE on October 15, 2013, 11:05:31 am
Your position is clear.
In that case, I would like to learn more on the example of what it means:
Code: [Select]
- 011 + nnnnn ..................... copy 'nnnnn' more decoded bytes"
October 13, 2013, 12:33:32 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
That's interesting. By reducing the size of the font #1 to 4 bytes, managed to increase the space for the font #2 to 3502 bytes. It seems that a limitation is imposed only on the amount of all the blocks of compressed data that come one after another. But not on a separate block.
011 must be preceded by another 011 or 1nn code.
With code_1nn you can copy only 7 bytes maxim ('111'). To copy more bytes you need split the size in some parts, 7 bytes for code_1nn, 31 bytes maxim for each code_011.

An example: If you need copy 50 decoded bytes from offset actual - xxxxxyyyyyyyy:
111 xxxxx yyyyyyyy <-- copy 111 binary = 7 decoded bytes from offset actual - xxxxxyyyyyyyy
011 11111 <- copy 11111 binary = 31 more bytes
011 01100 <--- copy 01100 binary = 12 more bytes

You can check "decoded.log" to see more examples.
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on October 23, 2013, 01:51:34 pm
So, I wrote a simple compressor for the font number 2. I adapted the font size under the current restrictions and everything works. For the title screen and the inscription "Game over" needs an efficient compressor.
Spoiler:
(http://i57.fastpic.ru/big/2013/1023/83/649ffd5ea47e1d39469c3e3aa2a61083.png)

CUE - you have not changed your mind?
Title: Re: Legend of Oasis/Story of thor 2 - compressed font
Post by: paul_met on December 30, 2013, 06:08:42 am
In general, the release took place. Thanks again to CUE!
Spoiler:
http://youtu.be/j1XvEbmJzek