Romhacking.net

Romhacking => Newcomer's Board => Topic started by: thescarfinator on August 11, 2022, 12:15:17 AM

Title: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: thescarfinator on August 11, 2022, 12:15:17 AM
Hey! I'm in the middle of planning a Tales of Hearts translation patch project with some friends (Nintendo DS version, not the Vita remake that was officially released in English in a pretty sorry state). My role is to figure out script extraction and re-insertion, but this is my first ROM hacking project and I've spent about a week running around the internet trying and failing to figure out how to rip the dialogue.


My guesses: In the directory, there is a folder called "s" that has a ton of files in it. I don't know if they're sound files (the game is voiced), or dialogue scripts, or both, or something else. Some of the files start with the letter C, many -many- others start with the letter S, so my newbie guess is that C is for the dialogue and S are either all the voices, or maybe they're the skit files and they're separate from the normal dialogue files (may be worth noting there is also a single file there called sound_data.sdat). Both kinds of files are exported without a file extension (though the C files also have matching files with a .lvd extension), Crystaltile claims their file type is unknown, and BatchLZ77 claims that they are not compressed (still can't find dialogue in them in the hex editor though...)

Otherwise, from my understanding it sounds like it could be in the ARM9/ARM7 or overlay. Either way, I haven't been able to get the script, and I'm honestly not sure where else I'd even look at this rate so I finally decided it was time for me to run to a forum for help.

Anyone have any suggestions on what I could do?

I'll also post some pics of what the file directories look like + upload the s folder I mentioned (zipped) + my table if anyone wants to take a direct look at them

https://imgur.com/a/CJhPsWK
https://mega.nz/folder/hLlw0DJK#zJAwe2MBdBV-2nMGpNKomQ

Thanks in advance to anyone who may be able to help!
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: Jorpho on August 11, 2022, 10:29:25 AM
I see QuickBMS (http://aluigi.altervista.org/quickbms.htm) has an entry for Tales of Hearts, but it is unclear if it is only for the PS Vita version. (I suppose that's kind of likely.)

QuoteI do have a table file; the game seems to use Shift JIS so idk if I needed a custom one or not, but I ended up making it anyway for good measure.
How did you make a table file without finding any text..?
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: thescarfinator on August 11, 2022, 11:39:39 AM
Yeah, I did find that QuickBMS script! I was wondering the same thing about whether it was for the original or for the Vita remake; on one hand the Vita remake is titled "Tales of Hearts R" so something simply titled "Tales of Hearts" Should refer to the original DS version, but also I couldn't get it to work with my ROM so either it was mislabeled or I was using it incorrectly

I made the table by cracking the font open. When I opened it in NFTRedit, it turned out that the characters were all already mapped (and thank god for that because there was a LOT of kanji oh my god). I double checked it in the hex editor too since I did have some text that showed up even if it was only menu text.
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: Jorpho on August 11, 2022, 11:54:57 PM
I mean, does the game use the same font for the menu as for everything else? It's conceivable that the game uses one table for the menu text and another table for other text.
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: thescarfinator on August 12, 2022, 12:24:29 AM
There's three different fonts! trialFont12.NFTR contains what seems to be the primary font. trialFont10.NFTR is a smaller, but otherwise identical version of trialFont12. trialFont8.NFTR is another tiny font, with almost everything removed: there are only numbers, the romaji alphabet (upper case and lowercase), and punctuation here. All the characters that each font shares has the same hex mapping, however (because of using Shift JIS standard, I presume), so my guess is this isn't a factor. Always good to double check these things tho, if I'm wrong please correct me

CrystalTile2 was also providing me with a preview look at the legible text sections I was able to find (man bless that program's heart), and the hexes matched the mapping I found in NFTRedit, so I'm assuming it would have shown things from any of the fonts

EDIT: Another note about that BMS script, and something important I completely forgot to mention: Tales of Hearts DS actually has two different versions, with the only difference being the animated cutscenes + opening animation. One of them (anime edition) uses the franchise-typical anime style 2D animation. The other one (CG movie edition) uses 3D animation in a similar pseudo-realistic style that games like Final Fantasy use. While they're mostly identical... if the BMS script doesn't specify which version it's for, I have a sinking feeling that it's gonna be for R because R did not have a CG movie edition; there was only one version of the remake and it was based off the anime one

(on that note, I'm working with the anime edition right now while trying to figure this out; we plan to make patches for both versions tho)
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: thescarfinator on August 25, 2022, 06:55:53 PM
Important update: I found the real location of the script. It's in a single file called fcsrc.dat

It's compressed, so it only shows up in the CrystalTile2 hex editor in disembodied bits of dialogue. That's why me searching for specific phrases wasn't working. It's fragmented, but bits are Just intact enough for me to tell that this is what I need to be looking at. Probably.

Anyway, I can't figure out how to rip from it. I can extract and decompress the file, but I can't find any resources on ripping scripts from an individual file.

I am assuming that I will need to find a method of script extraction from the full rom that involves getting the text out -and- decompressing it at the same time. Is there a way to do that with Cartographer that I'm not seeing? Or some other tool? Do I need to use Kruptar and a plugin or something? Am I gonna have to learn how to write my own program?

If anyone's able to help/point me in the right direction, thank you so much in advance (if it helps ppl answer at all, like a lot of DS things, this script file uses LZZ7 compression)
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: Jorpho on August 26, 2022, 12:24:21 AM
Quote from: thescarfinator on August 25, 2022, 06:55:53 PMI can extract and decompress the file
What exactly are you using to decompress the file?

QuoteI am assuming that I will need to find a method of script extraction from the full rom that involves getting the text out -and- decompressing it at the same time.
What makes you think that? If you can already extract and decompress the file, then you can work with the extracted and decompressed file. The tricky part will be putting it back into the ROM after you finish editing it with Cartographer or whatever – which is why it would be useful to know what you are using to decompress the file.

QuoteAm I gonna have to learn how to write my own program?
You're going to have to figure out what the different control codes in the script are supposed to do, probably through trial and error. There's no getting around that (unless there's a similar game that uses exactly the same control codes that someone else has already deciphered).
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: thescarfinator on August 26, 2022, 01:45:11 AM
QuoteWhat exactly are you using to decompress the file?

I was using BatchLZ77 to decompress the file!

QuoteI am assuming that I will need to find a method of script extraction from the full rom that involves getting the text out -and- decompressing it at the same time.

In the poking around I'd done, I didn't find any answers about extracting from already extracted files + people kept talking about plugins and such. I admit that was a hasty assumption, though; Google is not always a good friend, after all

Does knowing I used BatchLZ77 help at all?

QuoteYou're going to have to figure out what the different control codes in the script are supposed to do, probably through trial and error. There's no getting around that (unless there's a similar game that uses exactly the same control codes that someone else has already deciphered).

Roger that! (btw I didn't mean to sound like I was unwilling to learn advanced programming, I realized I might've come across that way whoops; I only said that because I'm at a low level atm. I'm more than willing to learn advanced techniques, and fully intend to! But since it'll be a while before I get there most likely, I'm first trying to explore my options before I resign myself to needing to learn a coding language to write a special program)
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: Jorpho on August 26, 2022, 10:07:01 AM
Okay, it looks like BatchLZ77 supports re-compression, so why not start with a simple test? Use a hex editor to make an obvious single-character change somewhere in the extracted and decompressed file, and then compress and re-insert it back into the game, and see if it still works – and that you can find your change.

You could do the whole translation with a hex editor if you really wanted to, but that would get tedious.

(It occurs to me that it's quite likely that the game shares features in common with Tales of Innocence, but I would have no idea if the tools used for that translation are available, or how to contact the people involved.)

ETA: Thinking about it some more, didn't they release at least a partial patch for Hearts?
https://www.crystalmods.com/index.php/tales-of-hearts-ds-english-patch/
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: thescarfinator on August 28, 2022, 01:58:01 AM
Quote from: Jorpho on August 26, 2022, 10:07:01 AMOkay, it looks like BatchLZ77 supports re-compression, so why not start with a simple test? Use a hex editor to make an obvious single-character change somewhere in the extracted and decompressed file, and then compress and re-insert it back into the game, and see if it still works – and that you can find your change.
I took a look into the decompressed files and found... that the hexes in them were even more garbled than they were while they were still compressed. I must have used the wrong decompressor... Iirc CrystalTile2 is supposed to have a decompression option, but I hadn't figured it out yet. I might try that next, hopefully I'll manage to get something I can work with

Quote from: Jorpho on August 26, 2022, 10:07:01 AMYou could do the whole translation with a hex editor if you really wanted to, but that would get tedious.
I already did some hex edit tests with the menu text that was in there. It is indeed extremely tedious, and I hear it's super easy to mess up your game doing a translation by hex editing anyway. Definitely not gonna go that route

Quote from: Jorpho on August 26, 2022, 10:07:01 AM(It occurs to me that it's quite likely that the game shares features in common with Tales of Innocence, but I would have no idea if the tools used for that translation are available, or how to contact the people involved.)
Innocence was outsourced to a completely different company, and I believe Tempest was as well. Hearts, meanwhile, was made in-house (and then they outsourced again for the R remakes of Innocence/Hearts)

One of my other friends who's not into romhacking but is programming savvy in general already took a look at Innocence on his own when I was talking about it to him, and reported back that the games do indeed have totally different file structures

Quote from: Jorpho on August 26, 2022, 10:07:01 AMETA: Thinking about it some more, didn't they release at least a partial patch for Hearts?
https://www.crystalmods.com/index.php/tales-of-hearts-ds-english-patch/
Yup! That was a demo patch released several years ago, and ever since then the project has either died or gone on indefinite hiatus. My friends and I started our own project not only to fill that void, but it's also a passion project for us because all four of us really love Hearts DS haha
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: akualung on August 28, 2022, 08:26:13 AM
Quote from: thescarfinator on August 28, 2022, 01:58:01 AMMy friends and I started our own project not only to fill that void, but it's also a passion project for us because all four of us really love Hearts DS haha

I wish you all the best of lucks in this project. After all, only the most passion-fueled projects, made by people who really love the game they're working on, are the ones being completed.
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: Jorpho on August 28, 2022, 12:18:26 PM
Quote from: thescarfinator on August 28, 2022, 01:58:01 AMI took a look into the decompressed files and found... that the hexes in them were even more garbled than they were while they were still compressed.
How you can conclude that they were decompressed at all, then..?

Quote from: thescarfinator on August 28, 2022, 01:58:01 AMYup! That was a demo patch released several years ago, and ever since then the project has either died or gone on indefinite hiatus.
Well, even without help from the developers of the patch or any of their tools, you might be able to look at what was done in that patch and get some ideas on how to proceed.  In some cases it is possible to replace a compressed script with an uncompressed one, for example.
Title: Re: Trying to rip the Tales of Hearts DS script (it's not going well)
Post by: thescarfinator on August 29, 2022, 01:34:48 AM
Quote from: Jorpho on August 28, 2022, 12:18:26 PMHow you can conclude that they were decompressed at all, then..?

If files are not compressed, then BatchLZ77 will reject them and say it's not a compressed file. fcsr.dat, however, was indeed processed and saved as a file called fscr.dat.decompressed

My guess is that either it uses a different sort of compression, or just plain needs another program...... Either way, still haven't had any luck there yet

I wanted to try using a utility on here called "Compression Tools" because it claimed to be very versatile compared to other programs, but couldn't wrap my head around it... It claims to be able to decompress as well, but I could only seem to access the compression options, not to mention all that talk about algorithms that I hope I wouldn't have to touch but I'm not optimistic about that

Quote from: Jorpho on August 28, 2022, 12:18:26 PMWell, even without help from the developers of the patch or any of their tools, you might be able to look at what was done in that patch and get some ideas on how to proceed.  In some cases it is possible to replace a compressed script with an uncompressed one, for example.

That is a very good idea, and also in retrospect a very obvious one! Thanks for that whoops

After reading your reply, I went and did that just that. Being able to read even bits of the dialogue in my native language was very Very helpful. I found out two things:

1. fcsr.dat is not actually the only location story script is kept in. In fact, this appears to actually be for skits specifically. I'm assuming that "fcsr" stands for "face chat script"
2. the cutscene script appears to be in a file called m.dat, which is also compressed
3. the demo patch seems have inserted the scripts in exactly the same way the original had them (compressed scripts I can't fully get into, in other words)