Technical Resources for Final Fantasy VI Text Editing

Started by abw, October 18, 2015, 05:46:05 PM

Previous topic - Next topic

abw

(In any context where it actually makes a difference, what I mean by FFVI in this post is FF3US v1.0.)

After falling off the romhacking bandwagon a few years ago, I've managed to scrape a bit of free time together over the past several months and have made significant progress on my translation of FFVI. Two of the last major items on my to-do list were the opening and ending credits, both of which required some code analysis to know what data values needed to be updated and what values they needed to be updated to. Since I haven't had to do any significant low-level work in a long time, I decided to do things the hard way (i.e. re-invent the wheel) as a refresher course. Now that I'm satisfied with my results, I thought I'd check with you wonderful folks at rhdn to find out how much of the work I've done would be useful for others (a.k.a. whether there are any resources I should have looked at instead of doing stuff mostly on my own :p).

For the opening and ending credits, the commentary on the disassembly on the wiki is pretty great in some places, but is much less so in other places, and I've noticed a couple of errors in both the commentary and the disassembly itself. I see that a few other people have produced patches that successfully alter the credits, so I'm definitely not the first person to go through the trouble of figuring out how the credits work at least well enough to be able to modify them, but it appears that not much of the required knowledge has been publicized. Should I bother updating the wiki with my notes, or would it be better to include/link a better version from somewhere else?

The code and data for the opening credits are stored within a compressed block in ROM and are interconnected with the code and data for 3 other game events. The compression is a variant on LZ77, but there are approximately a bajillion mutually incompatible variants on LZ77, and I didn't see anything on rhdn that dealt with FFVI's specific variant, so I wrote a little script to handle decompression and recompression. I'm not sure what algorithm Square was using to generate their LZ77 compressed data back in the 1990s, but the one I wrote appears to be better (not by much, but every byte counts, right?). If anyone's interested, I could clean my script up a bit and submit it here.

Unfortunately for me, there wasn't any free space around the opening credits text (which wasn't surprising considering that Square decided to compress the entire block; what was surprising was that despite going to the trouble of compressing the block, Square didn't really structure the decompressed code to take advantage of the compression algorithm [e.g. identical byte sequences from early in the code are repeated later in the code after the original sequence has exited the sliding window], some of the code and data appear to be unused [this assessment is supported by the trace logs], and some of the decompressed code could easily be shortened [in both bytes and cycles!]), so I also converted the entire decompressed block to a .asm file, adding about 500 hundred labels in the process so that I could muck about in the middle of the block without going through address rewrite hell all the time. It's not the world's prettiest .asm file, but it was enough to get the job done, and although I only went far enough with my analysis to accomplish what I wanted with at least a little bit of confidence that I wasn't unintentionally breaking anything else, I did add notes on most of the routines used by the opening credits. If anyone's interested, I could also clean that file up a bit and submit it here.

Besides the assembly-level stuff, I've also got a Cartographer script which (unless I've missed something) handles extracting every non-compressed, non-graphical text string in the game. It is of course subject to Cartographer's quirks and isn't a magic wand for changing the lengths of text strings embedded in assembly code, but it's a good place to start (and a variant of that script combined with something like this would make for an awesome starter kit, no?).

Madsiur

I did not saw this thread before today...  :huh:

If you did not updated the credits info on DataCrystal, I'd like a version of your notes. They could be handy for me. I'm also interested by your LZ77 algorithm if possible.... and the asm file too! (if not already uploaded here)

Nice to see that someone has done work on the credits :)


abw

I had updated various parts of the ROM map on Data Crystal earlier, but I've just added a disassembly of the code/data sections used by the opening credits and 3 other game events and submitted my LZ77 decompressor/recompressor and relocatable asm source code, so those should hopefully show up on the main page during the next documents and utilities updates.

Madsiur


Gi Nattak

#4
Any chance you could create the same decompressor/recompressor that can work with the J FFVI ROM? I've got the FFVI title screen patch applied and it seems this utility does not work with that. I've thought to revert the title screen back to the US one, but I am unable to without breaking my ROM now do to other changes or something, I'm not sure.

abw

Quote from: Gi Nattak on February 15, 2016, 02:39:15 PM
Any chance you could create the same decompressor/recompressor that can work with the J FFVI ROM? I've got the FFVI title screen patch applied and it seems this utility does not work with that. I've thought to revert the title screen back to the US one, but I am unable to without breaking my ROM now do to other changes or something, I'm not sure.

Could you be a little more specific? The differences between the Japanese and English releases come from differences in the compressed data rather than in the C2FF6D decompression routine itself (which is the same in all 3 versions), so unless I made a coding error somewhere, I suspect that's not the issue you're having. Additionally, the address range I was primarily interested in here (C2686C-C28A60 decompressed to 7E5000-7E7C62) doesn't contain much in the way of graphics data (there is some sprite and palette info, though).

Assuming you're referring to this patch, based on the addresses modified, I'm guessing that what that patch does is replace a couple of sections of compressed data in the English version(s) with the corresponding compressed data from the Japanese version. Unless you've got other conflicting patches applied or are starting from a savestate where the affected code/data has already been loaded, probably applying an anti-patch should be enough to safely revert the first patch's changes.

Edit: Oh, sorry, I saw and replied to this thread before I noticed your PMs :P. Since that discussion seems like it would be pretty useful for others too, I'll reply here.

End credits:
The text parts of the end credits aren't compressed, so editing them by hand isn't too hard. If you want to add some text into the middle of them, though, there are probably a couple dozen pointers that need to be updated, depending on exactly what changes you want to make. I took the fast and dirty route for editing the ending credits myself, so aside from a bunch of addresses which are now on the wiki, I don't have much that's particularly reusable to offer there.

Opening credits:
For the opening credits, though, adding text into the middle of that compressed block meant rewriting several hundred pointers, and I definitely did not want to do that by hand, so I sucked it up and spent the time to convert my disassembly into a reassembleable ASM file. At a high level, the process I used to edit the opening credits was this:

  • Extract and decompress the data from ROM to a separate decompressed file (this is where the decompression utility comes in)
  • Disassemble the decompressed file (I've posted the results on the wiki)
  • Convert the disassembled file to reassembleable ASM (which is what this is)
  • Edit the ASM to do whatever you want
  • Reassemble the ASM (I used Asar for this, but other assemblers should work too)
  • Compress and insert the assembled data (this is where the compression utility comes in again)

As you noted, there is a limit to how much space the compressed data can take up in the ROM before it starts overwriting stuff it shouldn't. My utility doesn't specifically do anything about that, but it's easy enough to check the size of the compressed data before you insert it into a ROM (e.g. write the compressed data to a separate file and check its size). As I noted earlier, the decompressed code could be optimized a bit to reduce its size, and it could be reorganized (anything except the first 12 bytes is fair game) to better take advantage of the LZ compression algorithm; I haven't actually tried any of that, but I feel like you could probably get an extra 10% compression without getting too crazy.

Edit 2: I actually tried extracting, decompressing, editing, compressing, and inserting the opening credits text from that patch with my utility, and as long as I don't change any string lengths, everything seems to work out pretty well. If I do change any string lengths, stuff breaks pretty quickly, but that's because of the 500-ish pointers that need rewriting before the code can start doing the right thing again (a.k.a. what the ASM file and assembler take care of).

Gi Nattak

#6
Hello there! Thanks for the reply.

That is indeed the patch I was referring to.

And this is the way I've been doing the process, just to make sure it's not some mistake here:

1) Open command promt and type FFVI_C2FF6D.pl
2) to decompress type: perl FFVI_C2FF6D.pl -m d -ho -s 0x2686C "rotds.smc"
3) copy that hex, paste in ROM at C2686C using HxD.
4) open ROM in Windhex and use FF3OpeningCredits.tbl. Go to C2686C and edit away.
5) export a bin file of the edited C2686C - C294CE credits.
6) to recompress as bin type: perl FFVI_C2FF6D.pl -m c -s 0x2686C "rotds.smc" < decompressed.bin

Here's the decompressed credits with a US ROM:



And how it's looking with the title screen patch applied:



Note that if I use the FF3 decompressed data in my ROM, the names show right like FFIII, but after compressing it back in there are some nasty issues, which is to be expected. Also, I tried with a clean Japanese ROM and it came out looking just like mine (hack). I also tried reverting the Japanese title screen data back byte for byte, and my ROM freezes, so somehow it has adapted itself to my ROM and cannot be easily reverted lol. That, and I noticed when I try to edit the credits in a clean FFIII ROM, and then apply the Japanese title screen patch, it creates some nasty graphical issues. So I doubt I could have them working together which is what I'd really like. Unless of course I'm doing something wrong...

Another small note is the decompressed data is shorter in my hack/ff6j.

Perhaps it's something to do with my custom splash screen, but I wouldn't think so if it's looking the same as the FF6j decompressed credits. And looking at the asm file of the custom splash screen, it actually doesn't appear to conflict with any of the credits data - but it might, I'm not too positive. But yeah If only I could get the credits to appear correctly with my hack, I'd be good to go. :)

Thanks for listening.

EDIT: Just saw your edit here, and am studying your response. So you got it looking all good to edit huh? Hm, perhaps you'll have an idea why mine is looking un-editable from the pics, before I try to discover what might be the culprit. Thanks for helping out!

EDIT 2: So I just went back and tried the FF6j ROM again with the process, and I am wrong it is showing up correctly to edit, so I guess it IS just my hack with the issue ;_;. Hoping you might have some idea, but it's looking like it might have to be found on my side.

Using the Japanese decompressed data and injecting it over to my hack, the names are showing correctly, but I suspect there will be some damage upon compressing it back in, like before. I'd imagine I'd need the decompression to come from my hack, in order to recompress correctly back in.

EDIT 3: Well, actually using/editing the decompressed Japanese ROM data, and compressing it back, then injecting that compressed hex into my hack and it seems to be working correctly! Thought I tried this yesterday, but perhaps got confused and made a mistake. So hopefully I am good to go now after all. :o Still wondering if I'm doing the process as intended though lol, reading your 6 steps it sure seems like not, but hopefully it is okay.

abw

If I'm reading that right, I think the source of the problem you're encountering might be coming from your step 3.

Unless you've also modified the code that handles decompression (starting at C26800), when you overwrite the compressed data with decompressed data, the game is still going to think the data is compressed and will try to decompress the already decompressed data. I'm honestly surprised that that would result in code that doesn't crash and burn almost immediately. Aside from that, since the decompressed data is larger than the compressed data, pasting it back into the ROM will end up overwriting other stuff you probably didn't want to touch.

Instead, make sure you keep the decompressed data in a separate file while you're working on it, and then only put it back into the ROM after it's been compressed again. If you set a breakpoint in an emulator just after the block has been decompressed, you could also paste into RAM for testing purposes.

It looks like your decompressed data is the right size for the English version, so this probably isn't a problem, but just in case, another thing to check is that you've got the right starting address - 0x2686C is for a headerless ROM, so if you're working with a headered ROM (which is what Yazoo's patch requires), you'll have to adjust the address to e.g. 0x26A6C.

Gi Nattak

#8
Argh, today testing everything was going swimmingly, but then I see the end credits cinematics are messed up. So my seemingly simpler method of copy and pasting decompressed & recompressed hex around isn't correct indeed. :'(

Which was, since I can explain it much better now:
to use the Japanese ROM and decompress the block using the utility, copy the hex data the utility provides and paste it into the same ROM using hex editor, load up the table file and edit the start credits, then export that block (C2686C - C29490) as binary data file, then using the utility to recompress the exported binary file back into the ROM. Then, export that newly recompressed data from C2686C - C29490 over into my hack, which uses the ff6j title screen. And it worked great, this method, if not for the end cinematics/credits being eradicated. I'm still holding on to some hope that this is a doable method, but I highly doubt it - I'm just in a bit of denial atm due to investing much time into doing it this way and thinking I was successful lol. I finally had my intro credits all customized and perfect...*sniff*.

I checked to see if it was something I might have done with the new credits like maybe I edited them too much or used a wrong byte(s), by starting over and not doing any actual change at all, and 'recompressing' it back in the same way, but the end credits/cenematics get messed up still, I'd guess from as you said overwriting stuff I didn't want it to. So, it looks I will follow your steps this time - I'm just still having trouble understanding them, probably because I don't fully understand the process, but that's my mission for the day. I do have experience with asar and patching with it, so I should be okay once I fathom the steps. For instance I'm not sure how to go about editing the credit names now, if not by hex editor/table file...do I just change the names in the assembly file, or the data next to the names (which would suck), or both? And is that all I'd do to achieve the same results that I had, or is there more that needs to be done when editing the asm file? Another thing is I had to add spaces to names and credit titles in order for it to line up all nice, would I need to do the same in the assembly file, or would it somehow do it automatically?

In any event, thanks for taking the time to help me understand this, and I shall study your step guide and report back!

EDIT:

1) Extract and decompress the data from ROM to a separate decompressed file

Could I put the decompressed data (hex) into a ROM and create a binary file of it like I was doing? Is this what you mean by a separate decompressed file?

4) Edit the ASM to do whatever you want

Meaning edit the credits, the names there? Since that's all I'm interested in. Or does anything else need to be changed as well?

5) Reassemble the ASM (I used Asar for this, but other assemblers should work too)
6) Compress and insert the assembled data (this is where the compression utility comes in again)


So once I edit the ASM file (still not sure how exactly to go about that), I would use asar to reassemble the file into a ROM? I'd assume, since that's what asar asks for after the asm file... And then after asar patches, do I make a binary file from that ROM, which I would then have the utility compress back into my hack? I'm confused here what the next step is after using asar, in order to continue on to the final step. I'm sure I'm making this more difficult than it needs to be, but it's just not clicking, please forgive my ignorance >_<.

assassin

btw, any idea why the decompression routine starts writing to an initial offset of FFDEh in the temporary output buffer (which spans F800h thru FFFFh)?  i realize this matches up with the max adjusted copy size of 34 bytes, but so what?

i can't figure out why they chose that..  do they have 34 bytes they want to frequently copy without having to perform the FFFFh=>F800h wrap?  that's my best guess.

it probably doesn't matter, but it's something that's puzzled me for awhile.

abw

Checking a trace log for the ending credits, C3D25B does indeed start reading data from C28A70, so overwriting C28A70 would explain why your end credits got messed up. In order to avoid that, you'll want to check that your compressed data doesn't run past C28A6F (or C28A5F if you want to be super safe; I strongly suspect C28A60 - C28A6F is free space, but I'm not actually 100% sure).

The compressed data block starting at (ROM address) 2686C is the same in both the Japanese ROM and the English ROM with Yazoo's title screen patch applied, and that block is smaller than the block in the English ROM, so it will still fit within the amount of space you have in the English ROM if you copy it over (just remember to also copy the other blocks Yazoo's patch modifies).

It sounds like you're basically doing the right thing, although writing the decompressed data back into your Japanese ROM does end up killing your Japanese ROM; I'd write the decompressed data to a new/temporary file instead and do your editing there. As a side note: you know you can redirect a program's (standard, non-error) output to a file using >, right? e.g. perl FFVI_C2FF6D.pl -m d -s 0x2686c "Final Fantasy III (U) (V1.0)smc" > decompressed.bin

However, even if you aren't changing the lengths of any strings in the opening credits, the different text might not compress as well as the original text (e.g. if you change "DIRECTOR", the compression algorithm wouldn't be able to use "DIRECTOR" when it wants to compress "GRAPHIC DIRECTOR"), so you could still end up with compressed data that exceeds the amount of space available in the ROM (hint: the first 2 bytes of the compressed data indicate how long the compressed data is, in little-endian format; the English data takes up 0x21F4 bytes, the Japanese data takes up 0x21E6 bytes, and any data that takes up more than 0x2204 bytes will start overwriting the end credits data starting at (ROM address) 28A70).

If you can steal some space from other strings in the opening credits, you might be able to get away with editing the pointer table (starting at 7E5D47 in English; looks like 7E5D0D in Japanese) and seeing if that helps enough. Otherwise, it's going to take some work.

The bad news is that the ASM file I've uploaded is for the decompressed block from the English ROM, so unfortunately it won't be directly useful for your work with the Japanese ROM. If you want to edit the opening credits this way, you're going to have to disassemble the decompressed Japanese data, identifying which sections are code and which are data and making sure to get the code instruction alignment right, then locate all of the pointers in both parts and convert all the pointers to labels. It's pretty much the exact opposite of fun, but the end result is quite powerful.

On that note, feel free to play around with the ASM file I've uploaded on an English ROM to get a feel for how much easier it is to do things when you have an assembler taking care of basically all of the address/pointer work. For instance, if you want to edit the credits text in the ASM file, all you have to do is change the text to whatever you want (as long as the game can still display it, anyway), and the assembler will work out how long the new text is and update all the pointers (not just the pointers to the text; ALL the pointers in the entire decompressed block) automatically.

Assuming the Japanese code is as wasteful as the English code, you might find a couple of blocks of unused code/data you could completely remove or some sections of code that could be rewritten more efficiently, but even without that, you should be able to rearrange the ASM blocks to keep similar sections near to each other so the compression utility can produce smaller output and allow you to fit in more text that way.

Once you're happy with your edited ASM file, you can use Asar (or whatever) to assemble it into binary; since you need to compress the binary before inserting it back into the ROM, you'll want to have Asar output to a new temporary file (or truncate an old file before running Asar; if you output to an old file but don't actually need as much space as the old file took up, the old file will not shrink to fit your smaller data, and the compressor will consequently generate larger output than you need), then run the compression utility on that file and insert it back into the ROM.



As for why the decompression routine starts writing to its buffer at 0xFFDE, uh, yeah, I have no idea either. For a block of this size, I'm pretty sure it makes absolutely no difference where in the buffer the routine starts writing.

Gi Nattak

#11
Oh joyous day! Seems like simply putting back my data from C28A60 onward did restore all the end credit stuff back to working, and... wait for it... custom intro credits are still all in there and working nicely. :happy:
I'll test around lots more but hopefully that will do it for now.

Thank you for your extremely helpful and knowledgeable assistance, breakdown of the process, and of course this utility and documentation!  :beer:

abw

Awesome! :woot!: Glad to hear that things appear to be going well. Feels pretty great to see those custom credits roll, doesn't it? :beer: