News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: General NES Hacking Questions  (Read 13451 times)

Choppasmith

  • Jr. Member
  • **
  • Posts: 94
    • View Profile
Re: General NES Hacking Questions
« Reply #60 on: March 10, 2019, 05:24:29 pm »
Nice!
Remember how we expanded the DW1 ROM by adding 4 new banks just before the final fixed bank? Enix did basically the same thing (except in hardware) when porting the game to English; they had to add 8 banks, but didn't end up using 3 of those.

In bank 5, you've got $17FE7 - $14010 = $3FD7 bytes, and bank 12 ($0C) gives you another $4000 bytes for a total of $7FD7 = 32,727 bytes, which should leave you with about 350 bytes to spare if you managed to get your script down to 32,378 bytes. Is that 32,378 bytes the script size after being converted to DW2's 5/10-bit encoding, or is that the file size when encoded as UTF-8?

Well, as said a few posts ago, I split my script into two parts to determine how much of it would fit. I took the bytes remaining as reported by abcde and subtracted from the total space available and did it again for the second part then combined the values which is how I got the 32,378 bytes. I had done the same thing before and I was getting over 34,000 bytes, so even if my math is somehow off it's still a significant decrease.

Nice!
Remember how we expanded the DW1 ROM by adding 4 new banks just before the final fixed bank? Enix did basically the same thing (except in hardware) when porting the game to English; they had to add 8 banks, but didn't end up using 3 of those.

In bank 5, you've got $17FE7 - $14010 = $3FD7 bytes, and bank 12 ($0C) gives you another $4000 bytes for a total of $7FD7 = 32,727 bytes, which should leave you with about 350 bytes to spare if you managed to get your script down to 32,378 bytes. Is that 32,378 bytes the script size after being converted to DW2's 5/10-bit encoding, or is that the file size when encoded as UTF-8?

If you're still having space issues, don't forget about the single-character dictionary entries - if you're e.g. using "p" more than "w", you could kick "w" into the 10-bit range and move "p" into the 5-bit range to get better compression. For English text, you can probably also replace the "q" entry with "qu".
Yeah that could work. I would assume, in layman's terms, that the first set of entries use less data than the 4 "C" tables? Or does each table use extra bits of data the further you go down? (Edit : forget that I'm dumb)I honestly still don't quite understand how those work, I initially thought it was like a special table where'd you use two bytes to load a dictionary value, and pretty much every word/letter from the dictionary wound be preceded by the corresponding value and then stuff from the first table fills in the rest with no dictionary value needed.

Edit : Just read the Wiki, I guess it works like I said above but in bits, not bytes?
« Last Edit: March 10, 2019, 05:41:47 pm by Choppasmith »

abw

  • Full Member
  • ***
  • Posts: 218
    • View Profile
Re: General NES Hacking Questions
« Reply #61 on: March 10, 2019, 06:12:36 pm »
Yeah that could work. I would assume, in layman's terms, that the first set of entries use less data than the 4 "C" tables? Or does each table use extra bits of data the further you go down? (Edit : forget that I'm dumb)I honestly still don't quite understand how those work, I initially thought it was like a special table where'd you use two bytes to load a dictionary value, and pretty much every word/letter from the dictionary wound be preceded by the corresponding value and then stuff from the first table fills in the rest with no dictionary value needed.

Edit : Just read the Wiki, I guess it works like I said above but in bits, not bytes?
Basically, yeah. I had this typed up before I saw your edit, so I'll post it anyway just in case it's still helpful:
--
Sort of, except as far as the game's concerned, the individual tokens are 5 bits each, not 8, and 4 of the entries from the first 5-bit table are used to switch to the corresponding "C" table. You could split that single table file into multiple table files if you wanted to; I just prefer having all the entries in a single table.

You can tell how much space each entry takes up by looking at its left-hand side. In a hexadecimal table, "80" takes up one byte (or 2 nybbles or 8 bits, depending on how you want to think of it), "80FF" takes up two bytes (/4 nybbles/16 bits), and so on; in a binary table, "11011" takes up 5 bits, "1110101111" takes up 10 bits, and so on. This will be an even bigger issue if you decide to tackle DW4, since it uses a Huffman encoding where the binary representation of individual characters ranges between 3 bits (e.g. for "e") and 18 bits (e.g. for "8").

Choppasmith

  • Jr. Member
  • **
  • Posts: 94
    • View Profile
Re: General NES Hacking Questions
« Reply #62 on: March 11, 2019, 03:18:44 pm »
Basically, yeah. I had this typed up before I saw your edit, so I'll post it anyway just in case it's still helpful:
--
Sort of, except as far as the game's concerned, the individual tokens are 5 bits each, not 8, and 4 of the entries from the first 5-bit table are used to switch to the corresponding "C" table. You could split that single table file into multiple table files if you wanted to; I just prefer having all the entries in a single table.

You can tell how much space each entry takes up by looking at its left-hand side. In a hexadecimal table, "80" takes up one byte (or 2 nybbles or 8 bits, depending on how you want to think of it), "80FF" takes up two bytes (/4 nybbles/16 bits), and so on; in a binary table, "11011" takes up 5 bits, "1110101111" takes up 10 bits, and so on. This will be an even bigger issue if you decide to tackle DW4, since it uses a Huffman encoding where the binary representation of individual characters ranges between 3 bits (e.g. for "e") and 18 bits (e.g. for "8").

Thanks, once I saw that the programmer's calculator had a binary converter, it just clicked.

Anyway, just from doing the above suggestion I'm down another 400 bytes! And just after some needed editing of the intro I'm now down to 760 bytes over the limit. I was really loose and liberal with my initial script (like gratuitous line breaks, pauses, and spaces), so it should be easy to trim the fat without sacrificing much. Although when I tried dumping with the new second address, I'd get horrible garbage dialog. The intro is only the 8th pointer in bank 5, so I'm not too sure what happened there. I know I have to update the ROM with the new second bank, but I'm not sure why dialog in the beginning wouldn't work properly. Did I miss something (other than updating the second bank?)

A couple other weird issues

Still having trouble with closing single quotes for some reason.



I double checked the ROM and I do indeed have $66 in the dictionary. I also have an entry with "!’" but that has Apostrophes ($67) work fine though

Also



Not sure what's going on here. There's a couple of windows like this with the oddly premature uncalled for line break. I'm assuming the window counts how many bits are used per line as opposed to individual letters/spaces/characters? Any suggestions? Or is it just a weird game programming/abcde limitation?

abw

  • Full Member
  • ***
  • Posts: 218
    • View Profile
Re: General NES Hacking Questions
« Reply #63 on: March 11, 2019, 08:03:16 pm »
Hurray! The proper choice of dictionary entries makes a huge difference to the compression ratio. Just make sure you leave the first 5 end tokens and the 4 "C" table switches alone (since the game cares about those) and fill out the rest of the main 5-bit table with your 23 best tokens.

As for your issues, it's kind of hard to say for sure without seeing the files. If dumping the inserted data didn't work but the game still displays it correctly, then something is wrong with your dump script. For the closing single quotes, my next guess would be an error in the dictionary lengths, though a length error would also mess up all the dictionary entries following the one with the wrong length. The random line breaks is definitely strange too, and not an issue I've come across myself; I've definitely inserted longer strings than that without getting inappropriate line breaks. I haven't looked at the code for it, but just based on observation, DW2's line wrapping algorithm seems pretty solid. Any chance it always happens around the words "What" or "is"? If so, that could indicate that you've got a line break instead of a space in one of the dictionary entries - just a guess!

Choppasmith

  • Jr. Member
  • **
  • Posts: 94
    • View Profile
Re: General NES Hacking Questions
« Reply #64 on: March 16, 2019, 06:27:57 pm »
I got it! I had used Pointer Tables to extract, edit, and insert the dictionary entries. Yeah, I know it's clunky, but easy for me. It was changing blank spaces (5F) to [no voice] (59). Once I changed it in the ROM it was fine. Similar problem with the closing single quotes and I fixed that too. I was hoping that this could've shrunk the script size, but, alas, it didn't. Oh well, more editing work for me!

Congrats on your Latin translation release, abw! I had looked at the Read Me and you said one of your improvements was editing the main party status windows to show (was this battle or field or both?) full character names instead of just showing the first four letters.

When working on DW1 menus, the thought of doing that for later entries struck me as a "would be nice if possible". You mentioned it was more than a one byte hack. Were there other entries for the window you had to change besides length? Or is it like monster/item/spell/etc names where you had to find that value that affected display length? Btw, I know it'd be easy to tell the magic byte that affects monster name length like DW1, but you did explain how you did it and I have every intention to try myself as part of my learning.

abw

  • Full Member
  • ***
  • Posts: 218
    • View Profile
Re: General NES Hacking Questions
« Reply #65 on: March 17, 2019, 05:53:07 pm »
I got it!
Congrats!

Congrats on your Latin translation release, abw! I had looked at the Read Me and you said one of your improvements was editing the main party status windows to show (was this battle or field or both?) full character names instead of just showing the first four letters.

When working on DW1 menus, the thought of doing that for later entries struck me as a "would be nice if possible". You mentioned it was more than a one byte hack. Were there other entries for the window you had to change besides length? Or is it like monster/item/spell/etc names where you had to find that value that affected display length?
Thanks! If you check the screenshots, you'll see that full character names are displayed both in and out of battle; I think I made changes for displaying full names in a total of 37 different menus, which typically also involved widening and repositioning them and sometimes also involved making updates to the menu wiping process; the WEAPON/ARMOR/SHILD/HELMET menus were particularly irritating to deal with in that respect. For the code changes, the basic problem is that DW2 stores the first 4 characters of each hero's name in one spot and the last 4 characters in a completely different spot (probably due to the English version's extra 4 characters being bolted on to the original 4 characters present in the Japanese version), so you can't just find the code that says "read 4 bytes from X" and update one byte to say "read 8 bytes from X", you need to add more code (which needs more space) to read 4 bytes from X and glue them together with 4 bytes from Y.

For the later games in the series, the mini status menu was rearranged into 1 column per hero instead of 1 row per hero, so widening the columns probably wouldn't work out very well due to lack of screen space if everybody has a long name, but you could switch the layout back to rows if you wanted. It's just a question of how much work would be involved.

Btw, I know it'd be easy to tell the magic byte that affects monster name length like DW1, but you did explain how you did it and I have every intention to try myself as part of my learning.
Trace logger is your friend there too!