News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: General NES Hacking Questions  (Read 4632 times)

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
General NES Hacking Questions
« on: September 19, 2018, 08:25:57 am »
So I've mostly finished my script revisions to Dragon Warrior 1 and I'm now looking at taking on the sequel. There isn't a dedicated table file posted, but when I plug in the table data from the first game, I'm able to read the text for spells, items and monster names in my hex editor. The curious thing is that the bulk of in game dialogue text does not show up in a readable form.

Glancing over information for later Dragon Warrior games, it looks like text compression exists for the dialogue in the 4th game. Am I most likely facing a compression issue in DW2?

If so, are there any good universal tools that would help me with the process of extracting, decoding, and reinserting the text?
« Last Edit: December 02, 2018, 11:35:30 pm by Chicken Knife »

FCandChill

  • Hero Member
  • *****
  • Posts: 501
    • View Profile
Re: Handling possible text compression
« Reply #1 on: September 19, 2018, 05:43:03 pm »
Chances are, this game uses DTE compression. It's very common in NES games. Can you find data that sort of ressembles dialogue like "S.e.h prince." ("Save the princess" compressed)?

Also, you might want to have a dedicated thread for your Dragon Quest questions ... just a suggestion.

September 19, 2018, 05:43:36 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
As for tools, check here https://www.romhacking.net/?page=utilities&category=14&platform=&game=&author=&os=&level=&perpage=20&title=&desc=&utilsearch=Go

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: Handling possible text compression
« Reply #2 on: September 19, 2018, 07:46:22 pm »
I've used the ROM map available on this page to determine the block of data where the dialogue is stored. As I scan through that data, it does not appear as an abbreviated form of actual text like your example of S.e.h prince. Instead, all the data in that section appears as 80-90 percent non letter entries, with occaisional letter entries interspersed.

As far as your point on forum etiquette, it's a fair one. I know I've been throwing a lot of separate topics onto this board lately. I actually do have a dedicated Dragon Quest thread under personal projects. I figured that thread would attract primarily readers interested in my Dragon Quest project, but posting here would catch a wider audience of people willing to help with the numerous technical obstacles I face as a very new person to this process. Perhaps it would be better to have a unified thread here that isn't really about my project but serves to keep my large mess of questions in one place?

FCandChill

  • Hero Member
  • *****
  • Posts: 501
    • View Profile
Re: Handling possible text compression
« Reply #3 on: September 19, 2018, 08:09:27 pm »
I've used the ROM map available on this page to determine the block of data where the dialogue is stored.

You didn't link to anything in your post, but I assume you meant to link here: https://datacrystal.romhacking.net/wiki/Dragon_Warrior_II:ROM_map

The info you referenced is under the section name Dragon Quest not Warrior. In other words ... the page covers the Japanese version, not the English version. That may be your issue.

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: Handling possible text compression
« Reply #4 on: September 19, 2018, 08:54:53 pm »
Fair point, but I have used a hex editor to carefully scroll through all the data in the rom, several times actually. On each search I come across the full list of items, monsters and spell names. The only other text I come across is the prologue text added for the US version intro. The dialogue text is no where to be found.

When I go through DW1 or even DW3, I easily locate the large block of general dialogue text, along with the items, monster names and spells.

Psyklax

  • Hero Member
  • *****
  • Posts: 781
    • View Profile
    • Psyklax Translations
Re: Handling possible text compression
« Reply #5 on: September 20, 2018, 02:00:55 am »
I second the idea of a dedicated thread for all this: no need to make lots of new threads. :)

As for the text compression question, if I can find the time I can have a look myself. I know for a fact that Dragon Warrior doesn't have compression but I can't remember about DW2. I'd be a bit surprised if it did because neither it nor the first game have an overwhelming amount of text.

Maybe later today I'll be able to spend five minutes checking the text to see how it's stored in the ROM. Compression does happen in text heavy games - my efforts translating Time Stranger from 1986 found a neat dictionary compression scheme - but again, I'd be a little surprised were that the case here.

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: Handling possible text compression
« Reply #6 on: September 20, 2018, 08:58:54 am »
Sounds great Psyklax. Once this issue gets figured out I'll change the title of this post to something generalized and put all my technical questions here going forward.

Going back to the issue at hand, I also found it strange that the game would use compression--especially since there is a ton of empty space in the US rom. I initially thought this might be an issue of the game having the alphabet under multiple table entries. To test that, I went into FCEUX Nametable Viewer and checked the hex code of the letters showing up in dialogue text. The hex code for the letters in dialogue was exactly the same as the code for the letters in monster, item and spell names--with all of the latter showing up in a hex editor when I load up table data. Would this indicate compression or could another factor be causing this? 

I appreciate you taking a look at the rom!

Psyklax

  • Hero Member
  • *****
  • Posts: 781
    • View Profile
    • Psyklax Translations
Re: Handling possible text compression
« Reply #7 on: September 20, 2018, 10:43:22 am »
I went into FCEUX Nametable Viewer and checked the hex code of the letters showing up in dialogue text. The hex code for the letters in dialogue was exactly the same as the code for the letters in monster, item and spell names--with all of the latter showing up in a hex editor when I load up table data. Would this indicate compression or could another factor be causing this?

I don't think simply looking at the nametable would indicate compression per se. The way I would do it is the more advanced method of debugging the game to see how the text gets from the ROM to the screen, and compression will become apparent then. A simpler way is the old fashioned relative search technique locating where the text is, and discovering the compression then.

I'll have a look later, anyway. I've already hacked this game to double experience and gold, so that's why I'm familiar with it.

KingMike

  • Forum Moderator
  • Hero Member
  • *****
  • Posts: 6610
  • *sigh* A changed avatar. Big deal.
    • View Profile
Re: Handling possible text compression
« Reply #8 on: September 20, 2018, 12:50:32 pm »
I know Dragon Warrior II and III used ROM expansion so I wouldn't expect something beyond dictionary, because what's the point once they're expanding the ROM.

Yes, in DW4 they used Huffman which is tricky but that's because they couldn't really expand further. SOROM or whatever it was (the 512KB MMC1 mapper) was already about the limit.
"My watch says 30 chickens" Google, 2018

Psyklax

  • Hero Member
  • *****
  • Posts: 781
    • View Profile
    • Psyklax Translations
Re: Handling possible text compression
« Reply #9 on: September 20, 2018, 06:00:12 pm »
Well, you were right: DW2 uses a rather tricky compression method which I don't fully understand yet. I'm still looking at it, and it's not like what I've seen before. I'd say it's a bit like dictionary compression, except not... I'll continue until I've figured it out, but it's weird.

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: Handling possible text compression
« Reply #10 on: September 20, 2018, 07:40:20 pm »
Thank you so much for diving into this Psyklax. Why did those knuckleheads at Nintendo of America feel the need to give us this headache with all that free space at their disposal.  :banghead:
« Last Edit: September 20, 2018, 07:45:33 pm by Chicken Knife »

Alchemic

  • Jr. Member
  • **
  • Posts: 11
    • View Profile
Re: Handling possible text compression
« Reply #11 on: September 21, 2018, 01:36:17 am »
I did some digging around in DW2 a few years back. My notes are kind of a mess, but to summarize:
  • Text is built up out of pieces that are five or ten bits in length.
  • The underlying characters for each piece are at 0xB49B - 0xB686. Abbreviated TBL below.
  • The piece lengths are at 0xB44B - 0xB49A. One nybble per piece.
  • You can think of the pieces as one table with 160 entries, or five tables with 32 entries each.
  • The 1st table handles bit patterns 00000 through 11011.
  • The 2nd table handles 11100 xxxxx, the 3rd 11101 xxxxx, the 4th 11110 xxxxx, the 5th 11111 xxxxx.
  • The text pointers are at 0xB762 - 0xB7C1.
  • Each pointer points at a blob of 16 concatenated strings.
  • Compressed text is at 0x14010 - 0x17FE6, and a bit more at 0xB7C2 to 0xBE0F.
Code: [Select]
TBL for 0xB49B:
00-09 = 0-9
0A-23 = a-z
24-3D = A-Z
3E-57 = a-z again
58=A
59=(space)
5A=[SUN]
5B=[STAR]
5C=[MOON]
5D=[DROP]
5E=[HEART]
5F=(space)
60=(space)
61='
62="
63=->
64="
65='
66='
67='
68=.'
69=,
6A=-
6B=.
6C=&
6D=(space)
6E=?
6F=!
70=;

Text-dumping code in Python

Psyklax

  • Hero Member
  • *****
  • Posts: 781
    • View Profile
    • Psyklax Translations
Re: Handling possible text compression
« Reply #12 on: September 21, 2018, 03:56:14 am »
Wow, Alchemic, that confirms what I saw: that it's pretty messed up. :D I got as far as noticing that each individual word is assembled letter-by-letter in SRAM before being passed off to the PPU, but it would've taken me time to get to the level of detail you've acquired. That has to be the most convoluted text handling scheme I've ever seen in a game.

I had a look so I can see what you're talking about there. It's still messing with my brain, so I think I'll leave it for now. :)

abw

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: Handling possible text compression
« Reply #13 on: September 21, 2018, 08:57:00 am »
Maybe later today I'll be able to spend five minutes checking the text to see how it's stored in the ROM.
If you actually had cracked that code in 5 minutes, you would have had my vote for the annual "God of ROMHacking" award :P.

I did some digging around in DW2 a few years back. My notes are kind of a mess, but to summarize:
[...]
Text-dumping code in Python
Nice work Alchemic! I'm sure somebody's next question is going to be: did you also happen to write an insertion script?

That has to be the most convoluted text handling scheme I've ever seen in a game.
The basic input -> output encoding is pretty simple, but I definitely agree that DW2's text engine could have been implemented in a much less complicated way. It took me a couple of days to work through it enough to get a decent script dump, but I can confirm that Alchemic's notes are indeed accurate.

I also found it strange that the game would use compression--especially since there is a ton of empty space in the US rom.
Why did those knuckleheads at Nintendo of America feel the need to give us this headache with all that free space at their disposal.  :banghead:
It's not always a question of how much free space in total, but where that free space is located. DW2's script is large enough that even with the compression scheme they added, it still doesn't fit inside a single PRG bank, which is why the script jumps from 0x17FE6 to 0xB7C2. Maybe they were trying to keep almost all of the text-related code and data confined to 2 PRG banks?

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: Handling possible text compression
« Reply #14 on: September 24, 2018, 07:04:44 am »
@abw got it right. I'm definitely asking for someone to help me write a script for this. Wouldn't I need two scripts: one to uncompress the text into readable / editable form and another to recompress for reinsertion? I'm willing to keep the maximum length of each line the same if it means not having to worry about readjusting all the pointers.

In the mean time, I am keeping myself busy figuring out the best way to edit the monster, spell and item lists but I would be so very grateful to have a means of moving forward with the script one day soon.

« Last Edit: September 24, 2018, 08:22:39 am by Chicken Knife »

abw

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: Handling possible text compression
« Reply #15 on: September 24, 2018, 09:57:45 pm »
@abw got it right. I'm definitely asking for someone to help me write a script for this. Wouldn't I need two scripts: one to uncompress the text into readable / editable form and another to recompress for reinsertion? I'm willing to keep the maximum length of each line the same if it means not having to worry about readjusting all the pointers.
If all else fails, abcde can definitely handle both extracting and inserting text like this along with updating the pointers, and you'll have all the original space at your disposal (there's no free space conveniently available after the original script ends, since the very next byte is code, so if you need more space than the original, it'll take a bit more work). As an added bonus, thanks to the original encoding being somewhat inefficient, abcde can actually re-insert the original script using $79 fewer bytes.

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: Handling possible text compression
« Reply #16 on: September 25, 2018, 08:42:31 am »
@abw

Am I to understand that your software can already handle unwinding the existing compression routine in DW2 without the need for a special script being written? Also, if I am to use the more efficient form of compression available through your software, it will automatically rework the code of the game to be able to read the new compression routine?

Something tells me it will be nowhere near this easy.  :laugh:

abw

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: Handling possible text compression
« Reply #17 on: September 25, 2018, 06:10:59 pm »
Am I to understand that your software can already handle unwinding the existing compression routine in DW2 without the need for a special script being written?
[...]
Something tells me it will be nowhere near this easy.  :laugh:
I'm not going to make any promises about how easy using abcde will be for anybody else, but... basically, yeah. Like I said earlier, the text encoding itself is actually pretty simple, it's just that DW2's code for parsing that encoding is fairly unpleasant. Give abcde the right table file, extract command file, and ROM, and watch it go. Stick a couple of extra lines in the script dump file it generates to turn it into a valid insert command file, and then run abcde again with the same table file, the insert command file (edited to have your new text, if so desired), and (a copy of) the ROM. That said, unless you start making ASM changes, you're still constrained by the total available space and text engine and stuff, so it's not a magic bullet.

Also, if I am to use the more efficient form of compression available through your software, it will automatically rework the code of the game to be able to read the new compression routine?
Haha, don't I wish it was that good :P. abcde doesn't make any changes to the game's code, it just uses a better insertion algorithm. The original data compression was sub-optimal; as one example, the game stores the string "'Welcome " as ['W][e][l][come][ ], with those 5 pieces taking up 10 + 5 + 5 + 10 + 5 = 35 bits. On the other hand, abcde is smarter and inserts the same string as ['][Welcome ] in 5 + 10 = 15 bits, for a savings of 20 bits. Little improvements like that that save 5 bits here, 10 bits there over the course of the entire script is how we end up with 121 bytes of reclaimed space.

Chicken Knife

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: General Hacking Questions
« Reply #18 on: November 29, 2018, 08:09:01 am »
I thought I'd bring this thread back to life as I'm back into my projects consistently and have a few things I need help with.

I'm nearly done with my Dragon Warrior 1 hack and my only issue left is that I want to fix some compromises I had to make porting over NPC sprite graphics from the famicom version due to left/right mirroring instructions in the DW1 for certain sprite tiles.

Does anyone have any good reading material to recommend that will present in a way that does a good job explaining itself to a non coder that will help me understand "sprite pointers" (if I'm using the right term) and will help me understand how to either disable the LR mirroring instructions I’m struggling with or simply redirect the game to alternative tiles for that sprite? If I can "repoint" them effectively like I do with text, I will have a ton of extra space since I turned all the 4 direction sprites into 1 direction sprites.

There are also other topics where I would like to do some reading and am having a hard time because I either can't find the resources or the resources I find are too technical:

I encountered a multiplier formula in Dragon Warrior 3 where the game takes the base Japanese gold and experience numbers saved for monsters saved in the IS game but applies a .25% boost, rounded up, to those values for what you actually earn in battle. I want to locate and disable that multiplier, restoring the straight Japanese values in gameplay. I assume a debugger would help me here. If so, any good reading material (or better yet videos) you guys would recommend for effectively using the FCEUX debugger for something like this?

Last, I've tried to play around with Atlas / Cartographer, ABCDE software in hopes of solving the DW2 script compression issue. The faq that comes with Atlas is too technical for me at this point and I haven't found much else out there. Any recommended reading?

Thanks guys!
« Last Edit: November 29, 2018, 08:56:32 am by Chicken Knife »

abw

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: General Hacking Questions
« Reply #19 on: November 29, 2018, 08:34:16 pm »
Does anyone have any good reading material to recommend that will present in a way that does a good job explaining itself to a non coder that will help me understand "sprite pointers" (if I'm using the right term) and will help me understand how to either disable the LR mirroring instructions I’m struggling with or simply redirect the game to alternative tiles for that sprite?
Maybe I shouldn't comment since I don't usually do much graphics work, but when I do, I usually end up consulting Y0SHi's NES Documentation, and although I'm pretty sure I originally had to read it a few times before it sunk in, it hasn't steered me wrong yet and it does have a couple of sections that talk about sprites.

Last, I've tried to play around with Atlas / Cartographer, ABCDE software in hopes of solving the DW2 script compression issue. The faq that comes with Atlas is too technical for me at this point and I haven't found much else out there. Any recommended reading?
Did you try abcde's readme or examples? Be honest - how bad are they really? I'm waaaay too deeply immersed in the source material to feel confident about my ability to pitch the documentation at a level that is useful without making other people barf, plus I implicitly assume that the end user is already familiar with Atlas/Cartographer :(.

I might have some recommended reading for you, though (based on abcde v0.0.2; provide your own "Dragon Warrior II (U) [!].nes") ;D:
Extraction:
Quote from: Cartographer.txt
#GAME NAME:      Dragon Warrior II (U) [!].nes

#BLOCK NAME:      Main Script, Part 1
#TYPE:         NORMAL
#METHOD:      POINTER_RELATIVE
#POINTER ENDIAN:   LITTLE
#POINTER TABLE START:   $B762
#POINTER TABLE STOP:   $B7BD
#POINTER SIZE:      $02
#POINTER SPACE:      $00
#STRINGS PER POINTER:   16
#AUTO JUMP START:   $17FE7
#AUTO JUMP STOP:   $B7C2
#ATLAS PTRS:      Yes
#BASE POINTER:      $C010
#TABLE:         dw2_script.tbl
#COMMENTS:      Yes
#END BLOCK

#BLOCK NAME:      Main Script, Part 2
#TYPE:         NORMAL
#METHOD:      POINTER_RELATIVE
#POINTER ENDIAN:   LITTLE
#POINTER TABLE START:   $B7BE
#POINTER TABLE STOP:   $B7C1
#POINTER SIZE:      $02
#POINTER SPACE:      $00
#STRINGS PER POINTER:   16
#ATLAS PTRS:      Yes
#BASE POINTER:      $10
#TABLE:         dw2_script.tbl
#COMMENTS:      Yes
#END BLOCK

Quote from: dw2_script.tbl
# NB: I didn't put much time into figuring out the control codes, so some of them are still unknown and some of them might be wrong. Caveat emptor!

/%00000=[end]\n\n
/%00001=.[end]\n\n
/%00010=?[’ ][FD][FD][end]\n\n
/%00011=[.’][end]\n\n
%00100=[FF]
%00101=y
%00110=c
%00111=o
%01000=d
%01001=e
%01010=f
%01011=g
%01100=h
%01101=i
%01110=j
%01111=
%10000=l
%10001=m
%10010=n
%10011=[line]\n
%10100=[.’]
%10101=[ ‘]
%10110=r
%10111=s
%11000=t
%11001=u
%11010=a
%11011=w
%11100=C0
%11101=C1
%11110=C2
%11111=C3

%1110000000=A
%1110000001=B
%1110000010=Ca
%1110000011=D
%1110000100=E
%1110000101=F
%1110000110=G
%1110000111=H
%1110001000=I
%1110001001=J
%1110001010=King
%1110001011=L
%1110001100=Moonbrooke
%1110001101=N
%1110001110=O
%1110001111=[item]
%1110010000=The
%1110010001=Rhone
%1110010010=S
%1110010011=;
%1110010100=U
%1110010101=”
%1110010110=Water Flying Cl
%1110010111=C
%1110011000=Y
%1110011001=Z
%1110011010=x
%1110011011=Village
%1110011100=z
%1110011101=[F9]
%1110011110=‟
%1110011111=K

%1110100000=v
%1110100001=q
%1110100010=[’ ][wait][line]\n
%1110100011=R
%1110100100=.
%1110100101=[FD][FD]
%1110100110=P
%1110100111=b
%1110101000=T
%1110101001=!
%1110101010=[sun]
%1110101011=[star]
%1110101100=[moon]
%1110101101=W
%1110101110=k
%1110101111=p
%1110110000=?
%1110110001=,
%1110110010=[monster]
%1110110011=....
%1110110100=:
%1110110101=[’ ]
%1110110110=-
%1110110111=[’ ]
%1110111000=[spell]
%1110111001=[letter]
%1110111010=[no voice]
%1110111011=[wait]
%1110111100=M
%1110111101=[name]
%1110111110=[number]
%1110111111=[FD]

%1111000000=Thou hast
%1111000001=hest
%1111000010=Midenhall
%1111000011=hou
%1111000100= of
%1111000101= is
%1111000110= thou has
%1111000111= and
%1111001000=to th
%1111001001= thee
%1111001010=ast
%1111001011= do
%1111001100=hat
%1111001101= shall
%1111001110= was
%1111001111=hou has
%1111010000=d the
%1111010001= has
%1111010010=gon
%1111010011=.[wait][line]\n
%1111010100= have
%1111010101=come to
%1111010110=ing
%1111010111= hast
%1111011000=ost thou
%1111011001=this
%1111011010= of the
%1111011011=Hargon
%1111011100=in the
%1111011101=thing
%1111011110=he
%1111011111= with

%1111100000=reasure
%1111100001=[ ‘]Hast
%1111100010=Erdrick
%1111100011=come
%1111100100=ere is
%1111100101=Welcome
%1111100110=rince
%1111100111= great
%1111101000=arr
%1111101001= for th
%1111101010=piece[(s)] of gold
%1111101011=[.’][wait][line]\n
%1111101100=But
%1111101101=here
%1111101110=can
%1111101111=ove
%1111110000=hee
%1111110001=not
%1111110010=for
%1111110011=one
%1111110100= any
%1111110101= to
%1111110110=descendant
%1111110111=Roge Fastfinger
%1111111000=all
%1111111001=thy
%1111111010=[ ‘]W
%1111111011=thank thee
%1111111100= it
%1111111101= tha
%1111111110= thou
%1111111111= the

Usage (replace \path\to\abcde\abcde.pl as appropriate):
Code: [Select]
perl \path\to\abcde\abcde.pl -m bin2text -cm abcde::Cartographer "Dragon Warrior II (U) [!].nes" Cartographer.txt Cartographer_out -s

Insertion:
Quote from: Atlas.txt
// Define, load, and activate a TABLE
#VAR(Table, TABLE)
#ADDTBL("dw2_script.tbl", Table)
#ACTIVETBL(Table)

// Jump to start of script
#JMP($14010)
#HDR($C010)
// auto-commands for when DW2 does a mid-string bankswap and resets its read address:
#AUTOCMD($17FE7, #HDR($10))
#AUTOCMD($17FE7, #JMP($B7C2, $BE0F))

// the rest of Cartographer_out.txt goes here

Quote from: insert.bat
@copy /Y "Dragon Warrior II (U) [!].nes" "Dragon Warrior II (U) [new].nes" > nul

perl \path\to\abcde\abcde.pl -m text2bin -cm abcde::Atlas "Dragon Warrior II (U) [new].nes" Atlas.txt

pause