News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: FF1 MMC5 Disassembly Updates  (Read 180626 times)

Jiggers

  • Sr. Member
  • ****
  • Posts: 373
    • View Profile
    • My Ko-Fi Page
Re: FF1 MMC5 Disassembly Updates
« Reply #580 on: May 24, 2020, 02:01:48 am »
I was having a look at this: https://github.com/sobodash/graveyardduck/blob/master/graveduck.py and wondering if I would be able to figure out enough to edit it for the RLE format FF1 uses... but haven't been up to really going over it yet. I do have Python installed, and that was such a nightmare for me I haven't touched it since then. Even the most basic tutorials seem to assume you know what they're talking about. I need really specific, detailed instructions on every single step.

I don't want to have to read the ROM, for sure. I think the way I set up map loading has each bank's pointer table in that bank... so it would get in the way and mess up the data... at the moment. Though having a way to re-compress the maps at assembly time would mean going back to the old method would make more sense.

Right now I have all the maps as compressed .bin files...

I'd like it if they were to remain separate files, since FFHackster remains the only good map editing program. Being able to import the uncompressed maps into it for editing, then save them and compress them during assembly would be fantastic!
I know exactly what I'm doing. I just don't know what effect it's going to have.

I wrote some NES music! Its a legal ROM file. - I got a Ko-Fi page too.

Vanya

  • Hero Member
  • *****
  • Posts: 1671
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #581 on: May 24, 2020, 12:58:47 pm »
Well... I can read the python code pretty easily and for the most part it makes sense to me.
There's just a few function labels I don't recognize.
It's seems very reminiscent of the QuickBASIC and C++ stuff I learned in college and the GML script used in GameMaker.

So maybe I can give it a go.
What do I need to install to get going?

Disch

  • Hero Member
  • *****
  • Posts: 2814
  • NES Junkie
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #582 on: May 24, 2020, 04:17:13 pm »
https://www.python.org/downloads/  <-  This is the install page.  IIRC the Windows install is incredibly painless -- just a normal install wizard.  I don't know why Jiggers had a hard time.   :'(

Once you have it installed, you can just write up scripts in your text editor of choice.  Run the scripts via commandline via:
Code: [Select]
> python your_script.py

FWIW I wrote a decompressor earlier today that pulls the maps from the ROM.  It assumes the ROM is named "ff.nes" and is in the current directory, then dumps a bunch of bin files in a 'maps' directory:
Code: [Select]

import os

NumberOfStandardMaps = 61


def decompress( src ):
    output = bytearray()
    srclen = len(src) - 1
   
    i = 0
    while i < srclen:
        v = src[i]
        i += 1
        if v == 0xFF:           # end of map data?
            break
           
        if v & 0x80:          # run?
            v = v & 0x7F
            runlen = src[i]
            i += 1
           
            if runlen == 0:
                runlen = 256
            for _ in range(runlen):
                output.append(v)
        else:
            output.append(v)
           
    return output




if __name__ == '__main__':
    # make a 'maps' director
    if not os.path.exists('maps'):
        os.mkdir('maps')
   
    infile = open('ff.nes', 'rb').read()

    # standard maps!
    for mapId in range(NumberOfStandardMaps):
        ptrOffset = 0x10010 + (mapId * 2)
        ptr = infile[ptrOffset] | (infile[ptrOffset + 1] << 8)
        ptr += 0x10010
       
        open('maps/%02d.bin' % mapId, 'wb').write( decompress( infile[ptr:] ) )
       
    # overworld map!
    ow = bytearray()
    for row in range(0x100):
        ptrOffset = 0x4010 + (row * 2)
        ptr = infile[ptrOffset] | (infile[ptrOffset + 1] << 8)
        ptr &= 0x3FFF
        ptr += 0x4010
       
        ow += decompress( infile[ptr:] )
       
    open('maps/ow.bin', 'wb').write( ow )

I started work on a compressor, but to help Jiggers out I was putting in HEAVY comments so I didn't finish it.



EDIT:

Here's most of the compressor.  This is untested because I didn't finish it, but hopefully you get the jist.


Code: [Select]
import os

# define a function which outputs the compressed version of a run of a given length
def outputRun( out, runByte, runLen ):
    # do this in a loop.  If there are runs longer than 256 tiles, we have to
    #   compress it in "chunks" that are no bigger than 256.  The while loop here
    #   lets us repeat this "chunking" process as many times as is needed
    while runLen > 0:
        if runLen == 1:         # was it a 'run' of only 1 byte?
            out.append(runByte) # just output it normally  (no compression)
            runLen = 0
        elif runLen == 2:       # was it 2 bytes?
            out.append(runByte) # just output it twice, compressing won't save any space
            out.append(runByte)
            runLen = 0
        elif runLen >= 256:     # Run of 256 or more, we can only do 256
            out.append(runByte | 0x80)
            out.append(0)
            runLen -= 256
        else:                   # else, it's between 3-255... compress it normally!
            out.append(runByte | 0x80)
            out.append(runLen)
            runLen = 0

# define a function to compress some data
#  'src' technically be a string, but we will be treating it as a list of bytes
#  we will be compressing the input bytes and returning a bytearray of the compressed data
def compress(src):
    out = bytearray()       # create our output array
   
    runByte = 0             # a var to keep track of the current 'run' byte
    runLen = 0              # how long the run is
   
    for val in src:         # loop over each byte in our input string
        if val == runByte:          # is this part of the run?
            runLen += 1             # if yes, tally it
        else:
            # otherwise, we're ending a run and starting another.
            outputRun( out, runByte, runLen )   # before starting this new run, output any previous run
                   
            #  Next, we need to start a new run
            if val == 0x7F:     # can't start a run with tile 7F, because that would erroneously create an $FF byte
                # so just output this 7F tile immediately and don't start a run
                out.append(0x7F)
                runByte = 0
                runLen = 0
            else:               # any other tile can have a run
                runByte = val
                runLen = 1

    # Now that we've gone through the whole map, output any run we haven't output yet
    outputRun( out, runByte, runLen )
    out.append(0xFF)                # add the terminator
    return out
   
   
# Another function!  One specifically to handle overworld maps, since those have
#  the additional hassle of a per-row pointer table
def overworld(src):
    ptrTable = bytearray()          # will hold the pointers
    rowGlob = bytearray()           # will hold the compressed rows

    compressedRows = {}             # a dictionary to hold the compressed rows.
                                    #  Key = compressed row, value = index in pointer table
                                    # This allows us to reuse mulitple rows if they are identical
                                   
    for rowIndex in range(0x100):
        row = compress( src[ (rowIndex * 0x100):0x100 ] )
       
        if row in compressedRows:
            x = compressedRows[row]
            ptrTable[(rowIndex * 2)    ] = ptrTable[(x * 2)    ]
            ptrTable[(rowIndex * 2) + 1] = ptrTable[(x * 2) + 1]
        else:
            compressedRows[row] = rowIndex
            ptr = ((len(rowGlob) + 0x200) & 0x3FFF) + 0x8010
            ptrTable.append( ptr & 0xFF )
            ptrTable.append( (ptr >> 8) & 0xFF )
            rowGlob += row
   
    return ptrTable + rowGlob
   
if __name__ == '__main__':
    #  todo -- read a file, see if it's overworld or not by looking at the size, and compress it!
« Last Edit: May 24, 2020, 05:45:01 pm by Disch »

Vanya

  • Hero Member
  • *****
  • Posts: 1671
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #583 on: May 25, 2020, 11:16:41 am »
Thanks, Disch.
I'll try it out and see what I can do.

One more thing.
How much of a hassle would it be to make these into a simple windows program with a basic UI?

Disch

  • Hero Member
  • *****
  • Posts: 2814
  • NES Junkie
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #584 on: May 25, 2020, 11:32:21 am »
One more thing.
How much of a hassle would it be to make these into a simple windows program with a basic UI?

I have no experience with GUI libs for Python so I couldn't say.  I pretty much just use it for quick one-off scripty programs, and for that I'm usually happy with a CMD interface.

But... for this I don't think you'd really need it anyway.  The way I see it, in this use case, the decompressor is only ever run once so it can be a little janky and hard to use.  And you WANT a CMD interface for the compressor so it can be put in the build script.

Vanya

  • Hero Member
  • *****
  • Posts: 1671
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #585 on: May 25, 2020, 12:36:24 pm »
OK Thanks again!

Jiggers

  • Sr. Member
  • ****
  • Posts: 373
    • View Profile
    • My Ko-Fi Page
Re: FF1 MMC5 Disassembly Updates
« Reply #586 on: May 25, 2020, 03:29:15 pm »
@Disch: Wow... thanks for all this!

@Vanya: Glad you're going to tackle this too. XD It'd take me a month to really start figuring it out. Summer makes my brain swampy.

I hope the re-compression at buildtime thing will be able to make the pointers work the way the original game does it? Then have actual map data start $100 bytes into the first map bank. Right now, with #60 maps, pointers take up $78 bytes of space. I'd like to make that be future-proof for doubling of the total map amount. Banks 16, 17, 18, 19, 1A, and 1B, maybe also 1C. And 1D for a second overworld...

So map data should start at $00058110 in the ROM.

Right now, it starts at $00058040, and if this thing is meant for vanilla FF1, decompressing will screw up starting at Coneria Castle 2F, because of the pointers at the tart of Bank 17. Not to mention the 0s at the end of Bank 16 and 17, and the pointers at the start of bank 18.

For now, keep it working with the original ROM to get all the map data. I'll decompress my 2 changed maps with Hackster,  and add them in. And when the compression-at-build works, we can tweak the decompression to work on the new ROM! (And keep both versions so people can revert to original maps if they want to.)

And it will be easy enough to make these python scripts be .exe files, right? I wouldn't want to make people install Python if they don't need it for anything else. Options, options!

Quote
I don't know why Jiggers had a hard time.

Installing Python itself might have been fine. I think it was all the OTHER things I had to download and figure out how to install with it, in order to install Pently (a NES sound driver I was curious about using.) One problem with Python was that it wasn't letting me run things from any directory in CMD.exe and I had to manually add some folder settings somewhere...? Windows 10 Home. So all these other things would say "just type in --py install XYZ" and I'm like, where do I type that in? Command line? Do I open Python first? Does Python have its own command line? What folder do I have to be in? Does the install package have to be in the Python folder? Why can't I just unzip this where I want it to go?!?



Bug fix update last night, and a minimap update today!

https://www.youtube.com/watch?v=YqGnFu1EJbo

Update May 27: Snipped things around even MORE and managed to get the zoomed out overworld decompressed while its drawing the zoomed in one, meaning that 4 second pause before the text shows up is gone. Wooohoo!

Weird bug: Everyone upstairs in Castle Coneria vanishes when you use the minimap...? Edit: Ah. That's all maps. Fixing it.
« Last Edit: May 27, 2020, 10:12:49 pm by Jiggers »
I know exactly what I'm doing. I just don't know what effect it's going to have.

I wrote some NES music! Its a legal ROM file. - I got a Ko-Fi page too.

Vanya

  • Hero Member
  • *****
  • Posts: 1671
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #587 on: May 28, 2020, 09:22:50 am »
Nice!
I noticed that the city maps were off center.
That's because of their position in the map data, right?
Are you going to try to make the map routine center it or leave it to users to properly center their maps within their given spaces?

Jiggers

  • Sr. Member
  • ****
  • Posts: 373
    • View Profile
    • My Ko-Fi Page
Re: FF1 MMC5 Disassembly Updates
« Reply #588 on: May 28, 2020, 01:58:58 pm »
That's why I want the maps decompressed!; the easiest way to center them is to use YY-CHR to move the bytes around, then re-compress, and re-do the NPC and teleport coordinates. Every map is different, so nothing I can code will work for all. The maps themselves need to be adjusted.

And a 4x4 tile area somewhere in each map has to have tiles the same pixel color as the background, as that tile will fill the left and right spaces. There's a table to cheese each map's fill-tile ID, that doesn't have to exist if each map had the same 4x4 bottom right corner.





So what did I do wrong...

I copied the decompress script to notepad, saved it as "map.py", put the vanilla ROM in the folder, named it ff.nes, went to the folder in the command line, and typed in "python map.py"... And it did nothing.  :huh:


IT PUT THE MAPS FOLDER IN THE PYTHON DIRECTORY! IT WORKED!! I USED PYTHON!!!!!
« Last Edit: May 28, 2020, 04:56:39 pm by Jiggers »
I know exactly what I'm doing. I just don't know what effect it's going to have.

I wrote some NES music! Its a legal ROM file. - I got a Ko-Fi page too.

Disch

  • Hero Member
  • *****
  • Posts: 2814
  • NES Junkie
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #589 on: May 28, 2020, 08:35:08 pm »
IT PUT THE MAPS FOLDER IN THE PYTHON DIRECTORY! IT WORKED!! I USED PYTHON!!!!!

Congrats!!!   :D  Though I don't know why it put it in the Python install directory, that's weird.

I might be able to spend some time to finish the recompressor.  I think that'd be a good thing to find some time for.


EDIT:

According to this:  https://www.pitt.edu/~naraehan/python3/file_path_cwd.html

Looks like Python puts the CWD in the install directory if you're running from a python shell.  Which you don't seem to be doing -- you're running from windows CMD and just launching the script normally.  So I am confused why it isn't using the script directory as the CWD.

I would dismiss this as "oh well you got it working so who cares", but this seems like a problem that will come back to bite you later.  Very puzzling
« Last Edit: May 28, 2020, 08:46:13 pm by Disch »

Jiggers

  • Sr. Member
  • ****
  • Posts: 373
    • View Profile
    • My Ko-Fi Page
Re: FF1 MMC5 Disassembly Updates
« Reply #590 on: May 28, 2020, 09:25:50 pm »
I did eventually run it from within the Python directory, as part of my "try different things to see what works" method... I didn't notice the map folder then though, was expecting some notification of success in the command window... Wasn't until I was going to zip up both to send to someone else to try that I saw it there. I'll try to run it from the desktop and see if it happens again.

Okay, it only maps the maps folder inside the Python folder when the script and rom are in it, AND I run it from the command line in the same folder. This might be part of why I kept having problems trying to install things (minGW) before...

Thank you again! Also gotta figure out how to set the recompressor up with the build.bat when its done. Probably be a week or two before I'm done with shifting the maps around.
I know exactly what I'm doing. I just don't know what effect it's going to have.

I wrote some NES music! Its a legal ROM file. - I got a Ko-Fi page too.

Disch

  • Hero Member
  • *****
  • Posts: 2814
  • NES Junkie
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #591 on: May 28, 2020, 09:48:46 pm »
FWIW Jiggers

A program to shift the maps for you might be a good first python project  ;)

A simple interface like:
Code: [Select]
python yourscriptname.py mapfile.bin 5
To shift the map right by 5 bytes/tiles.

This wouldn't be too hard to do programmatically -- you'd just read the file in 64-byte blocks and write the same block back just in a fragmented way:

Code: [Select]
shift_amt = __from_commandline__

row = file.read(64)         # read 64 bytes

shifted_row = row[shift_amt:]    # take the row starting from index 'shift_amt' and extending to end of row
shifted_row += row[0:shift_amt]  # append to that the first 'shift_amt' bytes in the row

# shifted_row is now your shifted row!  repeat for all rows and write them back to file and whatnot

Vanya

  • Hero Member
  • *****
  • Posts: 1671
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #592 on: May 29, 2020, 10:22:37 am »
I'm glad this is working out.
If  you end up needing to manually adjust the maps, don't hesitate to send at least some to me so I can help get some of the grunt work done.

Jiggers

  • Sr. Member
  • ****
  • Posts: 373
    • View Profile
    • My Ko-Fi Page
Re: FF1 MMC5 Disassembly Updates
« Reply #593 on: May 29, 2020, 02:12:37 pm »
Tiled has an offset feature that I think will work best for shifting the maps. Lets me visualize exactly where everything is going to be.

The problem now is that I have no idea how to import the map data into it.

https://pastebin.com/PrffYCyd

Its using decimal instead of hex, the only bulk hex-to-decimal converter I can find that works is https://md5decrypt.net/en/Conversion-tools/#results this, and it replaces all the commas and puts it on one line, which makes it a nightmare to fix up, because then Tiled complains that the data is corrupt if its not spaced out JUST SO.  :banghead:

Possible to make a python script to convert the .bin files in the map folder into Tiled files, and back?

It might just be faster to use YY-CHR as I was planning on--I can't really see what I'm doing, but it cuts out so many steps that I could just build a new rom and load a save state to check out how it went--but I can't see if its going to work until the re-compressor works, and its just so damn hot I can't focus on figuring anything out right now.
I know exactly what I'm doing. I just don't know what effect it's going to have.

I wrote some NES music! Its a legal ROM file. - I got a Ko-Fi page too.

Vanya

  • Hero Member
  • *****
  • Posts: 1671
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #594 on: May 30, 2020, 03:01:32 pm »
Tell me about it!
It's not even summer yet!
That's what I get for moving closer to the equator.

Disch

  • Hero Member
  • *****
  • Posts: 2814
  • NES Junkie
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #595 on: May 30, 2020, 06:39:23 pm »
OK I think the recompressor is done.

Here's the code:

Code: [Select]
import os
import sys
import glob

# define a function which outputs the compressed version of a run of a given length
def outputRun( out, runByte, runLen ):
    # do this in a loop.  If there are runs longer than 256 tiles, we have to
    #   compress it in "chunks" that are no bigger than 256.  The while loop here
    #   lets us repeat this "chunking" process as many times as is needed
    while runLen > 0:
        if runLen == 1:         # was it a 'run' of only 1 byte?
            out.append(runByte) # just output it normally  (no compression)
            runLen = 0
        elif runLen == 2:       # was it 2 bytes?
            out.append(runByte) # just output it twice, compressing won't save any space
            out.append(runByte)
            runLen = 0
        elif runLen >= 256:     # Run of 256 or more, we can only do 256
            out.append(runByte | 0x80)
            out.append(0)
            runLen -= 256
        else:                   # else, it's between 3-255... compress it normally!
            out.append(runByte | 0x80)
            out.append(runLen)
            runLen = 0

# define a function to compress some data
#  'src' technically be a string, but we will be treating it as a list of bytes
#  we will be compressing the input bytes and returning a bytearray of the compressed data
def compress(src):
    out = bytearray()       # create our output array
   
    runByte = 0             # a var to keep track of the current 'run' byte
    runLen = 0              # how long the run is
   
    for val in src:         # loop over each byte in our input string
        if val == runByte:          # is this part of the run?
            runLen += 1             # if yes, tally it
        else:
            # otherwise, we're ending a run and starting another.
            outputRun( out, runByte, runLen )   # before starting this new run, output any previous run
                   
            #  Next, we need to start a new run
            if val == 0x7F:     # can't start a run with tile 7F, because that would erroneously create an $FF byte
                # so just output this 7F tile immediately and don't start a run
                out.append(0x7F)
                runByte = 0
                runLen = 0
            else:               # any other tile can have a run
                runByte = val
                runLen = 1

    # Now that we've gone through the whole map, output any run we haven't output yet
    outputRun( out, runByte, runLen )
    out.append(0xFF)                # add the terminator
    return out
   
   
# Another function!  One specifically to handle overworld maps, since those have
#  the additional hassle of a per-row pointer table
def overworld(src):
    ptrTable = bytearray()          # will hold the pointers
    rowGlob = bytearray()           # will hold the compressed rows

    rawRows = {}                    # a dictionary to hold the uncompressed rows.
                                    #  Key = row, value = index in pointer table
                                    # This allows us to reuse mulitple rows if they are identical
                                 
    for rowIndex in range(0x100):
        row = src[ (rowIndex * 0x100):((rowIndex * 0x100) + 0x100) ]
       
        if row in rawRows:
            x = rawRows[row]
            ptrTable.append( ptrTable[(x * 2)    ] )
            ptrTable.append( ptrTable[(x * 2) + 1] )
        else:
            rawRows[row] = rowIndex
            row = compress(row)
            ptr = ((len(rowGlob) + 0x200) & 0x3FFF) + 0x8000
            ptrTable.append( ptr & 0xFF )
            ptrTable.append( (ptr >> 8) & 0xFF )
            rowGlob += row
   
    return ptrTable + rowGlob
   
   
# support function to change the file extension in a string
def changeExt(name, ext):
    pos = name.rfind('.')               # find the last dot in the string
    if pos < 0:                         # if one wasn't found...
        return name + '.' + ext         #   just append the ext to the name
    else:
        return name[0:(pos+1)] + ext    # replace everything after the dot with the given ext
       
def cmapIsOutdated(inName, outName):
    # if the outFile doesn't exist, it is outdated
    if not os.path.exists(outName):
        return True
       
    # get 'last modified' time of both input and output files.  If input was modified more recently, then
    #   the output is outdated
    return os.path.getmtime(inName) > os.path.getmtime(outName)
       
def doFile(inName, force):
    outName = changeExt(name, 'cmap')
   
    if force or cmapIsOutdated(inName, outName):
        rawdata = open(inName, 'rb').read()
       
        if len(rawdata) > 0x40*0x40:    # too large for standard map?  If so, assume it's the overworld map
            compressedData = overworld(rawdata)
        else:
            compressedData = compress(rawdata)
           
        open(outName, 'wb').write(compressedData)
   
if __name__ == '__main__':
    if len(sys.argv) < 2:
        print('Usage:  python ffcompress.py <filename> [-f]\nIf -f is specified, recompression will be forced')
    else:
        force = (len(sys.argv) >= 3 and sys.argv[2] == '-f')
       
        for name in glob.glob(sys.argv[1]):
            doFile(name, force)
 


Example usage:    python ffcompress.py maps\00.bin

The above will create the file 'maps\00.cmap' which will be the compressed map.

Additionally wildcard characters will work, so you can do this:    python ffcompress.py maps\*.bin

And it will compress ALL .bin files in that directory and create their respective .cmap file


Also, the compressor will examine 'last modified' timestamps and will only recompress maps if their cmap file is out of date.  So you can run this in a build step and it will mostly do nothing (and thus run very fast) unless you have recently modified a map.

However, this behavior can be ignored if you do:  python ffcompress.py yourfile.bin -f

The '-f' will force compression to happen even if the cmap file is up to date.


Last thing to note:

A compressed overworld map WILL contain a pointer table.  However compressed standard maps DO NOT.

I figure it'd make more sense for the standard maps to be individual files that can be incbin'd in the source, and you can probably use labels in ca65 to generate the pointer table.


Lemme know if/how this works for you.

Jiggers

  • Sr. Member
  • ****
  • Posts: 373
    • View Profile
    • My Ko-Fi Page
Re: FF1 MMC5 Disassembly Updates
« Reply #596 on: May 30, 2020, 11:56:24 pm »
Last thing to note:

A compressed overworld map WILL contain a pointer table.  However compressed standard maps DO NOT.

I figure it'd make more sense for the standard maps to be individual files that can be incbin'd in the source, and you can probably use labels in ca65 to generate the pointer table.


Lemme know if/how this works for you.

Woo! I'll test it out tomorrow sometime, maybe--its getting cooler so my brain is turning back on.

Yeah, this will work! I was doing that at first because I didn't realize at the time I could just import/export all the map labels to the other banks. This made a slight problem where editing Coneria meant I had to move the start of the last map in the bank to the start of the next bank, and so on. Anyway, that's all fixed because THEN I figured out just now how to set 1 bank file to be 3 banks in size. So Bank $16 is now also 17 and 18, all the maps are incbin'd in just that one file, the pointer table builds itself without needing to import/export ANYTHING and it works just fine!

The rest, I will explore further when I can. Thank you so much!
I know exactly what I'm doing. I just don't know what effect it's going to have.

I wrote some NES music! Its a legal ROM file. - I got a Ko-Fi page too.

Disch

  • Hero Member
  • *****
  • Posts: 2814
  • NES Junkie
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #597 on: May 31, 2020, 11:14:22 am »
Noticed a bug that I'm honestly amazed I didn't spot before.  Updated code:

Code: [Select]
import os
import sys
import glob

# define a function which outputs the compressed version of a run of a given length
def outputRun( out, runByte, runLen ):
    # do this in a loop.  If there are runs longer than 256 tiles, we have to
    #   compress it in "chunks" that are no bigger than 256.  The while loop here
    #   lets us repeat this "chunking" process as many times as is needed
    while runLen > 0:
        if runLen == 1:         # was it a 'run' of only 1 byte?
            out.append(runByte) # just output it normally  (no compression)
            runLen = 0
        elif runLen == 2:       # was it 2 bytes?
            out.append(runByte) # just output it twice, compressing won't save any space
            out.append(runByte)
            runLen = 0
        elif runLen >= 256:     # Run of 256 or more, we can only do 256
            out.append(runByte | 0x80)
            out.append(0)
            runLen -= 256
        else:                   # else, it's between 3-255... compress it normally!
            out.append(runByte | 0x80)
            out.append(runLen)
            runLen = 0

# define a function to compress some data
#  'src' technically be a string, but we will be treating it as a list of bytes
#  we will be compressing the input bytes and returning a bytearray of the compressed data
def compress(src):
    out = bytearray()       # create our output array
   
    runByte = 0             # a var to keep track of the current 'run' byte
    runLen = 0              # how long the run is
   
    for val in src:         # loop over each byte in our input string
        if val == runByte:          # is this part of the run?
            runLen += 1             # if yes, tally it
        else:
            # otherwise, we're ending a run and starting another.
            outputRun( out, runByte, runLen )   # before starting this new run, output any previous run
                   
            #  Next, we need to start a new run
            if val == 0x7F:     # can't start a run with tile 7F, because that would erroneously create an $FF byte
                # so just output this 7F tile immediately and don't start a run
                out.append(0x7F)
                runByte = 0
                runLen = 0
            else:               # any other tile can have a run
                runByte = val
                runLen = 1

    # Now that we've gone through the whole map, output any run we haven't output yet
    outputRun( out, runByte, runLen )
    out.append(0xFF)                # add the terminator
    return out
   
   
# Another function!  One specifically to handle overworld maps, since those have
#  the additional hassle of a per-row pointer table
def overworld(src):
    ptrTable = bytearray()          # will hold the pointers
    rowGlob = bytearray()           # will hold the compressed rows

    rawRows = {}                    # a dictionary to hold the uncompressed rows.
                                    #  Key = row, value = index in pointer table
                                    # This allows us to reuse mulitple rows if they are identical
                                 
    for rowIndex in range(0x100):
        row = src[ (rowIndex * 0x100):((rowIndex * 0x100) + 0x100) ]
       
        if row in rawRows:
            x = rawRows[row]
            ptrTable.append( ptrTable[(x * 2)    ] )
            ptrTable.append( ptrTable[(x * 2) + 1] )
        else:
            rawRows[row] = rowIndex
            row = compress(row)
            ptr = ((len(rowGlob) + 0x200) & 0x3FFF) + 0x8000
            ptrTable.append( ptr & 0xFF )
            ptrTable.append( (ptr >> 8) & 0xFF )
            rowGlob += row
   
    return ptrTable + rowGlob
   
   
# support function to change the file extension in a string
def changeExt(name, ext):
    pos = name.rfind('.')               # find the last dot in the string
    if pos < 0:                         # if one wasn't found...
        return name + '.' + ext         #   just append the ext to the name
    else:
        return name[0:(pos+1)] + ext    # replace everything after the dot with the given ext
       
def cmapIsOutdated(inName, outName):
    # if the outFile doesn't exist, it is outdated
    if not os.path.exists(outName):
        return True
       
    # get 'last modified' time of both input and output files.  If input was modified more recently, then
    #   the output is outdated
    return os.path.getmtime(inName) > os.path.getmtime(outName)
       
def doFile(inName, force):
    outName = changeExt(inName, 'cmap')
   
    if force or cmapIsOutdated(inName, outName):
        rawdata = open(inName, 'rb').read()
       
        if len(rawdata) > 0x40*0x40:    # too large for standard map?  If so, assume it's the overworld map
            compressedData = overworld(rawdata)
        else:
            compressedData = compress(rawdata)
           
        open(outName, 'wb').write(compressedData)
   
if __name__ == '__main__':
    if len(sys.argv) < 2:
        print('Usage:  python ffcompress.py <filename> [-f]\nIf -f is specified, recompression will be forced')
    else:
        force = (len(sys.argv) >= 3 and sys.argv[2] == '-f')
       
        for name in glob.glob(sys.argv[1]):
            doFile(name, force)
       
       

Vanya

  • Hero Member
  • *****
  • Posts: 1671
    • View Profile
Re: FF1 MMC5 Disassembly Updates
« Reply #598 on: May 31, 2020, 11:37:35 am »
Awesome!
It's all working out.
I have to tank you too, Disch.
It would have taken me much longer to get this done. :)

Jiggers

  • Sr. Member
  • ****
  • Posts: 373
    • View Profile
    • My Ko-Fi Page
Re: FF1 MMC5 Disassembly Updates
« Reply #599 on: June 01, 2020, 01:59:29 am »
Managed to get that into an exe file so people without Python can use it! Updating GitHub with the new build format.

Decided to keep the map compression out of the main build.bat for now--there seems to be a delay of a few seconds regardless... Unless it doesn't do the timestamp checking with just the wildcard setting? Or not in the .exe version? "maps\compressmap.exe maps\*.bin"...?

Bad News: YY-CHR is a graphics editor and therefore... doesn't work for shifting purposes the way I thought it would. So back to figuring out how to convert the decompressed maps to a format Tiled will recognize, or do it the Python way like Disch suggested... (I reallllyy wanted a visual way to make sure I was getting them right, though... plus a way to edit them, which Tiled does really well; I set up the tilesets and everything!)

I have Tiled set up to use a Python script to export. Having trouble understanding any of it.

https://doc.mapeditor.org/fr/stable/manual/python/#api-reference

Just want to read a tile, convert the number to hex, no spaces, no commas, no extra lines... Should be simple, but where do I even start! (Not to mention how to import the .bin files in the first place...)

Update: Having computer troubles. Took the water cooling unit off this 8-10 year old computer, tried to clean it out, didn't put it on right or something. Or put it on right but its making more noise than before. *shrug* I've backed up as much as I can, but if I lose interest in this stuff for the next few weeks... this is probably why.
« Last Edit: June 01, 2020, 03:41:04 pm by Jiggers »
I know exactly what I'm doing. I just don't know what effect it's going to have.

I wrote some NES music! Its a legal ROM file. - I got a Ko-Fi page too.