News: 11 March 2016 - Forum Rules

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - Geiger

Pages: 1 [2]
21
Programming / Re: Temporal Flux Journal
« on: January 12, 2012, 03:36:46 pm »
Didn't see much point in starting a new thread for this.  Why haven't I posted to this thread in eight months?  Enh, effort.  It was also unclear how much interest there was in the topic.  But I'm still working on Flux and something interesting came up (sort of).  (When is this going to get done?  The todo list is still as long as my arm, but I hope to have closed Alpha builds going by the end of the month.)

---

Implemented "Load On Demand".  This feature may go unnoticed by most.  Previously, Flux front-loaded every record in the ROM file into memory (frequently also decoding it).  This process usually only takes a few seconds, and is almost instantaneous on modern systems.

But LOD is something I have always really needed because in debug everything takes an order of magnitude longer to load.  Fifteen to thirty seconds.  This may not sound like much, but I frequently need to switch between ROMs when testing features or even just load the same ROM multiple times to check saving.  Making and checking small code changes (which happens a lot) or changing breakpoints in the debugger also costs me an additional quarter to half minute.

I have tried to implement this before, a few years back.  It didn't work too well and would have required a massive rewrite of the code (funny that) so everything got changed back.

Why did it get implemented now?  Because of a mistake I made while checking something, oddly enough.  Flux now stores its records to the project folder and reads them from there on load.  It didn't occur to me that my standard test ROM name had previously been the subject of a Reseat All Records test.  There were thousands of records to load from disk.  Reading from the disk is very slow, especially when its lots of small files.  So upon watching the release build run slower than debug typically does, I decided some changes needed to be made.

Now Flux will do a minimal amount of front-loading (fixed small or very important records) and then shift the rest of it to the background.  If the user selects something that isn't loaded yet, Flux immediately loads the requested records.  The background loader is smart enough to not load the same record twice, also.

22
Programming / Re: Program Design and Data Structure (TextAngel)
« on: April 12, 2011, 04:58:08 pm »
What do you think of that idea?

Sounds very similar to my setup, which should work well.  Mine is a little more general purpose though, since PointerRecord is used for all record types.  The other three arguments to the new operator up there are, in order:

. Address Adjustment (short) - when the stored pointer does not point directly to the data.  Usually it points to some value in the data of a fixed size record.  It is used in the Treasure data (and probably several other record types that I do not move).
. Bank Present (bool) - the pointer's Bank address is present
. Address Present (bool) - the pointer's Range and Byte addresses are present

The booleans are true for all instances of strings in Chrono Trigger, but other data sometimes only has a Bank or the Range and Byte addresses present.  While local pointers should be no mystery, the reason there is sometimes only a Bank pointer is because the Range and Byte addresses have already been loaded somewhere else.  Frequently through special register byte math.

23
Programming / Re: Program Design and Data Structure (TextAngel)
« on: April 12, 2011, 09:00:21 am »
I'd love to see some specific examples.

A lot of the string pointers are implemented in LDA statements.  (The first three arguments to creating a PointerRecord here are raw Bank, Range, and Byte addresses.)

Code: [Select]
Rec.Pointer[0] = new PointerRecord(0x02E4D9, 0x02E4D4, 0x02E4D3, 0, true, true);
Rec.Pointer[1] = new PointerRecord(0x02CA8B, 0x02CA86, 0x02CA85, 0, true, true);
Rec.Pointer[2] = new PointerRecord(0x02E5F7, 0x02E5F2, 0x02E5F1, 0, true, true);
Rec.Pointer[3] = new PointerRecord(0x02E85B, 0x02E856, 0x02E855, 0, true, true);
Rec.nRecords = 0x07;

And while I don't know that this is within the scope of your program, Flux also updates some hard-coded indexes so string banks can be moved.  (Seems the code tag removes tabs.)

// Adjust index based Years pointers
if(nID == (byte) StrRecType.Years) {
   switch(G.nRomType) {
   // Japanese
   case 0:
      // Indexed origin
      for(byte i = 0; i < 8; i++) {
         G.WorkingData[0x026944 + i] = i;
      }
      // '???' string
      G.WorkingData[0x0268F0] = 0x08;
      G.WorkingData[0x0268DE] = 0x08;
      // Other hardcoded year indexes
      G.WorkingData[0x0268E5] = 0x01;
      G.WorkingData[0x0268D3] = 0x02;
      break;
   // English
   case 1:
   default:
      // Indexed origin
      for(byte i = 0; i < 8; i++) {
         G.WorkingData[0x0269F0 + i] = i;
      }
      // '???' string
      G.WorkingData[0x02699C] = 0x08;
      G.WorkingData[0x02698A] = 0x08;
      // Other hardcoded year indexes
      G.WorkingData[0x026991] = 0x01;
      G.WorkingData[0x02697F] = 0x02;
      break;
   // Beta
   case 2:
      // Indexed origin
      for(byte i = 0; i < 8; i++) {
         G.WorkingData[0x026DD3 + i] = i;
      }
      // '???' string
      G.WorkingData[0x026D7F] = 0x08;
      // Other hardcoded year indexes
      G.WorkingData[0x026D74] = 0x02;
      break;
   }
}

24
Programming / Re: Temporal Flux Journal
« on: April 08, 2011, 09:24:26 pm »
The topic of most interest to me is having some kind of estimate as the program runs of whether there will be too much data when the time comes to save the ROM. As in, do you run simulations periodically or only when the time comes to save the ROM? From a user's perspective, it would be a bitch to do a bunch of editing only to find that the data generated is twice the feasible size a ROM can support (4 megabytes, 6 megabytes, whatever the practical decided limit is.) Naturally, it might not be feasible to estimate compressed data size since that could take a chunk out of processing time.

Such an estimate is not especially useful, due to bank boundaries and not all of the space being together to start with.  There might be a full 64k of space left, and the record in question may only be 512 bytes long.  But knowing both of those things does not help if all of the chunks of available space are only 256 bytes wide.  Even keeping track of the largest chunk of space would not help much, unless you are saving every record back to the ROM after every modification (which would lead to serious fragmentation).  The only way to know (for sure) if a specific record will fit at modification time is to sort and fit all of the records every single time.  For a small set of records without much compression, this may be doable.  But for Temporal Flux, this can take a couple of seconds to a few minutes (depending on the number of modified records).

In Flux, this is not really a large problem anyway.  If a record cannot be fitted, it throws up an error message and the user can expand the ROM directly from within the program.  With the new project approach, even running into the 6 meg barrier is not an immediately dire problem, since all of the data will still be stored to the hard drive anyway.

Quote
That's interesting, any details on what meta-data is being stored and is it uniform across records?

Meta-data (beyond what describes a Flux file) is currently only stored on a per record basis, as I do not see any commonality across all records.  And I have only implemented it in a few areas so far as there is so much work left to get the current WIP back into working condition.  The ones that immediately come to mind are Event Command labels and also false ECs, such as a comment or functionary link.

Quote
I'm dreading the day I have to delve into sort or packing algorithms

It is not really as hard as you may think.  Quicksort is sort of universally recognized as the best algorithm, so most languages have it implemented somewhere (you usually just need to tell them how to determine which is more and which is less).  The main issue will be implementing bin packing, but its not exactly outrageous either.  It mostly consists of defining a 'bin' and means of moving onto the next one if the current one will not fit.  My 'Freespace' class is effectively a bin packer.

Quote
On another note, have you changed any of the compression formats used in Chrono Trigger?

I have given some (non-serious) thought to implementing a zip algorithm.  But I have never touched the existing compression routines because it is important that they be able to work in real time, which they already do (if it ain't broke...).  And as a general rule, I try to touch as little of the code as I possibly can, even if it would make my life easier.  Even the few minor ASM hacks I have created are entirely optional.  I try to work within the boundaries of the original engine, because the more I change it the less of the original game remains.  The point is to edit Chrono Trigger, not something I cooked myself.

Quote
A map of unused bytes? Or is it broken down by some size, like 256 bytes, kilobytes, etc?

I am probably making it sound more complex than it actually is.  At the simplest level, it is just a list of paired addresses marking the start and end of a chunk of free space.  The addresses change and new entries are added as needed when records claim their space or get fitted for saving to a new location.  The container class also has built in functions for adding space back, which requires sorting (the new entry is placed on the end initially) and possibly collapsing (if the space bridges the two entries on either side).

I can provide more detail at a later date, as I do not have immediate access to my code.

April 12, 2011, 09:06:51 am - (Auto Merged - Double Posts are not allowed before 7 days.)
Er, what example I was going to share is not coming to me.  Is there something you would like to see?

25
Programming / Re: Program Design and Data Structure (TextAngel)
« on: April 08, 2011, 12:27:51 pm »
From a cursory glance, I do not immediately see anything wrong with your layout.  The pointer data (as I understand it) would not be able to support the complex situations I have encountered in Chrono Trigger though.  You would need the ability to have an arbitrary number of pointers and the capability to support pointers that do not have all of their data together.  You might also want to add the ability to read substring data from the ROM as part of the string table.

26
Programming / Re: Temporal Flux Journal
« on: April 08, 2011, 10:49:34 am »
Temporal Flux is not open source, right?

The main program is not currently open source, no.

Quote
I'd be interested to get some insight on your approach to managing how data gets saved back into the rom.

This is quite a lengthy topic, as it took me two years to get beyond the most basic saving, five years to get everything compartmentalized, and eight years to bring me to my current project-oriented design.  Do you have a more specific interest?  There is a lot of detail I could get into, I'll start with some very broad strokes here about how the data is treated.

First, its important to note that loading data and saving it are equal endpoints traveling in opposite directions.  Thinking of them as unrelated or wholly separate processes will not produce good code.  To this end, Temporal Flux employs record based loading and saving (as detailed in the Temporal Flux Plugin Architecture).  Each record contains all of the information needed to load the data and save it back, including extra pointers, custom data parsing, and more.

Each record may also have upto three states, though anything more than one is usually abstracted by the form that displays it, not something stored in permanent memory.

The most primitive state is identical to the way the record appears in the original ROM data.  Most of the 2000+ individual records Flux extracts only exist in this form as the data is simply not complex; there is no further need to parse the information.  More complex records only exist in this state briefly after being loaded or before being saved.  Examples of records in this state include Location Properties or Song Instruments.

The middle state is pre-parsed.  The data is complex, almost always not of a fixed length.  This makes it difficult to determine where a specific datum of interest is or should be located, so the data is parsed into a more readily accessed form.  Usually the data still bears a marginal resemblance to the original.  Example of records in this state include Overworld Exits (and previously Location Events).

The most complex state is totally abstracted.  Even a parsed form would be totally unusable to the display of the data (or make it much harder to use than is useful).  The data will be converted and kept in a useful form in perpetuity.  Conversion back to the original happens only in a temporary copy during save.  This state is common for compound records like Events but is also used for records as simple as Strings.  The new project approach keeps the data in this state permanently, even between sessions.  This allows for several enhancements, such as storing meta-data directly in the record.

Flux has always used First Fit for bin packing.  I had each of the separate record type saving processes manually ordered so that the typically largest record type saved first.  But as of v3.00 it uses First Fit Decreasing.  The records had been universalized, so I could tell an individual record to save instead of an entire record type.  This allowed me to put them in a sorted queue and ultimately make much better use of the ROM's free space.

Flux maintains a Free Space map.  This map contains all of the unused space, as well as all of the space used by editable, movable objects.  On Save, the unmodified records claim their space out of the map.  Then the modified records try to save themselves back to their previous home, if they can fit.  If they do not, they will save to a new location.

27
Programming / Temporal Flux Journal
« on: March 31, 2011, 01:15:40 pm »
I have been toying with the idea of starting this thread for awhile now.  Is there any interest in seeing the sort of coding problems a program like Temporal Flux encounters, and how they are resolved?  This will involve tidbits, lengthy explanations, or anywhere inbetween.  Occasionally, I may also post a little code to better illustrate the issue.

I think there is a lot that could be said given the comprehensive nature of the program.  The primary barrier would be in my ability to adequately convey the information to the audience.  I have never posted at length on a regular basis, so I am not sure if I would be good or bad at it.

The other issue would be the irregular posting schedule.  I would only be posting when I am actually working on the project, and when interesting issues occur.

(As this primarily involves code problem solving, I have posted here.  But if this better belongs in Personal Projects, feel free to move the thread.)

---

I have been running into an issue with the original ROM running out of free space when all data records have been marked as modified, without changing the data.  This is unusual because, in theory, if none of the data has actually been changed, it should not take up any more space than it did originally.  The cause appears to be several things.

The first was somewhat easy to find.  Several of the record types lived in space not marked as usable to the program.  Even worse, some of the record types were only partially marked as usable.  For example, I had only marked the primary group of Location Maps as usable, but there are two or three smaller groups of maps stored in other places in the ROM.  Tracking this down manually would have been time consuming and error prone, comparing the list of usable space to the entire list of offsets (which contains records, unused space, and code).  Fortunately, this process could be automated.  After each of the records retrieved its data, it then checked the free space list to see if its home address was available.  If not, it popped up an error message.

Unfortunately, this does not appear to be the only problem, as even after correcting for all of the error messages, the ROM still runs out of space.  It is not immediately clear why, but I suspect it has something to do with new records being created.  Flux can split an overlapping record into two new ones, or create entirely new ones in cases where there is room for more of that type.  There is little point into looking into this because it is not falsifiable:  new records will always take up more space.  I need to make sure there are no other causes before I try to eliminate this.

Maybe Flux is not saving the records as efficiently as the original ROM, taking up more space.  This is the avenue I am currently investigating.  Hampering my progress, Flux has never saved compressed data in a matching one-to-one manner as the original game.  While it still works, the compression addresses never matched up.  The fix here appears to be surprisingly simple.  Instead of looking for matching patterns to compress from the beginning and moving forward, it seems the original tool started at the end and worked backward.  Merely changing the end I started at appears to have made it sync up and post one-to-one results.  I was so surprised that was all it took that, at first, I figured the game was not actually saving the data and was just dumping the ROM back out to file again.  I had to test it several more times and even actively debug the program and physically see it doing the work to believe it.  This will help the investigation immensely.

That is all for now.  Let me know if there is any interest in more.

28
Wanted to point out, since you started from my code, that neither the Chrono Trigger nor Final Fantasy VI routines match the original compression output.  I quickly realized while reverse engineering these that so long as the decompressed data was not corrupted, trying to figure out the original compression tool's magic numbers would only be a gigantic time sink.

29
ROM Hacking Discussion / Re: Editing Stats and Abilities of Monsters in CT
« on: February 07, 2011, 11:02:02 am »
Why not use a different tool that is... altogether more legal and thus, possibly a source of donation money?

Just wanted to point out that Temporal Flux along with the vast majority of game editors are entirely legal.  But if you meant the ROM itself, only distributing the image is illegal.  Under case law, software modification can be distributed legally, even commercially, provided it does not contain the original copyrighted material (and even that stipulation is questionable).

However, IANAL, and even if I were, you'd probably get the judge the ESA has in its pocket.

30
While Woolsey made a few errors here and there in his translations, they still tend to be significantly warmer, more vibrant, and generally better than any fan translation I have ever seen.  The people behind these translations don't seem to understand how localization is supposed to work.

I would have never purchased Final Fantasy VI if it had the fan translation dialogue.

31
Personal Projects / Re: Temporal Flux
« on: February 07, 2011, 10:39:16 am »
There haven't been any ROM corruption issues for quite some time now.  The only reports I have heard about in the last several versions were purely anecdotal.  And even back when there were issues, every version from around 1.05 up have been corner cases that most people won't encounter.

But even if some sort of "stealth" corruption issue still exists, between auto-archiving (introduced a version or two ago), and the new project approach the WIP version uses, you cannot permanently lose any changes you make.

32
Personal Projects / Temporal Flux
« on: February 03, 2011, 04:47:07 pm »
I have been able to fix most of my server issues, so WIP coding has restarted in earnest.

Thanks to the Crimson Echoes C&D, the primary Chrono Trigger editing community sort of imploded.  So I have no idea what sort of interest there is left in the romhacking community for this project.  Regardless...

In the last release before the C&D, I included a plugin architecture (TFPA).  I will need to update it for the next release.  If you have a specific interest in this area, I am taking ideas for additions.

Currently I plan on adding:
Code: [Select]
class RecordInfo {
   byte nID                (Previous RecDict "value".  Indicates record type.)
   string sName            (Previous RecDict "key".  Display Name for export/import.)
   string sDefaultPrefix   (Default filename prefix.  See below.)
}

IFluxPlugin
. List<RecordInfo> RecInfo - This will replace the RecDict to include a default filename for exports and imports.  This is important because of the new approach to ROM projects.
. ushort nInterfaceVersion - Since there will be a new interface, there will be a need to indicate which version is being used.

GlobalShared
. string sProjectDir - any ROM-specific files should be saved in this directory.


I do not currently have a projected release date for the next version.  Most of the hard work is done, but there is still a significant amount of relatively easy work to do (rewriting and delegatizing several hundred blocks of code :D)

33
News Submissions / Re: Utilities: Temporal Flux v3.00 Release
« on: January 04, 2011, 04:56:20 pm »
why do you continue to update and perform Temporal Flux,if there are few manuals...?

I have a limited amount of time to work on Chrono Trigger editing.  I can either write manuals or further enhance the editor.  I always choose the latter.  I leave it up to others to write manuals (all existing Flux manuals were written by someone else).

I strive to make the interface user friendly, but at least some of the onus is on the user to raise their level of understanding.  It is not a tool for first time hackers.

After a C&D Temporal Flux has been discontinued AFAIK.

I never received a C&D.  I would ignore one if I did, since nothing my program does is illegal or infringing.  The next release will have a copyright screen pointing this out.

The reason Flux has not been updated in two years is largely because of the following reasons.

- I no longer have a beta team.  This makes everything much, much slower.
- I have had major server problems that have still not been fully resolved.  While I can technically code without the server, its not something I care to do.
- The approach to projects has been radically altered requiring major internal rewrites.  This is a natural consequence of the changes made for the TFPA in the last version.  I expect this will be the last major rewrite and the first version of the code I feel is useful, thorough, and generalized enough to show others.

34
News Submissions / Re: Utilities: Geiger's Snes9x Debugger M10R1 released
« on: January 30, 2010, 12:02:34 am »
Can you do full memory dump instead of only half in this one?

I don't know offhand.  Probably not.  I usually try not to broach banks because of the way mapping works, but I can try to look into.

Tab stops too?

Perhaps the wrong term.  Tab Order.

uneven tab stops in the trace logs was enough of a problem for me to keep using version 8

I don't think you understand what a tab-delimited file is.  Turn that feature off and it should go back to regular spacing.  As far back as there's been a debugger (and maybe before).

In Chrono Trigger, setting a read breakpoint on FF0000 (start of menu portrait data) and opening the menu does nothing. Setting a write breakpoint, however, stops it like it should.

A lot of stuff was changed in the 1.51 codebase.  Some so radically that I had to make educated guesses as to what it meant and how it would effect my debug code.

Does a DMA transfer occur there?  I may be interpreting the flag for it incorrectly.  Previously it was called something like TransferDirection.  Now its called ReverseTransfer.  (So which direction does the Snes9x team consider "reverse"?  Why did some guy decide that needed to be changed?  Why do they use variable types as variable names?  Why is everything a jungle of #defines?  Why does no one who ever works on this project seem to know what a proper coding style, class, enumeration, inheritance, code reuse, or even a freakin' whitespace is?  BLARGH!) :banghead:

But I digress...

35
News Submissions / Re: Utilities: Geiger's Snes9x Debugger M10R1 released
« on: January 29, 2010, 05:07:42 pm »
If you get some error about the application or side-by-side "configuration", its a brand new type of Hell that Microsoft introduced with VS2005 to replace DLL Hell called Manifest.

What's worse is that they don't seem to keep their redistributables up to date either.  So I've posted the one that comes with Visual Studio up to my website.  http://geigercount.net/crypt/vcredist_x86.exe

This has resolved the problem for at least one person, hopefully it works for the rest of you as well.

Pages: 1 [2]