News: 11 March 2016 - Forum Rules

Author Topic: Thoughts on making a set of universal program fragments for ROM hacking tasks  (Read 3933 times)

FAST6191

  • Hero Member
  • *****
  • Posts: 3389
    • View Profile
Title probably says most of it and the answer I imagine is do it and see how it works as it is probably not that hard, indeed I would probably say the same thing if someone asked.

On newer consoles that use filesystems there are lots of formats of a slight difference from those in another ROM but with many of the broad strokes (magic stamp, followed so many bytes later by a file size and then an archive table or some such). There are tools and methods that can work around this (a command line file trimmer, a hex editor's boolean/binary operations and find/replace and a spreadsheet often being my weapons of choice here) but they are far from ideal there (their read options for one) and even worse if you want to put something back together.

My thoughts would then be to make a series of files aimed at handling this custom stuff and maybe abstracting away the filesystem side of things (even to the point of straight command line call to an existing program though the goal would probably be a library) but comment them up a treat and maybe even include some compression handling (again external as most will have telltale signs and methods will not tend to be too custom on the filesystem using consoles I have dealt with- in the case of LZSS it is a shock if the split between the pointer and length section changes).

The putting it back together stuff might be tricky but a lot of hacking happened when output/helper/debug files were left behind and I reckon that would be a good model to follow here (basically dump the relevant header sections/values to a debug file and use it in rebuilding- as it stands a few info tools that spit out text would be quite handy).

Now we have had tools that aim for this- I am sure we have all fought complex text insertion/extraction tools and there are things like Romulan as well but I reckon a proper programming language (leaning hard towards python at this stage* but a few others are doing things in Ruby and other "web dev" type languages get discussed in increasingly large amounts- I vote no java if only because I try not to have it on my systems and I am doing well enough in that quest) would be a far better option in the short and the long run. Thankfully ROM hacking seems to have been spared the LISP is good idea that infested a few other areas of computing (see most macro languages that MS did not have a hand in). As it stands we have also had to explain the finer (or indeed basic) points on GPL on a few occasions when nice scripts get released with source but here it would be aiming as a learning tool (probably more than a rapid development framework which would be the second possible use here) and almost to the point of risking creating a scripting engine inside the "fragment" in some cases.

*some of the fire emblem stuff had it, Zahlman's audio stuff for the GBA is pretty nice, http://nvwr.googlecode.com/svn/trunk/libs/

I freely admit I am basing much of this solely on the DS and related consoles but given what happens there I hold it to be useful. I would probably push for http://creativecommons.org/choose/zero/ as the license here but there definitely room for debate there. I doubt I will have much time to do anything here in the coming weeks and months and my python code is not something that I would say makes a good teaching tool . However if you have ever copied a bit of code and used it as the base for the next project then hopefully you can see something in this.

Nightcrawler

  • Hero Member
  • *****
  • Posts: 5792
    • View Profile
    • Nightcrawler's Translation Corporation
These concepts can be applied to all consoles (not just new). There are always general processes, procedures, and high level structural elements that are common to most ROM hacking projects. There is always opportunity to abstract much of the game specific elements away by being able to extract and build the structural elements you're dealing with.

We certainly could use more high level tools that do these types of things and work on nearly any game. I've been working on one such tool (TextAngel) for awhile in the dumping and insertion realm. It's a project based utility that provides all the building blocks you would need to set up structural elements to deal with near any game. By revolving around building blocks and structures, you can extract, rebuild, and insert with near limitless possibilities. You're limited only by the building block elements you provide. I imagine this type of tool is along the lines of what you're talking about.

I can tell you it is a very large task to do tools like this. It takes much effort to abstract these structures to a high enough level and break the building blocks down small enough to be flexible enough for use on a wide number of games. Although my project is going along fine, it's taken me much longer than I ever anticipated to make it truly globally useful. I hope for your sake it is simpler to deal with virtual filesystems than text storage structures!

One thing I disagree with is using a programming language like Python unless you want to only target a small user base. Once you require all users to know a programming language, this eliminates over half of all ROM hackers. Then, of the remaining half, a number of them would probably build their own utilities if they have to program algorithms anyway.

I think one fundamental goal of such projects should be to simplify as much as possible to make hacking these items more accessible to all hackers, not just those that know programming languages. That's the whole point to abstracting away the details. You don't want to have to be writing individualized pieces of code or target only those users who already can deal with the details.
TransCorp - Over 20 years of community dedication.
Dual Orb 2, Wozz, Emerald Dragon, Tenshi No Uta, Glory of Heracles IV SFC/SNES Translations

LostTemplar

  • Hero Member
  • *****
  • Posts: 910
    • View Profile
    • au-ro-ra.net
My personal favorite language is C++, but still I'd think it would be best to use C as a base for a project like this. My reasoning revolves around the following aspects:
  • it's (inherently) a compiled language, yet still very portable; however, compared to C++ - which nowadays probably is just as portable with respect to the three major platforms - it doesn't lack a standardized ABI (-> so you can more easily provide pre-compiled libs)
  • starting from C, it should be easy to write wrappers for all other major languages - C++, Python, Java, C#, ...; definitely not as easy with C++, and writing wrappers from Java or Python is not very practical
  • even without wrappers, C arguably implies the largest audience
A general ROM hacking library should supply a lot more facilities than just file systems, but it's a very good start I think. That's because almost every post-PSX game uses them, and directories and file names provide very good means to guess the content of certain files, as opposed to ye olde SNES ROM, for example. Also, I've yet to see a tool that lets you browse, un- & repack most common file systems and archives (mostly the same thing in my opinion), for instance ISO, NitroROM and the Cri Filesystem (CPK). That's why I think it might be a good idea to first implement a generic file system library and base a file browser (& maybe viewer) on it. Then one could think about what to do next.

creeperton

  • Hero Member
  • *****
  • Posts: 604
    • View Profile
.
« Reply #3 on: March 17, 2013, 01:34:25 am »
.
« Last Edit: November 16, 2015, 01:22:41 am by creeperton »