An 'All in One' data-oriented ROM editor

[Ed101]

Hi All,

I'm working on an "all in one" hex editor/tile editor/ROM hacking program. The key feature for me are "bookmarks", mouse and keyboard input and GUI. The closest examples I can find are WindHex, Crystal Tile 2, and good old Thingy. I think of it like a mirror to IDA Pro/Ghidra: i.e. it is a data-oriented reverse-engineering tool.

The best program I have found for "bookmarks" is HexWorshop (.hbk format), it has the best visual representation of a blob of data I've seen, but no .TBL support (and the .map custom format cannot deal with non-byte character codes). I also want to be able to see bookmarks in a tile editor. I also want a broadly-usable set of objects that can be placed at these bookmarks: e.g. String, Pointer, Tile, Audio, ASM, Table/Array etc, to more accurately document one's findings in a game.

The idea is to use a GUI tool with bit-accurate bookmarks to document a ROM; these bookmarks can then be used to import/export data/objects in the same way Atlas/Cartographer (or abcde) does. These "found objects" can have useful relationships with each other:
e.g. a tile object can be within a data blob object (in RAM), that is the result of the Decode() function of an RLE object within a data blob object (on disk). e.g. A tileset object is an array of tile objects. e.g. A tilemap object is a matrix of tileset object references and properties.

Here is a screen shot from my 1st prototype (C# WPF) featuring a hex editor with: awful graphics and performance!, basic byte-aligned bookmarks, ASCII and .TBL support (via a plugin system), and a basic String object type:

I've created 2 string objects in the above example, they are highlighted in green.

Before I start on prototype 2, I wanted to see if there were any other on-going projects with similar aims that I should contribute to instead?
If I continue with my own project, the plan is to put it on github and ask for plugin contributions (compression algos, system configurations/functionality, graphic formats etc); this seems the best way to give broad support to any individuals needs and allow me to concentrate on making the program work well .

This is a quick mock-up of what a user project might look like in my finished tool:


What do you all think? Is the "tables/pointers" approach still used by a lot of people to translate/hack games? Or am I the only one who doesn't want to go straight to disassembly/avoid it entirely?

Is this "visual data inspection/ interactive notes" approach still useful if your aim is to disassemble the game?

[DragonAtma]

I'm just one person, but since I don't know SNES coding I go with text/pointers.

That said, having both byte-based editing (HxD, WindHex, etc.) and tile-based editing (Tile Molester, Tile Layer Pro, etc.) in the same program would definitely be useful.
And then six different people tell me such a program already exists, only I don't know about it.

[Ed101]

Thanks for the input

A similar idea for an all-in-one tool that does exist, but targeted at NDS, is tinke. I haven't tried it yet, and it seems to not be quite what I'm after.

And just to note that Crystal Tile 2 has both a hex editor and a tile editor, but I find the interface quite lacking.

[FAST6191]

Marginally related but some of the NEF stuff (which Crystaltile2 does actually support in some fashion) could be of interest
http://www.problemkaputt.de/gbatek-symbolic-debug-info.htm

I have played with hex workshop bookmarks in the past, though usually just to make nice coloured images for screenshots rather than my own reference. I don't particularly recall going in for that sort of thing with my usual freeware answer to commercial stuff package (xvi32, icy hexplorer, mirkes.de tiny hexer and hxd being it).

https://github.com/Gericom/EveryFileExplorer could be worth a look but like tinke it is more file based than hex editor/tile editor/disassembler that also happens to be aware of a few formats.

[Ed101]

Thank you. I had a quick look through your crystaltile2 documentation on GBATEMP, as well as the link you gave about NEF. I'll look further in to the NEF format and how Crystaltile2 uses it, in particular how it interfaces with no$gba. Import/Export of SYM, NEF, or other symbol/debug files should be no problem, but note that actual DASM/debugging support is listed as 'blue sky' on my design doc!

I hadn't heard of EveryFileExplorer, I'll have a look at that alongside tinke. I'll review/re-review those hex editors too, and see what they have to offer.

I have been looking at WpfHexEditorControl as a potential start for the hex editor component, it is nice and fast, but is missing some key features for me. I could then submit pull-requests back to that github project. I am also thinking of using AvalonDock to allow the program to look like an IDE with a suite of tools (e.g. Visual Studio).

[thunderdisk]

HexManiacAdvance seems pretty similar to what you're describing, however it is specifically made for the GBA Pokemon games and I have no idea how well it would work with other games.

[Ed101]

Nice, I just gave HexManiacAdvance a quick go, and it doesn't seem to support anything of use to my Game Gear ROM. It is interesting though .

I also found Text Angel, not sure if it was every released publicly, but is the same high level concept, but for text only.

[nesrocks]

This is a very useful tool and I have no idea why it hasn't been made before. 

[Hamtaro126]

This looks like a cool project... With plenty of development time and testing, I say go for it! 

[Ed101]

Thanks for the kind words. I do think I am going to pursue my plans, I can't quite find what I'm looking for out there.

I found another close call: ImHex, it has .TBL support but I can't copy/paste text using the .TBL file as a transcoder (this is a key keature). Note that Translhextion does allow you to dump/ reinsert TBL transcoded data using a text file, but it is not as easy to use as I imagine it could be.

[FAST6191]

Re plugins. There are a few out there. That said for graphics at least what about going more generic https://github.com/puggsoy/tiledggd-pe-
For the sake of text/those unaware then tiledggd is a graphics viewer (not editor) that opts for a more programmable as it were setup than the classic tile editor that might instead select by console (nothing wrong with having such premade setups in a dropdown either that just quickly set parameters). Or if you think about it there are only so many arrangements of red, green and blue, so many bits able to be dedicated to things, so many ways of doing alpha (tiledggd actually not quite making it for some of the DS 3d texture formats where tools like tinke can), endianness and such. A tool that could apply a known (de)compression routine to something and have a temporary workspace for it (like most tile editors do) would be something new and worth looking at as well.

Transcoding via table... I figure there is a reason there are usually ripping and insertion tables, even for same language projects. You also immediately run into the "what do I allow people to program in?" question to handle placeholders, end of line and markup that plagues atlas and cartographer and everything that wants to be it (granted that for the most part means kruptar7 http://www.magicteam.net/index.php?page=documents https://www.romhacking.net/forum/index.php?topic=16496.0 https://www.romhacking.net/utilities/612/ https://romhack.github.io/doc/kruptarPlugins/ ). Starts with a just, leads to feature creep and then barrier to entry regardless of whether it is programmed or a lot of boxes to select.
That said there are enough cases of the generic case of "text is text, pointers is pointers" that it could be useful for general purposes and quick fixes, and those that care to take care of any thinking themselves (often a fatal mistake when it comes to something as tedious as text editing but hey) and can work around such issues might well appreciate it.

Similarly I don't think it is open source but the principle is simple enough and seems relevant at this point
https://www.romhacking.net/utilities/504/

Re disassembler being a distant dream. If you can crack text-hex-text then binary-static disassembled equivalent and possibly binary again should not be hard.

[Ed101]

Thanks FAST, all of that is valuable food for thought. In particular, "barrier to entry" is something I want to manage well, and is probably the most daunting task when trying to make something all encompassing.

I'm a bit under the weather at the moment, so I'm just annotating interesting offsets with my current project: translate Shining Force Gaiden (GG) from Japanese to English, and using abcde to dump/insert text.
To echo the "barrier to entry" sentiment: I'm currently struggling to get abcde to work with an Adaptive Huffman coding (Vitter algorithm I think)... abcde works with normal Huffman coding (you create a TBL file for it), no idea if abcde works with an adaptive scheme or not, or if it is just my brain that needs to catch up!

I'll hopefully start prototype 2 of my tool next week, using my own project as a reference for functionality needs.

The plan is to build up to an RC1 version, and for that to clarify how a technical end-user can "expand" the functionality to cover the game(s) they are interested in (plugins, XML files, UI options etc). If the expandability seems non-viable at this point, I'll re-evaluate the project (at worst it becomes an SFG editor).

[Bunkai]

It looks great, be sure to add a good documentation if you can please.

[Ed101]

Yep, documentation (in-program and out-of-program) are high up the list. Also multiple language support is already present in my unreleased prototype. I'm going to make it as easy as possible to have all UI elements presented in the language of your choice (Obviously will require someone to do the translation in the first place). I'll try and get right-to-left reading order in as soon as possible too.

This is either "go big, or go home". All I know right now is: that I have no idea if this is all going to come together!

I'm hoping that prototype 2 will answer that question. I've done some thinking/writing today so I have a good idea of my aims.


Is there a generally accepted standard for the "text file" format for rom data extraction/insertion tools?
e.g. CSV: label, offset, length, data (with one row per data block, ordered to user's preference/needs)
Or a de-facto tool most use?

[FAST6191]

Everything here tends to be fairly custom from what I have seen, though there are not many generic tools. It also necessarily gets complicated -- pointers alone needing handling for relative, offset*, standard, plain length values, and possibly patterning or boolean/general binary fiddling (easy to blank a top bit set as an indicator with an AND of 0111...1111).
This usually drives people directly into the arms of a full programming language** rather than attempting a markup approach, be it custom or XML, with Atlas and Cartographer being the main exception. Barrier to entry though where someone could happily do a markup based effort for simpler things.

*which also doubles for memory level stuff if working with that.

Plugins for tinke might be of interest, as might the others. There are a few older "programmable"/scripting languages (I tried to find one in particular the other day but could not, had a weird license where using it required you note it in the readme of your hack).

**indeed simple file format listings are often split between as a C programmer might envisage it and other https://www.romhacking.net/documents/[469]nds_formats.htm#SDAT
https://web.archive.org/web/20080214135331/http://kiwi.ds.googlepages.com/nsbmd.html
https://web.archive.org/web/20080106134616/http://kiwi.ds.googlepages.com/sdat.html

If I were to suggest something to look at, and it would be a dim and distant thing if the other goals are first, then probably would have to be avisynth
http://avisynth.nl/index.php/Main_Page
Other than macros in Excel it is one of the things to stand the test of time here. Everything else is more straight programming language like Lua (which many emulators support), visual basic, autoit or something in the macros world (autohotkey and whatever the flight sim and such people are using being a good jumping off point)

[Hamtaro126]

Could you add a Text-to-Tile map mode? similar to Djinn tile mapper or YYCHR.NET?

https://www.romhacking.net/utilities/349/

YYCHR.NET (includes PRG Editor) can be found somewhere else.

[Ed101]

FAST6191: Thank you so much, you have given me so much valuable reading in your posts - I'm so glad I made this topic! I like the mention of avisynth, it is always my go to when trying to make some obscure video/audio work, and for anything simple I use MeGUI (which writes avisynth scripts for you).

Here are some current thoughts on expandability:

• if all compression algos are in one/few plugins, there is lots of example code for adding a new algo (i.e. easy to teach yourself the plugin language, whatever it may be)
• can all RLE methods be supported by parameters?
• Can the concept of "data interleaving" be applied in a general sense, so that any weird data storage, once implemented via plugin as an "interleaved data type", will work seamlessly with the core functionality?

I have ideas for most of those pointer types, and the ones I hadn't thought of (bitwise actions) shouldn't be too bad either. My inspiration earlier today was to generalise "patterns" as a version of sed for hex/binary/variables, as a means to search for data, if I do crack it (if!) it shouldn't be too hard to use the same pattern concept for "value adjustment".


Hamtaro126: Yeah, I think I know what you mean: interpret (rom) data as a reference to a tile map (e.g. 1 byte per reference, map of 256 tiles), and then browse the data similar to a tile editor.

That's part of the plan, and the tile map can of course be the game's font, or any other map of tiles (doesn't have to be from the same file).

[FAST6191]

The self documenting nature of avisynth (how many ROM hacking projects have gone awry when someone makes too many changes and does not know where the crash was introduced, of course we can say take backups and maybe after the fifth time that sinks in...) and its clip focused approach (though I would look more to something like git for change tracking) might also be a boon but that is going away from original intentions and more pie in the sky.

Interleaving... You mean like column mode in notepad++? https://www.technipages.com/notepad-column-mode

I generally teach compression as a parameters thing anyway and RLE is just a special case of LZ for the most part, which varies in the length of flag and type of flag, or split within flag (10 bit flags, one does 6 bits and 4 for the read back and length to read part, another 5 and 5... totally incompatible even if they had not flipped order, trivial however to make the general case).
I don't think I linked it earlier but https://ece.uwaterloo.ca/~ece611/LempelZiv.pdf is my usual read this to learn compression thing, and despite the name it does cover everything short of the compression contest AI things (for some compression research is AI) or more applied stuff from the video world where somewhere underneath it all it is LZ but between frame types, macro blocks, the order it takes through the frame, weightings and whatnot it is a lot more on top of things, bit like learning electronics engineering signals theory and then being lumped with a website to run).
That said RLE was kind of a dead concept by the time games I mostly look at are in play. I don't know what some of the more exotic efforts in the late 16 bit era (never mind PC or oddities in Japan*) will be like.

*I was still seeing LZH files well into the 2000s coming out of Japan (ACE having been dead for several years by this point), and the few times I glance over into the visual novel camp and don't run away in abject terror then strange things there as well.

sed as a general case... I don't know I have particularly thought of doing anything like that before (if pondering the wonder combo of grep, sed, awk then most ROM hacking stops at "binary grep is nice") but could prove useful, even more so if you are combining what could be a relative search type setup in that if I am reading that correctly.
If you are going for relative search then while you are at it https://www.romhacking.net/forum/index.php?topic=27800


I also hope your temporary workspace code is sorted now as trying to crowbar that in backwards when someone wants to decompress a file into RAM (or temp hard drive maybe), interleave it with something and then run a fancy abstract search (and replace) that you can undo all on a chromebook... yeah.

[Ed101]

That's good to know RE compression etc, I'm sure I came across that doc, not sure if you linked it or not! I'll read that when I implement compression.

Yes, I was implying relative search in what I was saying, so that topic will be a great read.

I haven't done any coding since my first post in this topic, so haven't sorted the workspace code yet! I'm going to make a start now though.

I will use the huffman coded text I have as a test bed for the temp workspace, and I'll allow temp file on disk or full RAM file.

Edit: And column mode is something to reference for what I mean by "interleaving", thanks. I was thinking of voice marks being on the "line above" the kana, there is also some weird example in the abcde examples (can't remember which of the games it is now), and more generally, multiplexing video/audio/subtitles with digital video containers.

Oh and my simple solution to "change tracking" is to backup a time stamped copy of the project's "text files" to a 7zip archive every time the project is saved. Crude, but should work well I think. On by default.

[dawnbomb]

Hey there, i actually wanted to make a very similar tool, and i started on it, made a thing, and then said not good enough and started remaking it in a sequel. After some burnout i took a break for a month for another project, then had an idea and came to this forum attempting to see if i could recruit an ally. Lucky me, here you are. Due to our project / goals being quite similar, although yours being for old roms and mins for newer data, it still seems like we could work together.

Can you add me on discord? My discord is Dawnbomb#3408 and i also take new user onboarding and barrier to entry very serious, it's the entire reason i started my projects. Please contact me i would really love to talk.