News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: abcde: Atlas + Cartographer + A Whole Lot More  (Read 4560 times)

abw

  • Sr. Member
  • ****
  • Posts: 328
    • View Profile
abcde: Atlas + Cartographer + A Whole Lot More
« on: March 11, 2018, 09:20:15 am »
abcde (http://www.romhacking.net/utilities/1392/) is a script extraction/insertion utility that aims to take as many of the things that are good about Atlas and Cartographer as possible and make them even better.

Features:
  • Implements all features supported by Cartographer.
  • Implements all features supported by Atlas except for running Atlas extensions.
  • Corrects many features that were broken to varying degrees in Cartographer and Atlas.
  • Uses new table file syntax that allow you to:
    • Work with non-octet text encodings (e.g. 6-bit tokens, Huffman encodings, etc.).
    • Handle mid-string encoding changes (a.k.a. table switching within a single string) with a great deal of flexibility.
  • Adds new extraction commands that allow you to:
    • Dump multiple strings from a single pointer.
    • Re-align the ROM read position when an end token is encountered.
    • Dump strings with internal gaps.
  • Adds new insertion commands that allow you to:
    • Execute arbitrary insertion commands when the insertion point reaches a specified ROM address.
    • Count things and write the counter value to ROM like a pointer.
    • Handle Pascal strings that specify their length in number of tokens rather than number of bytes.
    • Write pointers (embedded or normal) whose value is the distance between the pointer and the text rather than the text's address.
    • Change the current insert position by a relative amount rather than specifying an absolute ROM address.
  • Open source code that you can modify or extend to suit your own needs or preferences.

I'm looking for some testers, new test cases, and feedback on the utility itself and its supporting documentation/examples. Any volunteers? I've got some specific concerns for each area, but I'll listen to whatever you've got to say.
  • Testers: Does this even run on your system, or did I end up making it platform-specific by accident? Was I concentrating too much on testing the complicated stuff that I missed some glaringly obvious bug in the simple stuff?
  • Test cases: Do you have enough experience working with some game that uses a complex text encoding to be able to try using abcde to extract and insert its text? If abcde can't handle it (for reasons other than embedded pointers), I'd love to get a detailed description of the game's text engine so I can make abcde even more powerful.
  • Utility: Better/more user-friendly debug output is on the to-do list, but aside from that, what do you think?
  • Documentation/examples: Do they provide a sufficiently descriptive high level overview? Did I overlook explaining some crucial detail required to get this thing to actually do what you want it to do? I'm a little too close to abcde to be entirely confident that the supporting text doesn't assume too much prior knowledge.

Thanks in advance!

Pennywise

  • Hero Member
  • *****
  • Posts: 2259
  • I'm curious
    • View Profile
    • Yojimbo's Translations
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #1 on: March 11, 2018, 11:16:09 am »
I'll have to use this when I get a chance.

I'm working on a game that uses a byte to swap tables and it uses the same end control code to restore the default. It's basically like a hiragana/katakana switch. Is this something your tool supports?

abw

  • Sr. Member
  • ****
  • Posts: 328
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #2 on: March 11, 2018, 11:50:34 am »
Absolutely. One of the included examples is Dragon Quest IV, and it does pretty much exactly what you're describing - the game starts off interpreting tokens (DQIV chops its text into 6-bit tokens instead of the more common 8-bit [byte] tokens) as hiragana, then when it reads a 0b111100 token, it switches to using katakana until the next time it reads a 0b111100 token while still in the katakana table, at which point it reverts to hiragana. DQIV is actually a little more complicated since it also has a dictionary table (or 4 of them, depending on how you look at it), so if you've only got two tables to worry about and that's the only way the game switches between tables, abcde should be able to handle your game quite easily.

meunierd

  • Jr. Member
  • **
  • Posts: 65
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #3 on: March 11, 2018, 01:08:28 pm »
👏 Perl, that's awesome

abw

  • Sr. Member
  • ****
  • Posts: 328
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #4 on: March 12, 2018, 11:09:24 am »
I know, right?  :D

RedComet

  • Hero Member
  • *****
  • Posts: 3166
    • View Profile
    • Twilight Translations
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #5 on: March 25, 2018, 01:07:37 am »
Wow. This is awesome! Definitely gonna give this a try when I get a chance. :thumbsup:
Twilight Translations - More than just Dragonball Z. :P

abw

  • Sr. Member
  • ****
  • Posts: 328
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #6 on: March 26, 2018, 10:45:05 am »
It's still not as awesome as I'd like it to be (definitely still WIP), but it is already capable of making short work of some fairly complex scripts. Hopefully it'll save people a lot of time!

justin3009

  • Hero Member
  • *****
  • Posts: 1625
  • Welp
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #7 on: April 09, 2018, 03:59:23 pm »
Random question, heck, don't even know if Atlas was able to do this.

There's some games where there's a string repeated multitudes of times and of course, inserting that back in on some games can cause some issues since it goes over the normal bank length, does this program (Or even Atlas), have the ability to NOT write the same exact text string repeatedly, but rather, just group them all as the same pointer to fix any of the issues?
'We have to find some way to incorporate the general civilians in the plot.'

'We'll kill off children in the Juuban district with an infection where they cough up blood and are found hanging themselves from cherry blossom trees.'

Gideon Zhi

  • IRC Staff
  • Hero Member
  • *****
  • Posts: 3520
    • View Profile
    • Aeon Genesis
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #8 on: April 09, 2018, 04:30:34 pm »
Atlas can absolutely do this, though it becomes harder to do if you're using an AUTOWRITE directive. If you're just using standard WRITEs, W16s, or W24s, you can just do something like:

#W16(addr1)
#W16(addr2)
#W16(addr3)
string here<end>

abw

  • Sr. Member
  • ****
  • Posts: 328
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #9 on: April 09, 2018, 06:34:41 pm »
Yup, sticking multiple pointer write commands just before the text they all point to is the way to do it in both Atlas and abcde. One of the many things on my to-do list is to add an option on the Cartographer side to automatically group overlapping strings, only output one copy of them, and add the appropriate Atlas commands for you, both for completely duplicate pointers like in Gideon's example and for pointers that point into the middle of some other string, e.g.

#W16(addr1)
Sentence 1.
#W16(addr2)
Sentence 2.<end>

Pennywise

  • Hero Member
  • *****
  • Posts: 2259
  • I'm curious
    • View Profile
    • Yojimbo's Translations
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #10 on: June 01, 2018, 10:16:13 pm »
So, I'm looking over this finally to properly rip this script and just wanted to make sure I'm on the right track.

For reference, here are the two tables.

http://yojimbo.eludevisibility.org/Stuff/GSJ.tbl
http://yojimbo.eludevisibility.org/Stuff/GSJ2.tbl

Now, $F7 triggers the switch to table GSJ2 and it switches back when it hits $FF. So all I have to do is update the table file and run cartographer?

!F7=[table SWITCH],[GSJ2]:$FF

Am I on the right track?

Guadozoku

  • Sr. Member
  • ****
  • Posts: 257
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #11 on: June 01, 2018, 10:29:02 pm »
I don't need the feature in this because I coded my own dumper/inserter long before I knew about this thing, but for other games that might do the same thing, will abcde allow you to multiply pointers? I know at least one game that divides pointers by 2 and adds a value.

abw

  • Sr. Member
  • ****
  • Posts: 328
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #12 on: June 02, 2018, 09:00:18 pm »
@Pennywise:
Assuming you start from GSJ.tbl, then yeah, it looks like you're on the right track. You'll need to add a table ID line in GSJ2.tbl, e.g.
Code: [Select]
@GSJ2and stick an "@" in the table ID part of the table switch entry in GSJ.tbl, e.g.
Code: [Select]
!F7=[table SWITCH],[@GSJ2]:$FFIncidentally, including the "[table SWITCH]" part means abcde will print out "table SWITCH" whenever it matches F7 in GSJ.tbl; if that's what you want, then great. If not, you could do e.g.
Code: [Select]
!F7=,[@GSJ2]:$FFto get a cleaner-looking dump.

Like I said in the original post, I'm very interested in any feedback you might have - let me know how abcde works for you!


@Guadozoku:
Currently, abcde's pointer-related functionality closely mirrors the functionality offered by Cartographer and Atlas, so unfortunately that level of pointer arithmetic isn't available straight out of the box. That said, extending the interface to support some basic arithmetic using constant values wouldn't be difficult. I'll add it to the to-do list!