News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: abcde: Atlas + Cartographer + A Whole Lot More  (Read 10543 times)

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
abcde: Atlas + Cartographer + A Whole Lot More
« on: March 11, 2018, 09:20:15 am »
abcde (http://www.romhacking.net/utilities/1392/) is a script extraction/insertion utility that aims to take as many of the things that are good about Atlas and Cartographer as possible and make them even better.

Features:
  • Implements all features supported by Cartographer.
  • Implements all features supported by Atlas except for running Atlas extensions.
  • Corrects many features that were broken to varying degrees in Cartographer and Atlas.
  • Uses new table file syntax that allow you to:
    • Work with non-octet text encodings (e.g. 6-bit tokens, Huffman encodings, etc.).
    • Handle mid-string encoding changes (a.k.a. table switching within a single string) with a great deal of flexibility.
  • Adds new extraction commands that allow you to:
    • Dump multiple strings from a single pointer.
    • Re-align the ROM read position when an end token is encountered.
    • Dump strings with internal gaps.
  • Adds new insertion commands that allow you to:
    • Execute arbitrary insertion commands when the insertion point reaches a specified ROM address.
    • Count things and write the counter value to ROM like a pointer.
    • Handle Pascal strings that specify their length in number of tokens rather than number of bytes.
    • Write pointers (embedded or normal) whose value is the distance between the pointer and the text rather than the text's address.
    • Change the current insert position by a relative amount rather than specifying an absolute ROM address.
  • Open source code that you can modify or extend to suit your own needs or preferences.

I'm looking for some testers, new test cases, and feedback on the utility itself and its supporting documentation/examples. Any volunteers? I've got some specific concerns for each area, but I'll listen to whatever you've got to say.
  • Testers: Does this even run on your system, or did I end up making it platform-specific by accident? Was I concentrating too much on testing the complicated stuff that I missed some glaringly obvious bug in the simple stuff?
  • Test cases: Do you have enough experience working with some game that uses a complex text encoding to be able to try using abcde to extract and insert its text? If abcde can't handle it (for reasons other than embedded pointers), I'd love to get a detailed description of the game's text engine so I can make abcde even more powerful.
  • Utility: Better/more user-friendly debug output is on the to-do list, but aside from that, what do you think?
  • Documentation/examples: Do they provide a sufficiently descriptive high level overview? Did I overlook explaining some crucial detail required to get this thing to actually do what you want it to do? I'm a little too close to abcde to be entirely confident that the supporting text doesn't assume too much prior knowledge.

Thanks in advance!

Pennywise

  • Hero Member
  • *****
  • Posts: 2294
  • I'm curious
    • View Profile
    • Yojimbo's Translations
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #1 on: March 11, 2018, 11:16:09 am »
I'll have to use this when I get a chance.

I'm working on a game that uses a byte to swap tables and it uses the same end control code to restore the default. It's basically like a hiragana/katakana switch. Is this something your tool supports?

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #2 on: March 11, 2018, 11:50:34 am »
Absolutely. One of the included examples is Dragon Quest IV, and it does pretty much exactly what you're describing - the game starts off interpreting tokens (DQIV chops its text into 6-bit tokens instead of the more common 8-bit [byte] tokens) as hiragana, then when it reads a 0b111100 token, it switches to using katakana until the next time it reads a 0b111100 token while still in the katakana table, at which point it reverts to hiragana. DQIV is actually a little more complicated since it also has a dictionary table (or 4 of them, depending on how you look at it), so if you've only got two tables to worry about and that's the only way the game switches between tables, abcde should be able to handle your game quite easily.

meunierd

  • RHDN Patreon Supporter!
  • Jr. Member
  • *****
  • Posts: 65
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #3 on: March 11, 2018, 01:08:28 pm »
👏 Perl, that's awesome

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #4 on: March 12, 2018, 11:09:24 am »
I know, right?  :D

RedComet

  • Hero Member
  • *****
  • Posts: 3167
    • View Profile
    • Twilight Translations
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #5 on: March 25, 2018, 01:07:37 am »
Wow. This is awesome! Definitely gonna give this a try when I get a chance. :thumbsup:
Twilight Translations - More than just Dragonball Z. :P

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #6 on: March 26, 2018, 10:45:05 am »
It's still not as awesome as I'd like it to be (definitely still WIP), but it is already capable of making short work of some fairly complex scripts. Hopefully it'll save people a lot of time!

justin3009

  • Hero Member
  • *****
  • Posts: 1637
  • Welp
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #7 on: April 09, 2018, 03:59:23 pm »
Random question, heck, don't even know if Atlas was able to do this.

There's some games where there's a string repeated multitudes of times and of course, inserting that back in on some games can cause some issues since it goes over the normal bank length, does this program (Or even Atlas), have the ability to NOT write the same exact text string repeatedly, but rather, just group them all as the same pointer to fix any of the issues?
'We have to find some way to incorporate the general civilians in the plot.'

'We'll kill off children in the Juuban district with an infection where they cough up blood and are found hanging themselves from cherry blossom trees.'

Gideon Zhi

  • IRC Staff
  • Hero Member
  • *****
  • Posts: 3524
    • View Profile
    • Aeon Genesis
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #8 on: April 09, 2018, 04:30:34 pm »
Atlas can absolutely do this, though it becomes harder to do if you're using an AUTOWRITE directive. If you're just using standard WRITEs, W16s, or W24s, you can just do something like:

#W16(addr1)
#W16(addr2)
#W16(addr3)
string here<end>

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #9 on: April 09, 2018, 06:34:41 pm »
Yup, sticking multiple pointer write commands just before the text they all point to is the way to do it in both Atlas and abcde. One of the many things on my to-do list is to add an option on the Cartographer side to automatically group overlapping strings, only output one copy of them, and add the appropriate Atlas commands for you, both for completely duplicate pointers like in Gideon's example and for pointers that point into the middle of some other string, e.g.

#W16(addr1)
Sentence 1.
#W16(addr2)
Sentence 2.<end>

Pennywise

  • Hero Member
  • *****
  • Posts: 2294
  • I'm curious
    • View Profile
    • Yojimbo's Translations
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #10 on: June 01, 2018, 10:16:13 pm »
So, I'm looking over this finally to properly rip this script and just wanted to make sure I'm on the right track.

For reference, here are the two tables.

http://yojimbo.eludevisibility.org/Stuff/GSJ.tbl
http://yojimbo.eludevisibility.org/Stuff/GSJ2.tbl

Now, $F7 triggers the switch to table GSJ2 and it switches back when it hits $FF. So all I have to do is update the table file and run cartographer?

!F7=[table SWITCH],[GSJ2]:$FF

Am I on the right track?

Guadozoku

  • Sr. Member
  • ****
  • Posts: 259
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #11 on: June 01, 2018, 10:29:02 pm »
I don't need the feature in this because I coded my own dumper/inserter long before I knew about this thing, but for other games that might do the same thing, will abcde allow you to multiply pointers? I know at least one game that divides pointers by 2 and adds a value.

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #12 on: June 02, 2018, 09:00:18 pm »
@Pennywise:
Assuming you start from GSJ.tbl, then yeah, it looks like you're on the right track. You'll need to add a table ID line in GSJ2.tbl, e.g.
Code: [Select]
@GSJ2and stick an "@" in the table ID part of the table switch entry in GSJ.tbl, e.g.
Code: [Select]
!F7=[table SWITCH],[@GSJ2]:$FFIncidentally, including the "[table SWITCH]" part means abcde will print out "table SWITCH" whenever it matches F7 in GSJ.tbl; if that's what you want, then great. If not, you could do e.g.
Code: [Select]
!F7=,[@GSJ2]:$FFto get a cleaner-looking dump.

Like I said in the original post, I'm very interested in any feedback you might have - let me know how abcde works for you!


@Guadozoku:
Currently, abcde's pointer-related functionality closely mirrors the functionality offered by Cartographer and Atlas, so unfortunately that level of pointer arithmetic isn't available straight out of the box. That said, extending the interface to support some basic arithmetic using constant values wouldn't be difficult. I'll add it to the to-do list!

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #13 on: April 07, 2020, 06:48:33 am »
I hope you don't mind if i put a feature request here:

So i have been switching fully from Atlas + Cartographer to abcde.
So far it works like a charm, but it would be neat if you could implement an additioan "general options" function regarding tables.

Because i was using Atlas to reinsert the scripts into the game files i always had to declare the table file that i wanted to use for the script:

Code: [Select]
#VAR(dialogue, TABLE)
#ADDTBL("abcde.tbl", dialogue)
#ACTIVETBL(dialogue)

#VAR(PTR, CUSTOMPOINTER)
#CREATEPTR(PTR, "LINEAR", >>TEXTBLOCK<<, 32)

#VAR(PTRTBL, POINTERTABLE)
#PTRTBL(PTRTBL, >>POINTERSTART<<, 4, PTR)

#JMP(>>TEXTBLOCK<<)
#HDR(>>TEXTBLOCK<<)

abcde has a general function that lets you define a table file to be used by the command that you want to execute, for example this one:

Code: [Select]
perl abcde.pl -t "abcde.tbl" -cm abcde::Atlas "PATH\GL6_SCEN DAT\000000aa.flk" "PATH\00-abcde\Reimport\000000aa.flk STXT Ore Descriptions [TRANSLATED]"

But the problem is: there is currently no function that lets you define the table in the command to "overwrite" what is inside the script file, which gives this error:

can't open TABLE 'PATH\abcde.tbl' at PATH\00-abcde/abcde/Table/Table
.pm line 49, <COMMAND_FILE> line 2. at PATH\00-abcde\Reimport\000000aa.flk STXT Ore Descriptions [TRANSLATED] line 2!

It would be cool if you could implement an additional function to the -t option, something like:

-tp - table primary - overwrites the script table command and forces the command to use this one
-ts - table secondary - if no table is found but defined in the script, it will use this table file

I will probably just delete the table line in the script files so the command is forced to use the -t option, but i still think it would be a neat addition to the -t option.

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #14 on: April 07, 2020, 10:53:42 pm »
I hope you don't mind if i put a feature request here:
Not at all - this seems like the best place for feedback of any kind :thumbsup:!


As for your feature request, it looks like you're just having an issue with relative paths - relative paths for table files that are loaded from the command line get resolved from your current working directory, but relative paths from inside command files get resolved relative to the command file, which is a change from the behaviour of Cartographer/Atlas that is probably not sufficiently highlighted in abcde's documentation.

Just to confirm that diagnosis, what happens if you move abcde.tbl from PATH\ into PATH\00-abcde\Reimport\ or update "000000aa.flk STXT Ore Descriptions [TRANSLATED]" to say either
Code: [Select]
#ADDTBL("..\..\abcde.tbl", dialogue)
or
Code: [Select]
#ADDTBL("PATH\abcde.tbl", dialogue)
instead?

If any of those changes fixes your issue, what I can do is add a command to let the user specify how relative paths in command files are resolved, with options for using your current working directory or the command file as the base path.

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #15 on: April 07, 2020, 11:19:41 pm »
Not at all - this seems like the best place for feedback of any kind :thumbsup:!


As for your feature request, it looks like you're just having an issue with relative paths - relative paths for table files that are loaded from the command line get resolved from your current working directory, but relative paths from inside command files get resolved relative to the command file, which is a change from the behaviour of Cartographer/Atlas that is probably not sufficiently highlighted in abcde's documentation.

Just to confirm that diagnosis, what happens if you move abcde.tbl from PATH\ into PATH\00-abcde\Reimport\ or update "000000aa.flk STXT Ore Descriptions [TRANSLATED]" to say either
Code: [Select]
#ADDTBL("..\..\abcde.tbl", dialogue)
or
Code: [Select]
#ADDTBL("PATH\abcde.tbl", dialogue)
instead?

If any of those changes fixes your issue, what I can do is add a command to let the user specify how relative paths in command files are resolved, with options for using your current working directory or the command file as the base path.

Yes, if i put the "abcde.tbl" file into the "PATH\00-abcde\Reimport\" folder, where all the script files reside in, it works without a problem.
And it probably also does if i define the path in the script command line, but i didn't test that yet.

I just think it would be a good addition to have an option to force the command line table location and ignore what is written in the script table command.

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #16 on: April 13, 2020, 09:17:57 pm »
v0.0.7 is now available!

The latest version adds support for passing multiple insert script/command files at once so you can pass e.g. "dir/*.txt" (on Unix-like systems, anyway; the syntax for achieving the same effect on Windows is very different) to insert all .txt files inside dir instead of having to list each one individually in separate abcde invocations and having to update that list every time you add or remove a file; files are processed sequentially and separately by default, but there's a new command to prevent resetting the internal insertion state between files, which allows you to essentially split one giant file across several smaller files and not have to worry about tracking the current insert position between files.

On the extraction side, abcde's Cartographer interface gained some new output formatting commands; abcde::Cartographer's default behaviour of including the end address of each string in its output file can now be disabled, and a new option exists to allow trimming multiple trailing newlines at the end of a string.

I just think it would be a good addition to have an option to force the command line table location and ignore what is written in the script table command.
After flip-flopping on this for a few days, I've decided to change abcde's default so that all relative paths are resolved relative to your current working directory, which brings them back in line with the behaviour of Atlas and Cartographer, and to add a new option to set the base path for Atlas/Cartographer commands that interact with files (e.g. #ADDTBL, #TABLE, etc.). Now that Atlas can accept multiple input files, one of the options is to use the current input file's directory as the base, which is useful if your input files are spread across multiple directories and using the same base for all files wouldn't work well.


And since I wasn't keeping this thread up to date as new versions came out, here's an abbreviated list of upgrades since v0.0.1:
  • To make getting started easier, abcde now comes pre-bundled with its dependent modules, which otherwise had to be separately installed from CPAN.
  • Multiple tables can be contained in a single file.
  • abcde's Atlas/Cartographer interfaces now have some table ID support.
  • Default NFD/NFC Unicode normalization (so you can have e.g. table entries for base characters like "カ" (U+30AB) and combining marks like "゙" (U+3099) and get combined character output like "ガ" (U+30AC).
  • Negative hexadecimal values can now be specified using -$hex or $-hex (Atlas and Cartographer disagreed over which form was correct).
  • Loading table files on the command line is no longer necessary as they can now all be loaded in the command files.
  • You now have the ability to specify whether you want to match bytes or bits when extracting non-text data.
  • abcde's Cartographer now outputs more Atlas commands for you.
  • Cartographer would output #W8 commands for Atlas, but Atlas had no #W8 command, so abcde added one.
  • When extracting text, you can now output both a commented and non-commented version of the text.
  • #SCRIPT STOP can now be used with all #METHOD values, not just RAW, which is helpful when you have pointers you want to include in the extraction output but the conditions that determine the end of the strings are difficult to express via table files.
  • When multiple pointers point to the same string, Cartographer would output that string multiple times; the new #STRINGS END AT NEXT POINTER command can be used to cut down on that duplicate output, particularly when used in combination with the new #SORT OUTPUT BY STRING ADDRESS.
  • #TYPE: FIXED_STRING && FIXED_LINE now possibly does something useful.

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #17 on: April 20, 2020, 02:19:55 am »
Hi abw,

love your new version! Makes dumping text so, so much easier.

I have a question for a feature request:

Spoiler:
//POINTER #11 @ $104C - STRING #11 @ $1581
#W32($104C)
//[CHARCB03][V0000]
//971_000どのふとん犬の足の速さが一番か![NLINE]
//それを決する舞台、[NLINE]
//ドッグレースコンテストを開催いたします![END-FE]
[CHARCB03][V0000]
971_000どのふとん犬の足の速さが一番か![NLINE]
それを決する舞台、[NLINE]
ドッグレースコンテストを開催いたします![END-FE]





//POINTER #12 @ $1050 - STRING #12 @ $15F1
#W32($1050)
//[CHARCB03][V0100]
//971_001スタートからゴールまで[NLINE]
//他のふとん犬よりも速く辿り着く。[NLINE]
//シンプル故に、能力の優劣がつこうというもの。[NWIN]
//
//[CHARCB03][V1F00]
//971_031ブリーダーの下目掛けて、[NLINE]
//まっしぐらに走って貰いましょう![END-FE]
[CHARCB03][V0100]
971_001スタートからゴールまで[NLINE]
他のふとん犬よりも速く辿り着く。[NLINE]
シンプル故に、能力の優劣がつこうというもの。[NWIN]

[CHARCB03][V1F00]
971_031ブリーダーの下目掛けて、[NLINE]
まっしぐらに走って貰いましょう![END-FE]

Here you can see that i configured my table in a way that after a control code "[V1F00]" it will do a newline.
Now, in this particular script file there are other control codes used:

971_000
971_031

Do you think it is possible to implement a feature that lets you remove a line before the text, and creates one after it?
A reverse \n?
So in the end it would look like this:

Spoiler:
//POINTER #11 @ $104C - STRING #11 @ $1581
#W32($104C)
//[CHARCB03][V0000]971_000
//どのふとん犬の足の速さが一番か![NLINE]
//それを決する舞台、[NLINE]
//ドッグレースコンテストを開催いたします![END-FE]
[CHARCB03][V0000]971_000
どのふとん犬の足の速さが一番か![NLINE]
それを決する舞台、[NLINE]
ドッグレースコンテストを開催いたします![END-FE]





//POINTER #12 @ $1050 - STRING #12 @ $15F1
#W32($1050)
//[CHARCB03][V0100]971_001
//スタートからゴールまで[NLINE]
//他のふとん犬よりも速く辿り着く。[NLINE]
//シンプル故に、能力の優劣がつこうというもの。[NWIN]
//
//[CHARCB03][V1F00]971_031
//ブリーダーの下目掛けて、[NLINE]
//まっしぐらに走って貰いましょう![END-FE]
[CHARCB03][V0100]971_001
スタートからゴールまで[NLINE]
他のふとん犬よりも速く辿り着く。[NLINE]
シンプル故に、能力の優劣がつこうというもの。[NWIN]

[CHARCB03][V1F00]971_031
ブリーダーの下目掛けて、[NLINE]
まっしぐらに走って貰いましょう![END-FE]

So after writing [V1F00] with the new line, it will remove that new line and create a new one after 971_031.

Something like "\rn[971_031]\n"?

abw

  • Sr. Member
  • ****
  • Posts: 427
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #18 on: April 21, 2020, 09:17:11 am »
Hi abw,

love your new version! Makes dumping text so, so much easier.
Glad to hear you're enjoying it!

I have a question for a feature request:
Given a choice between a) doing one thing you don't want then doing a second thing to counteract the first thing and b) just doing the thing you want in the first place, I'm strongly inclined to choose option b). It's not so bad for extraction, but when it comes to insertion, tokens being able to modify previous tokens is absolutely not a path I want to start down!

For this case, is there any issue with specifying the behaviour you want in your table file? E.g. instead of just this:
Code: [Select]
hex1=[V1F00]\n
you can say which combinations of tokens should not have a newline like this:
Code: [Select]
hex1=[V1F00]\n
hex2=[V1F00]971_000\n
hex3=[V1F00]971_031\n

If you have a large number of tokens like that, it might be a little annoying, but some light scripting or some decent search-and-replace in a text editor should let you deal with any number of combinations in probably 5 to 10 minutes.

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #19 on: April 22, 2020, 03:27:37 am »
Hi abw,

thank you for your reply!

Hmmm, i thought about doing this

Code: [Select]
hex1=[V1F00]\n
hex2=[V1F00]971_000\n
hex3=[V1F00]971_031\n

before posting in your thread. But the problem here is that there are a shitton of possible combinations.
"[V1F00]" has over 500 variations and so does "971_000".
So, a combination of them would require more than 5000+? lines in the table file that i would have to put in.
I think i will create an additional table where only the "971_031..." have newline statements.
I believe its the easiest way for now.