Romhacking.net

Romhacking => Personal Projects => Topic started by: abw on March 11, 2018, 09:20:15 am

Title: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on March 11, 2018, 09:20:15 am
abcde (http://www.romhacking.net/utilities/1392/) (http://www.romhacking.net/utilities/1392/) is a script extraction/insertion utility that aims to take as many of the things that are good about Atlas and Cartographer as possible and make them even better.

Features:

I'm looking for some testers, new test cases, and feedback on the utility itself and its supporting documentation/examples. Any volunteers? I've got some specific concerns for each area, but I'll listen to whatever you've got to say.

Thanks in advance!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Pennywise on March 11, 2018, 11:16:09 am
I'll have to use this when I get a chance.

I'm working on a game that uses a byte to swap tables and it uses the same end control code to restore the default. It's basically like a hiragana/katakana switch. Is this something your tool supports?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on March 11, 2018, 11:50:34 am
Absolutely. One of the included examples is Dragon Quest IV, and it does pretty much exactly what you're describing - the game starts off interpreting tokens (DQIV chops its text into 6-bit tokens instead of the more common 8-bit [byte] tokens) as hiragana, then when it reads a 0b111100 token, it switches to using katakana until the next time it reads a 0b111100 token while still in the katakana table, at which point it reverts to hiragana. DQIV is actually a little more complicated since it also has a dictionary table (or 4 of them, depending on how you look at it), so if you've only got two tables to worry about and that's the only way the game switches between tables, abcde should be able to handle your game quite easily.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: meunierd on March 11, 2018, 01:08:28 pm
👏 Perl, that's awesome
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on March 12, 2018, 11:09:24 am
I know, right?  :D
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: RedComet on March 25, 2018, 01:07:37 am
Wow. This is awesome! Definitely gonna give this a try when I get a chance. :thumbsup:
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on March 26, 2018, 10:45:05 am
It's still not as awesome as I'd like it to be (definitely still WIP), but it is already capable of making short work of some fairly complex scripts. Hopefully it'll save people a lot of time!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: justin3009 on April 09, 2018, 03:59:23 pm
Random question, heck, don't even know if Atlas was able to do this.

There's some games where there's a string repeated multitudes of times and of course, inserting that back in on some games can cause some issues since it goes over the normal bank length, does this program (Or even Atlas), have the ability to NOT write the same exact text string repeatedly, but rather, just group them all as the same pointer to fix any of the issues?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Gideon Zhi on April 09, 2018, 04:30:34 pm
Atlas can absolutely do this, though it becomes harder to do if you're using an AUTOWRITE directive. If you're just using standard WRITEs, W16s, or W24s, you can just do something like:

#W16(addr1)
#W16(addr2)
#W16(addr3)
string here<end>
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on April 09, 2018, 06:34:41 pm
Yup, sticking multiple pointer write commands just before the text they all point to is the way to do it in both Atlas and abcde. One of the many things on my to-do list is to add an option on the Cartographer side to automatically group overlapping strings, only output one copy of them, and add the appropriate Atlas commands for you, both for completely duplicate pointers like in Gideon's example and for pointers that point into the middle of some other string, e.g.

#W16(addr1)
Sentence 1.
#W16(addr2)
Sentence 2.<end>
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Pennywise on June 01, 2018, 10:16:13 pm
So, I'm looking over this finally to properly rip this script and just wanted to make sure I'm on the right track.

For reference, here are the two tables.

http://yojimbo.eludevisibility.org/Stuff/GSJ.tbl
http://yojimbo.eludevisibility.org/Stuff/GSJ2.tbl

Now, $F7 triggers the switch to table GSJ2 and it switches back when it hits $FF. So all I have to do is update the table file and run cartographer?

!F7=[table SWITCH],[GSJ2]:$FF

Am I on the right track?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Guadozoku on June 01, 2018, 10:29:02 pm
I don't need the feature in this because I coded my own dumper/inserter long before I knew about this thing, but for other games that might do the same thing, will abcde allow you to multiply pointers? I know at least one game that divides pointers by 2 and adds a value.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on June 02, 2018, 09:00:18 pm
@Pennywise:
Assuming you start from GSJ.tbl, then yeah, it looks like you're on the right track. You'll need to add a table ID line in GSJ2.tbl, e.g.
Code: [Select]
@GSJ2and stick an "@" in the table ID part of the table switch entry in GSJ.tbl, e.g.
Code: [Select]
!F7=[table SWITCH],[@GSJ2]:$FFIncidentally, including the "[table SWITCH]" part means abcde will print out "table SWITCH" whenever it matches F7 in GSJ.tbl; if that's what you want, then great. If not, you could do e.g.
Code: [Select]
!F7=,[@GSJ2]:$FFto get a cleaner-looking dump.

Like I said in the original post, I'm very interested in any feedback you might have - let me know how abcde works for you!


@Guadozoku:
Currently, abcde's pointer-related functionality closely mirrors the functionality offered by Cartographer and Atlas, so unfortunately that level of pointer arithmetic isn't available straight out of the box. That said, extending the interface to support some basic arithmetic using constant values wouldn't be difficult. I'll add it to the to-do list!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on April 07, 2020, 06:48:33 am
I hope you don't mind if i put a feature request here:

So i have been switching fully from Atlas + Cartographer to abcde.
So far it works like a charm, but it would be neat if you could implement an additioan "general options" function regarding tables.

Because i was using Atlas to reinsert the scripts into the game files i always had to declare the table file that i wanted to use for the script:

Code: [Select]
#VAR(dialogue, TABLE)
#ADDTBL("abcde.tbl", dialogue)
#ACTIVETBL(dialogue)

#VAR(PTR, CUSTOMPOINTER)
#CREATEPTR(PTR, "LINEAR", >>TEXTBLOCK<<, 32)

#VAR(PTRTBL, POINTERTABLE)
#PTRTBL(PTRTBL, >>POINTERSTART<<, 4, PTR)

#JMP(>>TEXTBLOCK<<)
#HDR(>>TEXTBLOCK<<)

abcde has a general function that lets you define a table file to be used by the command that you want to execute, for example this one:

Code: [Select]
perl abcde.pl -t "abcde.tbl" -cm abcde::Atlas "PATH\GL6_SCEN DAT\000000aa.flk" "PATH\00-abcde\Reimport\000000aa.flk STXT Ore Descriptions [TRANSLATED]"

But the problem is: there is currently no function that lets you define the table in the command to "overwrite" what is inside the script file, which gives this error:

can't open TABLE 'PATH\abcde.tbl' at PATH\00-abcde/abcde/Table/Table
.pm line 49, <COMMAND_FILE> line 2. at PATH\00-abcde\Reimport\000000aa.flk STXT Ore Descriptions [TRANSLATED] line 2!

It would be cool if you could implement an additional function to the -t option, something like:

-tp - table primary - overwrites the script table command and forces the command to use this one
-ts - table secondary - if no table is found but defined in the script, it will use this table file

I will probably just delete the table line in the script files so the command is forced to use the -t option, but i still think it would be a neat addition to the -t option.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on April 07, 2020, 10:53:42 pm
I hope you don't mind if i put a feature request here:
Not at all - this seems like the best place for feedback of any kind :thumbsup:!


As for your feature request, it looks like you're just having an issue with relative paths - relative paths for table files that are loaded from the command line get resolved from your current working directory, but relative paths from inside command files get resolved relative to the command file, which is a change from the behaviour of Cartographer/Atlas that is probably not sufficiently highlighted in abcde's documentation.

Just to confirm that diagnosis, what happens if you move abcde.tbl from PATH\ into PATH\00-abcde\Reimport\ or update "000000aa.flk STXT Ore Descriptions [TRANSLATED]" to say either
Code: [Select]
#ADDTBL("..\..\abcde.tbl", dialogue)
or
Code: [Select]
#ADDTBL("PATH\abcde.tbl", dialogue)
instead?

If any of those changes fixes your issue, what I can do is add a command to let the user specify how relative paths in command files are resolved, with options for using your current working directory or the command file as the base path.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on April 07, 2020, 11:19:41 pm
Not at all - this seems like the best place for feedback of any kind :thumbsup:!


As for your feature request, it looks like you're just having an issue with relative paths - relative paths for table files that are loaded from the command line get resolved from your current working directory, but relative paths from inside command files get resolved relative to the command file, which is a change from the behaviour of Cartographer/Atlas that is probably not sufficiently highlighted in abcde's documentation.

Just to confirm that diagnosis, what happens if you move abcde.tbl from PATH\ into PATH\00-abcde\Reimport\ or update "000000aa.flk STXT Ore Descriptions [TRANSLATED]" to say either
Code: [Select]
#ADDTBL("..\..\abcde.tbl", dialogue)
or
Code: [Select]
#ADDTBL("PATH\abcde.tbl", dialogue)
instead?

If any of those changes fixes your issue, what I can do is add a command to let the user specify how relative paths in command files are resolved, with options for using your current working directory or the command file as the base path.

Yes, if i put the "abcde.tbl" file into the "PATH\00-abcde\Reimport\" folder, where all the script files reside in, it works without a problem.
And it probably also does if i define the path in the script command line, but i didn't test that yet.

I just think it would be a good addition to have an option to force the command line table location and ignore what is written in the script table command.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on April 13, 2020, 09:17:57 pm
v0.0.7 is now available!

The latest version adds support for passing multiple insert script/command files at once so you can pass e.g. "dir/*.txt" (on Unix-like systems, anyway; the syntax for achieving the same effect on Windows is very different) to insert all .txt files inside dir instead of having to list each one individually in separate abcde invocations and having to update that list every time you add or remove a file; files are processed sequentially and separately by default, but there's a new command to prevent resetting the internal insertion state between files, which allows you to essentially split one giant file across several smaller files and not have to worry about tracking the current insert position between files.

On the extraction side, abcde's Cartographer interface gained some new output formatting commands; abcde::Cartographer's default behaviour of including the end address of each string in its output file can now be disabled, and a new option exists to allow trimming multiple trailing newlines at the end of a string.

I just think it would be a good addition to have an option to force the command line table location and ignore what is written in the script table command.
After flip-flopping on this for a few days, I've decided to change abcde's default so that all relative paths are resolved relative to your current working directory, which brings them back in line with the behaviour of Atlas and Cartographer, and to add a new option to set the base path for Atlas/Cartographer commands that interact with files (e.g. #ADDTBL, #TABLE, etc.). Now that Atlas can accept multiple input files, one of the options is to use the current input file's directory as the base, which is useful if your input files are spread across multiple directories and using the same base for all files wouldn't work well.


And since I wasn't keeping this thread up to date as new versions came out, here's an abbreviated list of upgrades since v0.0.1:
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on April 20, 2020, 02:19:55 am
Hi abw,

love your new version! Makes dumping text so, so much easier.

I have a question for a feature request:

Spoiler:
//POINTER #11 @ $104C - STRING #11 @ $1581
#W32($104C)
//[CHARCB03][V0000]
//971_000どのふとん犬の足の速さが一番か![NLINE]
//それを決する舞台、[NLINE]
//ドッグレースコンテストを開催いたします![END-FE]
[CHARCB03][V0000]
971_000どのふとん犬の足の速さが一番か![NLINE]
それを決する舞台、[NLINE]
ドッグレースコンテストを開催いたします![END-FE]





//POINTER #12 @ $1050 - STRING #12 @ $15F1
#W32($1050)
//[CHARCB03][V0100]
//971_001スタートからゴールまで[NLINE]
//他のふとん犬よりも速く辿り着く。[NLINE]
//シンプル故に、能力の優劣がつこうというもの。[NWIN]
//
//[CHARCB03][V1F00]
//971_031ブリーダーの下目掛けて、[NLINE]
//まっしぐらに走って貰いましょう![END-FE]
[CHARCB03][V0100]
971_001スタートからゴールまで[NLINE]
他のふとん犬よりも速く辿り着く。[NLINE]
シンプル故に、能力の優劣がつこうというもの。[NWIN]

[CHARCB03][V1F00]
971_031ブリーダーの下目掛けて、[NLINE]
まっしぐらに走って貰いましょう![END-FE]

Here you can see that i configured my table in a way that after a control code "[V1F00]" it will do a newline.
Now, in this particular script file there are other control codes used:

971_000
971_031

Do you think it is possible to implement a feature that lets you remove a line before the text, and creates one after it?
A reverse \n?
So in the end it would look like this:

Spoiler:
//POINTER #11 @ $104C - STRING #11 @ $1581
#W32($104C)
//[CHARCB03][V0000]971_000
//どのふとん犬の足の速さが一番か![NLINE]
//それを決する舞台、[NLINE]
//ドッグレースコンテストを開催いたします![END-FE]
[CHARCB03][V0000]971_000
どのふとん犬の足の速さが一番か![NLINE]
それを決する舞台、[NLINE]
ドッグレースコンテストを開催いたします![END-FE]





//POINTER #12 @ $1050 - STRING #12 @ $15F1
#W32($1050)
//[CHARCB03][V0100]971_001
//スタートからゴールまで[NLINE]
//他のふとん犬よりも速く辿り着く。[NLINE]
//シンプル故に、能力の優劣がつこうというもの。[NWIN]
//
//[CHARCB03][V1F00]971_031
//ブリーダーの下目掛けて、[NLINE]
//まっしぐらに走って貰いましょう![END-FE]
[CHARCB03][V0100]971_001
スタートからゴールまで[NLINE]
他のふとん犬よりも速く辿り着く。[NLINE]
シンプル故に、能力の優劣がつこうというもの。[NWIN]

[CHARCB03][V1F00]971_031
ブリーダーの下目掛けて、[NLINE]
まっしぐらに走って貰いましょう![END-FE]

So after writing [V1F00] with the new line, it will remove that new line and create a new one after 971_031.

Something like "\rn[971_031]\n"?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on April 21, 2020, 09:17:11 am
Hi abw,

love your new version! Makes dumping text so, so much easier.
Glad to hear you're enjoying it!

I have a question for a feature request:
Given a choice between a) doing one thing you don't want then doing a second thing to counteract the first thing and b) just doing the thing you want in the first place, I'm strongly inclined to choose option b). It's not so bad for extraction, but when it comes to insertion, tokens being able to modify previous tokens is absolutely not a path I want to start down!

For this case, is there any issue with specifying the behaviour you want in your table file? E.g. instead of just this:
Code: [Select]
hex1=[V1F00]\n
you can say which combinations of tokens should not have a newline like this:
Code: [Select]
hex1=[V1F00]\n
hex2=[V1F00]971_000\n
hex3=[V1F00]971_031\n

If you have a large number of tokens like that, it might be a little annoying, but some light scripting or some decent search-and-replace in a text editor should let you deal with any number of combinations in probably 5 to 10 minutes.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on April 22, 2020, 03:27:37 am
Hi abw,

thank you for your reply!

Hmmm, i thought about doing this

Code: [Select]
hex1=[V1F00]\n
hex2=[V1F00]971_000\n
hex3=[V1F00]971_031\n

before posting in your thread. But the problem here is that there are a shitton of possible combinations.
"[V1F00]" has over 500 variations and so does "971_000".
So, a combination of them would require more than 5000+? lines in the table file that i would have to put in.
I think i will create an additional table where only the "971_031..." have newline statements.
I believe its the easiest way for now.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on April 22, 2020, 09:07:22 pm
Hmm, yeah, 500+ * 500+ = 250000+, which is definitely a lot. Assuming your single table file currently looks like this (1024 entries + whatever else you've got):
Code: [Select]
1F00=[V1F00]\n
1F01=[V1F01]\n
...
20FF=[V20FF]\n

971000=971_000
971001=971_001
...
9711FF=971_1FF
you could make use of table switching to refactor the existing entries into multiple tables and then handle the combinations like this (805 entries over 5 tables + whatever else you've got):
Spoiler:
Code: [Select]
!1F=<[V1F>,@V:1
!20=<[V20>,@V:1

!%0001111100000000100101110001=<[V1F00]971_>,@nybble_01:1,@nybble:1,@nybble_newline:1
!%0001111100000001100101110001=<[V1F01]971_>,@nybble_01:1,@nybble:1,@nybble_newline:1
...
!%0010000011111111100101110001=<[V20FF]971_>,@nybble_01:1,@nybble:1,@nybble_newline:1

!%100101110001=<971_>,@nybble_01:1,@nybble:1
Code: [Select]
@V
00=00]\n
01=01]\n
...
FF=FF]\n
Code: [Select]
@nybble_01
# only matches 0 and 1
%0000=0
%0001=1
Code: [Select]
@nybble
%0000=0
%0001=1
...
%1111=F
Code: [Select]
@nybble_newline
%0000=0\n
%0001=1\n
...
%1111=F\n
(And wow that would be so much more readable if the left-hand side of table entries allowed nybbles instead of full bytes; I think I'll look into adding that in v0.0.8!)

This approach is obviously not perfect and has serious scaling issues since your table file needs to individually list all of the [V****] tokens before getting to the 971_* tokens, but 800+ entries is definitely better than 250000+ entries.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on April 28, 2020, 09:34:42 am
Hi abw,

thank you for your reply.
Hm, i see. I read about the function of using multiple table files before.
This is not a script file which i have to fix instantly, so i will take a look at it another time and see how it goes.
Thank you for your help!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: _Ombra_ on May 05, 2020, 10:46:36 pm
Hey abw,

this looks great but i have a few questions. I'm trying to use it and i'm stumbling upon some stuff that i don't have with Cartographer for example.

If the tbl files contain a character assigned to < or > i get an error straight away. If i comment that out and i have letters i added (for example accents) it just says they aren't in the unicode table.

It that normal behavior? is it fixable?

Thanks
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on May 06, 2020, 09:34:55 pm
If the tbl files contain a character assigned to < or > i get an error straight away.
Yup, that's expected. You'll have to update your table files to use something else like [], {}, or ‹› (U+2039/U+203A) instead.

If i comment that out and i have letters i added (for example accents) it just says they aren't in the unicode table.
My guess here is that your table files aren't encoded as UTF-8; simply saving them as UTF-8 instead of something else like ISO-8859-1 or Windows-1252 should solve that problem. Note that UTF-8 is expected for all text input, so if you have other pre-existing files in some other encoding, you'll need to update their encoding in order to have them work properly.

May 13, 2020, 07:28:43 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
v0.0.8 is up. This one's mostly a bugfix release for some regression errors in v0.0.7 and some funky #AUTOWRITE behaviour I recently noticed that was also present in Atlas. The only real improvement in this version is the ability to use hex table entries with odd lengths, so e.g. "248=foo" will match 12 bits of data instead of being flagged as an error.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: nejimakipiyo on July 24, 2020, 08:57:28 am
Hello! This is my first time using this software. Attempting to do an extraction of all of the lists in Dragon Warrior Monsters (GBC). Chicken Knife was helping me get the extraction set up but we ran into a problem and he suggested I reach out to you after he couldn't figure it out. So here I am! And here's my problem:

(https://i.imgur.com/wPHgS3E.png)

When I try to run this command in the command line, it tells me it can't find Table.pm.
But that file is right here!

(https://i.imgur.com/n5xIIrP.png)

If it helps, here's the directory where my hacking stuff is organized.

(https://i.imgur.com/WVUoSWx.png)


This is all 100% new to me, so I'm at a total loss for how to fix this problem. Any help would be appreciated!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on July 24, 2020, 08:39:45 pm
Hmm, what's in that first "abcde" directory? It sort of looks like you've got an extra level of directory in there that shouldn't be... did you maybe originally have abcde.pl located under "DQM Hack\abcde\" and then move just that one file to "DQM Hack\" while leaving all of its supporting files under "DQM Hack\abcde\abcde\"?

Whatever ended up happening, you'll need to make it so that abcde.pl is located 2 levels up from Table.pm just like it is when unzipping the zip file normally, so you'll have e.g. "DQM Hack\abcde.pl" and "DQM Hack\abcde\Table\Table.pm".

Personally I'd probably put abcde in its own location e.g. C:\Users\wings\abcde\ and then call it from DQM Hack as ..\abcde\abcde.pl, but where you want to put it is up to you; just make sure not to separate individual files from the rest of their friends ;).
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: nejimakipiyo on July 24, 2020, 10:51:54 pm
Thank you! That certainly helped me get one step forward, I think. I downloaded abcde back in January and haven't made a serious attempt to use it until now, so I couldn't begin to tell you what happened.

Now it's asking something else I'm not sure about. A Hash::Priority Queue Module?
Chicken Knife doesn't know this one either!

----

EDIT: We figured that one out, I had to shuffle my directories around more. This is where we're stumped now:

(https://i.imgur.com/rt8a0Hm.png)

It doesn't seem to like my table file for some reason. It's accessing the table file, since it can see the first line. The tablefile.tbl is written that way in my cartographer text as well as in the command I'm putting into the command line. I'm lost.  :banghead:
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on July 25, 2020, 10:23:34 am
EDIT: We figured that one out, I had to shuffle my directories around more.
Yup, leaving the original directory structure intact is definitely the way to go!

This is where we're stumped now:
It looks like you're trying to pass a table file where a Cartographer command file is expected. What you want is probably something like
Code: [Select]
perl abcde.pl -cm abcde::Cartographer "Dragon Warrior Monsters.gbc" dwm_Cartographer_commands.txt dwm_Cartographer_out -s
with an appropriate dwm_Cartographer_commands.txt file that describes where in the ROM any text and pointers that you want to extract is located and what table file to use to translate the binary into text. Cartographer comes with example files for Final Fantasy (NES), and I'm assuming you already know where the text and pointers are located for Dragon Warrior Monsters since you've been editing it already, so plugging in the appropriate values shouldn't be too hard.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: nejimakipiyo on July 25, 2020, 04:53:42 pm
Thank you for your help again! I fixed the wrong command I was using in the command line with the one you wrote in your previous reply instead, and managed to get an output file created...

(https://i.imgur.com/g8qnDf4.png)

...except that the output file is blank. Completely empty. I tried it both with and without the "bin2text" part of the command from my previous attempts, to no result. Chicken Knife said it might have something to do with parts of the Cartographer sections that he doesn't understand, and I couldn't find anything elucidating in the documentation. Hope it's ok that I sent you a PM with a link to my files.  :angel:
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on July 25, 2020, 11:21:22 pm
It looks like your table file doesn't have any end tokens defined. Without some way of knowing when to stop, the extraction process starts from the first pointer and just keeps on running until it reaches the end of the ROM, then it starts from the second pointer and keeps on going until it reaches the end of the ROM. Lather, rinse, repeat until the final pointer, which again dumps everything until the end of the ROM. With over 1500 pointers starting about 1 MB from the end of the ROM, that's going to generate well over 1.5 GB of text; the disk I/O time alone for a file that large will be substantial, to say nothing of abcde's own processing time.

Fortunately the solution is simple: just add a "/" before your "F0=[F0]\n" to make it an end token "/F0=[F0]\n" :).
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: nejimakipiyo on July 26, 2020, 08:00:28 pm
That worked! Thank you so much for all your help!  ;D
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Chicken Knife on July 27, 2020, 08:15:09 pm
I'll give the next response since nejimakipiyo can't bump the thread.

We produced the big list dump, which is great, but when we reinsert it as an atlas file, we are now 233 characters over. nejimakipiyo got that result first, then I set up the file independently and got the same result as well. My other attempts to turn a cartographer out file into an atlas file have always been successful and didn't result in any kind of increase in bytes. I don't know how that would even be possible.

Anyway, nejimakipiyo is sending you a link to the most updated files. We appreciate your help as always!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on July 27, 2020, 11:12:00 pm
That's because some strings are pointed at by multiple pointers, resulting in those strings being extracted multiple times, and by default they appear multiple times in Cartographer's output, so unless you do something to remove duplicate strings, you'll be inserting multiple copies of the duplicate strings, which is where your size increase is coming from.

abcde adds a "#STRINGS END AT NEXT POINTER: Yes" option that will automatically remove duplicate strings for you, though I think I'll expand its documentation to make that side effect more obvious in the next release. In your case, it looks like the pointers to all the duplicate strings are adjacent, so #STRINGS END AT NEXT POINTER is probably enough, but in cases where the pointer values are not in ascending order, you might also want to set "#SORT OUTPUT BY STRING ADDRESS: Yes".
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on August 28, 2020, 07:07:19 pm
v0.0.9 is now up. The most noteworthy part of this version is the introduction of the ability to perform more-or-less arbitrary user-supplied arithmetic calculations during insertion and write the result to ROM in a variety of ways; among other things, this makes it possible to insert all of the weird pointer formats that I've seen and a very large variety of formats that I haven't seen.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: meunierd on August 29, 2020, 09:01:33 pm
Is there a git copy of this online somewhere? It's an amazing tool!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on August 29, 2020, 09:57:31 pm
No, just my private repo filled with all kinds of embarrassing commit messages :P. I'm glad you're finding it useful!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: FlashPV on September 05, 2020, 04:37:00 am
Hi, looks like this tool may be what I've been looking for. Phantasy Star II uses some relative pointer. Each pointer is the sum of all previous pointers. Is there any way to extract and insert the script using abcde?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on September 05, 2020, 11:40:37 am
So, there's a base pointer somewhere and then the value for each of the individual pointers is the running total of the lengths of all the previous strings or the distance from the base pointer? E.g. if each string happens to be $10 long, you'd have pointer values like $00, $10, $20, etc.?

If so, then yes, you could extract that pretty normally with "#METHOD: POINTER_RELATIVE", setting "#BASE POINTER:" to whatever the address of the first string is. You'd then get pointer values of <first string address> + $00, <first string address> + $10, <first string address> + $20, etc. lining up with your string addresses.

For insertion, you could use the new CALCVAR features, filling in <> variables with your desired values:
Code: [Select]
// define some variables
#VAR(base, CALCVAR)
#VAR(length, CALCVAR)

// jump to insertion point, set an upper bound on insertion address
#JMP(<script start>, <script stop>)
// save current address into "base" variable
#CALC(base, "!CURRENT_ADDRESS")
// calculate "length" = current address - "base" (= $00 since they're currently the same value)
#CALC(length, "!CURRENT_ADDRESS - !base")
// use the current value of "length" ($00) as the value for the first pointer
#W8(length, <pointer 0 address>)
String #1 here.[end]

// calculate "length" = current address - "base" (= $10 since string 1 was $10 long)
#CALC(length, "!CURRENT_ADDRESS - !base")
// use the current value of "length" ($10) as the value for the second pointer
#W8(length, <pointer 1 address>)
String #12 here.[end]

// calculate "length" = current address - "base" (= $20 since both strings 1 and 2 were $10 long)
#CALC(length, "!CURRENT_ADDRESS - !base")
// use the current value of "length" ($20) as the value for the third pointer
#W8(length, <pointer 2 address>)
String #3 here.[end]

// etc.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: FlashPV on September 05, 2020, 01:37:27 pm
Hum sorry, I didn't explain correctly.In fact each pointer indicate the distance from the previous pointer. If the first string is $07 long and the second is $10 long then the second pointer will be 07 and the third 10.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on September 05, 2020, 03:05:46 pm
Ah, so then if each string happens to be $10 long, you'd have pointer values like $00, $10, $10, etc.? Yup, that is a slightly different setup.

Unfortunately, I haven't yet extended Cartographer's interface enough to make extracting text from pointer values like that easy. You can still do it by using a separate block for each pointer, but you'll have to calculate each string's start address yourself and use that value in the corresponding Cartographer block. So it is possible, but pretty annoying.

Inserting, on the other hand, is much easier since you can now control the pointer value calculations in your insert script:
Code: [Select]
// define a variable
#VAR(prev_pointer, CALCVAR)

// jump to insertion point, set an upper bound on insertion address
#JMP(<script start>, <script stop>)
// set "prev_pointer" variable to 0
#CALC(prev_pointer, "0")
// use the current value of "prev_pointer" ($00) as the value for the first pointer
#W8(prev_pointer, <pointer 0 address>)
String #1 here.[end]

// calculate "prev_pointer" = current address - "prev_pointer" (= $10 since string 1 was $10 long)
#CALC(prev_pointer, "!CURRENT_ADDRESS - !prev_pointer")
// use the current value of "prev_pointer" ($10) as the value for the second pointer
#W8(prev_pointer, <pointer 1 address>)
String #12 here.[end]

// calculate "prev_pointer" = current address - "prev_pointer" (= $10 since string 2 was $10 long)
#CALC(prev_pointer, "!CURRENT_ADDRESS - !prev_pointer")
// use the current value of "prev_pointer" ($10) as the value for the third pointer
#W8(prev_pointer, <pointer 2 address>)
String #3 here.[end]

// etc.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on October 03, 2020, 04:03:43 am
Hi abw,

as always, thank you very much for this awesome tool.
I have been assisting someone with figuring out how the pointer table for the Growlanser 1 script works.
Didn't take long and i managed to figure out where its stored and make a Cartographer script to dump its contents.
But it seems like the end token \n issue still persists:

Spoiler:
(https://i.imgur.com/Cjd4tJJ.png)

Spoiler:
(https://i.imgur.com/Xtzn8R6.png)

Spoiler:
(https://i.imgur.com/JaDtN6d.png)

Spoiler:
(https://i.imgur.com/vLcX0Co.png)

I am using the latest abcde version. Do you know what might cause this?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on October 07, 2020, 07:16:41 pm
Can you remind me again what output you were expecting in this case? It looks like your table file says to add 1 newline after each [END-FF] and that's what you're getting; if you want 2 newlines instead, FFFF=[END-FF]\n\n should do the trick.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on October 08, 2020, 02:24:45 am
Hi abw, i apologize for the crappy table picture.
1 line is supported, multiple lines are not:

Spoiler:
(https://i.imgur.com/MlDXPb9.png)

Spoiler:
(https://i.imgur.com/zz3nyuH.png)

I hope this makes it more obvious.

Another thing:
Inside Growlanser 1 ELF we came across a very strange pointer table.
The table counts 2 bytes as 1, and we looked through the Cartographer and Atlas docs but couldn't find anything that might support these kind of pointers.
Our super MIPS hacker decided to hack this function into Atlas.pm and Cartographer.pm.
It seems to work so far and we would like to give you the code in case you want to implement it into abcde:

Code: [Select]
Cartographer:
#POINTER MULTIPLIER:        $XX

Atlas:
#MULTIPLIER($xx)
It gets added automatically with the new cartographer if #POINTER MULTIPLIER is provided

https://bitbucket.org/Risae/growlanser-6-english-translation/src/master/GL1/01-abcde%20scripts/Cartographer/MODIFIED%20FOR%20ELF%20Cartographer.pm
https://bitbucket.org/Risae/growlanser-6-english-translation/src/master/GL1/01-abcde%20scripts/Atlas/MODIFIED%20FOR%20ELF%20Atlas.pm

He doesn't have any Perl experience and apologizes if anything explodes :laugh:.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on October 08, 2020, 07:38:33 pm
Ah, I see what you mean now. Try adding "#TRIM TRAILING NEWLINES: No" (it defaults to Yes) to the Cartographer script and see if you like that any better; you should get something more like:
Quote
//POINTER #0 @ $1F24 - STRING #0 @ $20FA
#W16($1F24)
//[END-FF]
//
//
//
//
[END-FF]




//POINTER #1 @ $1F26 - STRING #1 @ $20FC

The table counts 2 bytes as 1, and we looked through the Cartographer and Atlas docs but couldn't find anything that might support these kind of pointers.
Huh, that is weird. I guess the game must expect all strings to start on an address that's a multiple of 2 since it wouldn't be able to store pointers to any odd addresses. For insertion, it's certainly wordier, but the new CALCVAR features should be able to handle this with e.g.
Code: [Select]
#VAR(addr, CALCVAR)
#CALC(addr, "!CURRENT_ADDRESS / 2")
#W16(addr, $1F2A)
For extraction, though, you're right that there wasn't any existing support for multiplying pointer values by 2, so I'm glad you were able to add it, hopefully without too much trouble!
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on October 09, 2020, 12:04:26 pm
Hi abw,

thank you for your reply.

Quote
Ah, I see what you mean now. Try adding "#TRIM TRAILING NEWLINES: No" (it defaults to Yes) to the Cartographer script and see if you like that any better; you should get something more like:

Hm, that wouldn't really help in my case, since i only want those 4 spaces between each text block. Right now, using Comments: Both, it also does the spaces inside the commented lines.

Quote
Huh, that is weird. I guess the game must expect all strings to start on an address that's a multiple of 2 since it wouldn't be able to store pointers to any odd addresses. For insertion, it's certainly wordier, but the new CALCVAR features should be able to handle this with e.g.

Growlanser 1 uses a really shitty text encoding in which each character has to be 2 bytes.
It's similar to a shift-jis table that has all the ASCII/1 byte characters stripped out.
In the SCEN.DAT archive, which stores all the script files, the pointers count in 1 byte increments, but in the ELF its always 2 bytes. No idea why the devs decided to do that.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on October 10, 2020, 10:59:29 am
Hm, that wouldn't really help in my case, since i only want those 4 spaces between each text block. Right now, using Comments: Both, it also does the spaces inside the commented lines.
Keep in mind that there's always the option of doing a little post-processing on your own; something like
Code: [Select]
sed -i 's/\(\/\/POINTER\)/\n\n\n\1/' Cartographer_G1_dumped_script*.txt
should get the job done nicely.

Everybody's got their own preferred formatting, and adding dozens of commands for separately controlling every place where whitespace is output is pretty annoying from a development perspective without providing much real benefit. I might look into adding some sort of customizable templating system for the Cartographer output, which could be much more flexible and could be used to address other formatting issues that take more than a single search-and-replace to resolve.

No idea why the devs decided to do that.
This is a common refrain throughout the ROM hacking community :D.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on October 16, 2020, 10:02:07 am
Hi abw,

thank you very much for your reply.

Keep in mind that there's always the option of doing a little post-processing on your own; something like
Code: [Select]
sed -i 's/\(\/\/POINTER\)/\n\n\n\1/' Cartographer_G1_dumped_script*.txt
should get the job done nicely.

Everybody's got their own preferred formatting, and adding dozens of commands for separately controlling every place where whitespace is output is pretty annoying from a development perspective without providing much real benefit. I might look into adding some sort of customizable templating system for the Cartographer output, which could be much more flexible and could be used to address other formatting issues that take more than a single search-and-replace to resolve.
This is a common refrain throughout the ROM hacking community :D.

Yup, i tried "\n\n\n\n//POINTER #" using Notepad++ and it perfectly does its job. But thank you for telling me about the option "#TRIM TRAILING NEWLINES: No", might come in Handy in the future

abw, i have another question regarding an issue that we came across.
Inside Growlanser 1 ELF there are textblocks stored. It is build up like this:

NOTHING
NOTHING
STUFF
STUFF
TEXTBLOCK 1
POINTERTABLE 1
TEXTBLOCK 2
POINTERTABLE 2
TEXTBLOCK 3
POINTERTABLE 3
TEXTBLOCK 4
POINTERTABLE 4
...
STUFF
STUFF
NOTHING
NOTHING

We have a size issue, because right after the end of the textblock 1/2/3... the pointer table starts.
So, i thought we can just move the textblock below the last STUFF and have an almost unlimited text space.
I found a spot that is close enough for 2 byte pointers that are devided by 2 to still reach (0xFFFF x 2).
Now here comes the problem that we came across.
When Atlas calculates the pointers, it cuts off everything after 0xFFFF, even though the pointers get divided by 2.

Spoiler:
(https://i.imgur.com/lVjPTzU.png)

0x13C610 - 0x12C4A4 = 0x1016C which would roll over to 0x016C
But 0x016C divided by two would be 0x00B6.

Do you know what we could do to make Atlas properly calculate the pointers?


Update:

our super MIPS Hacker and a buddy took a look at this issue in our Discord and suggested a modified part of the Atlas file:

https://bitbucket.org/Risae/growlanser-6-english-translation/commits/8fcbd02b7a51190ff583a7658ff4e330f55d1643

Seems to work like a charm now:

Spoiler:
(https://i.imgur.com/wNv0pP7.png)

Spoiler:
(https://i.imgur.com/WTlp8gH.png)
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on October 16, 2020, 05:30:13 pm
Do you know what we could do to make Atlas properly calculate the pointers?
The previous extension to add a #MULTIPLIER command applied the division *after* the pointer had been reduced to 16 bits, so if your original pointer value was $12340, that would get reduced to $2340 and then divided by 2, resulting in $11A0. It looks like the new version divides first and then reduces to 16 bits, so now if your original pointer value was $12340, that would get divided by 2 to become $091A0 and then reduced to 16 bits, resulting in $91A0, which sounds like it matches up with the game's logic a bit better.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on November 03, 2020, 08:50:36 am
Hi abw,

i am once again in need of your assistance.
Currently i'm trying to dump so called "Bytecode" of Growlanser 6.
The bytecode sits right atop of the game's script:

Spoiler:
(https://i.imgur.com/hmctAot.png)

The Bytecode is right below the second pointer table.
At the bottom of the file sits the script:

Spoiler:
(https://i.imgur.com/KlX7RcF.png)

The first 4 byte pointertable points to a smaller 2 byte pointer table that holds all the bytecode stuff that i currently need.
Here comes the issue: The bytecode doesn't have proper ending tokens.
So, what i did is to use

Code: [Select]
#STRINGS END AT NEXT POINTER: Yes
to be able to have a proper dump of the code without it going to the end of the file.
But here comes the next issue: The 2 byte pointer table has values that are sometimes lower than the previous, which breaks some of the pointers:

Spoiler:
(https://i.imgur.com/ZPTFeHN.png)

In bigger files it seems to be even worse:

Spoiler:
(https://i.imgur.com/ZpjI4G1.png)

My question: Do you know if abcde's Cartographer supports a solution for this issue? Maybe something like only respond to pointers that are higher than the previous?

November 04, 2020, 01:14:52 am - (Auto Merged - Double Posts are not allowed before 7 days.)
Hi abw,

i have another question.
Is there a specific reason why abcde's Atlas doesn't allow dublicate table entries, even if the character is different?
For example, this here:

D901=者
D901=[CC.01FC]

The character that Atlas should relate to are completely different, but it doesn't accept it.
Do you know how to fix that?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on November 04, 2020, 09:48:06 am
My question: Do you know if abcde's Cartographer supports a solution for this issue? Maybe something like only respond to pointers that are higher than the previous?
The first option that comes to mind is adding "#SORT OUTPUT BY STRING ADDRESS: Yes", which in combination with "#STRINGS END AT NEXT POINTER: Yes" should handle this situation fairly well; internally, abcde sorts the pointers by the addresses of their strings rather than the addresses of the pointers and then extracts them in that order, with each string continuing until the start of the next string. I've also made #SCRIPT STOP available for pointer-based methods instead of being restricted to raw dumps, so you could try that too, particularly for the final string if the game doesn't store a separate pointer to indicate where the final string ends.

Is there a specific reason why abcde's Atlas doesn't allow dublicate table entries, even if the character is different?
For example, this here:

D901=者
D901=[CC.01FC]

The character that Atlas should relate to are completely different, but it doesn't accept it.
Do you know how to fix that?
If you want to have two different text values for the same hex value, I suspect you're setting up your table file incorrectly. Thinking about it from an extraction/display perspective, when the game is reading a string, is it going to treat D901 as "者" or as "[CC.01FC]"?
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Risae on November 04, 2020, 10:01:32 am
Hi abw,

thank you very much for your reply!


The first option that comes to mind is adding "#SORT OUTPUT BY STRING ADDRESS: Yes", which in combination with "#STRINGS END AT NEXT POINTER: Yes" should handle this situation fairly well; internally, abcde sorts the pointers by the addresses of their strings rather than the addresses of the pointers and then extracts them in that order, with each string continuing until the start of the next string. I've also made #SCRIPT STOP available for pointer-based methods instead of being restricted to raw dumps, so you could try that too, particularly for the final string if the game doesn't store a separate pointer to indicate where the final string ends.

Ok, i will give that a try.
And thank you for letting me know about #SCRIPT STOP!
I was looking for a way to let the Cartographer know the "real" end of the code, as currently it was writing all the hex until the end of the file after going to the last pointer.

If you want to have two different text values for the same hex value, I suspect you're setting up your table file incorrectly. Thinking about it from an extraction/display perspective, when the game is reading a string, is it going to treat D901 as "者" or as "[CC.01FC]"?

It should treat both as the same, reason:
A kind soul send us their VWF code for Growlanser 1.
He modified the original control codes and made them smaller.
In this case both "者" and "[CC.01FC]" share the same hex "D901".
What i could do, is to replace "者" with "<$D9><$01>" in the script.
But having Atlas not give me any errors would be a lot easier, since the table is specifying what Atlas should write (even if the hex is the same).

November 04, 2020, 11:21:49 am - (Auto Merged - Double Posts are not allowed before 7 days.)
Another example:

Growlanser 1 PSP doesn't have ASCII support (yet). It only supports full width characters.
But because we don't want to write in full width in our script files i would like to let the table file contain both the ASCII and the shift jis full width characters (using the same hex), so we can at least test the translation without breaking anything:

Spoiler:
(https://i.imgur.com/aEfIu29.png)

https://bitbucket.org/Risae/growlanser-1-english-translation/src/master/GL1/01-abcde%20scripts/UTF-8%20GL1%20Table.tbl

November 08, 2020, 01:45:46 am - (Auto Merged - Double Posts are not allowed before 7 days.)








Hi abw,

i hope i don't bother you too much, but unrelated to the other issues that i talked about:

I have been tinkering around with multitable files yesterday.
I managed to properly create one, but i got an error message after i tried to let abcde do its magic:

Code: [Select]
too many table IDs at 'Bytecode test abcde.tbl' line 19! at abcde_v0_0_9/abcde/Table/Table.pm line 75, <TABLE> line 19.
I looked at the code and found the reason why it gave me that:

(https://cdn.discordapp.com/attachments/748899605101412385/774692505865814086/50.PNG)

I would suggest changing the error message, because i didn't know that you need to have the switch

Code: [Select]
--multi-table-files
in order to use multitable files.
Maybe something like:

Code: [Select]
} else {
die ("too many table IDs at '$fileName' line $.!\nIn order to use multi-table files, you need to add the switch --multi-table-files to your abcde command\nCode error location");
}

It will look like this:

Code: [Select]
\abcde_v0_0_9>perl abcde.pl -m bin2text -cm abcde::Cartographer "00000004.SCEN" "abcde Cartographer 00000004.SCEN.txt" Dumped_bytecode -s
too many table IDs at '\Bytecode abcde.tbl' line 61!
In order to use multi-table files, you need to add the switch --multi-table-files to your abcde command
Code error location at \abcde_v0_0_9/abcde/Table/Table.pm line 75, <TABLE> line 61.

I also would propose to maybe add "\nCode error location" at the end of every error message.
It makes it easier to see where the error location is for either the script/table or the abcde file.
Right now it writes those information without any line breaks in between, makes it more difficult to understand in my opinion.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: RedScorpion on December 11, 2020, 08:05:41 am
Hi guys,
i saw the new tool today and have a new project where a lot of pointer are not in blocks. So it is also possible to dump a rom text via seperate pointer file?

Thanks

red
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Choppasmith on December 16, 2020, 04:52:19 pm
Hey abw, hope you're well. I actually have an issue with abcde that's NOT related to the DQ stuff I'm doing.

I'm trying to do a raw text dump of a PSP BIN file. The text is Japanese, in standard SJIS, downloaded a premade SJIS table from the site my cartogtapher.txt as so

Code: [Select]
#GAME NAME: BOOT.bin

// if you need to edit it, the dictionary for the main script is at $B49B-$B686, with the entry lengths stored at $B44B-$B49A
#BLOCK NAME: ISP Dump
#TYPE: NORMAL
#METHOD: RAW
#SCRIPT START:          $3C62B0
#SCRIPT STOP:           $412310
#TABLE: sjis.tbl
#COMMENTS: Both
#END BLOCK

I noticed the table file I downloaded was in Shift JIS and not UTF BOM. When I try to change the format to the latter using Notepad++, most of the text values turn into junk and running it either way in abcde I get the following error

Code: [Select]
error: UTF-8 "\x81" does not map to Unicode at C:/Perl64/lib/Encode.pm line 228, <TABLE> line 60.
 when reading sjis.tbl line 60

EDIT: Nevermind Got it. I overlooked the "Convert to UTF" in the Notepad++ Encoding menu.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: Choppasmith on December 31, 2020, 09:42:30 am
Okay I lied, having a problem with said game above. I can rip text RAW just fine but after discovering where the pointers were Cartographer just hangs. Text starts at 003C62B0, Pointer is at 00412320 with the value of 50 62 3C 00 so my cartographer.txt looks like this

Code: [Select]
#BLOCK NAME: ISP Dump
#TYPE: NORMAL
#METHOD: POINTER_RELATIVE
#POINTER ENDIAN: LITTLE
#POINTER TABLE START: $412320
#POINTER TABLE STOP: $41243F
#POINTER SIZE: $04
#POINTER SPACE: $00
#SCRIPT STOP: $41231F
#ATLAS PTRS: Yes
#BASE POINTER: $60
#SORT OUTPUT BY STRING ADDRESS: Yes
#STRINGS END AT NEXT POINTER: Yes
#TABLE: sjis.tbl
#COMMENTS: No
#END BLOCK

So with this it should be adding $60 to the 50 and looking at the text 3C62B0 but it's not. I tried limiting the dump to just the first couple of pointers and found it's just dumping the bytes at 3C6250 instead of 3C62B0. Am I just missing something obvious?

Edit: Took me a while but I got it. The pointers were 24 bit. Made the pointer size 3,the space 1 and removed Strings End at Next Pointer and it worked perfectly.
Title: Re: abcde: Atlas + Cartographer + A Whole Lot More
Post by: abw on April 04, 2021, 07:24:27 pm
Sorry for the delay in replying - Real Life has been sucking up all of my free time for the past several months.

It should treat both as the same, reason:
A kind soul send us their VWF code for Growlanser 1.
He modified the original control codes and made them smaller.
In this case both "者" and "[CC.01FC]" share the same hex "D901".
What i could do, is to replace "者" with "<$D9><$01>" in the script.
But having Atlas not give me any errors would be a lot easier, since the table is specifying what Atlas should write (even if the hex is the same).
Yup, I hear you. It would occasionally be useful to have different text map to the same hex for insertion, but as I alluded to earlier, that would break extraction.
Spoiler:
We already allow different hex to map to the same text, so also allowing different text to map to the same hex would make table files like
Code: [Select]
00=A
01=A
01=B
valid, and then you're in trouble. When you extract 01, do you get A or B? When you insert A, do you get 00 or 01? You wouldn't even be able to re-insert a script you just extracted with any confidence that the game would still display the same text.
If you have insert scripts from different sources that represent the same byte sequences with different characters, making the insert scripts consistent is definitely one option.

Another example:

Growlanser 1 PSP doesn't have ASCII support (yet). It only supports full width characters.
But because we don't want to write in full width in our script files i would like to let the table file contain both the ASCII and the shift jis full width characters (using the same hex), so we can at least test the translation without breaking anything:
I can also definitely understand not wanting to write in full width SJIS when the rest of your system is set up for ASCII :P. One of abcde's features is that you can use the same table file for insertion as extraction, but you don't have to. When you have specific special circumstances such as these "者" vs. "[CC.01FC]" and ASCII vs. full width SJIS cases where you have insert scripts using different encodings, it might very well make sense to use one or more different table files for insertion. E.g. when inserting a file that uses "者" you use a table file with "D901=者" and when inserting a file that uses "[CC.01FC]" you use a table file that has "D901=[CC.01FC]" instead.

I would suggest changing the error message, because i didn't know that you need to have the switch
Technically it is mentioned in the usage message ("--multi-table-files: allow multiple logical tables inside a single table file.") but yes, I agree that the actual error message should be more descriptive. I'll be sure to add that to the next release - thanks for pointing it out! :beer:

I also would propose to maybe add "\nCode error location" at the end of every error message.
It makes it easier to see where the error location is for either the script/table or the abcde file.
Right now it writes those information without any line breaks in between, makes it more difficult to understand in my opinion.
Hmm, I'm actually squelching perl's default behaviour of printing the error location for most errors, so I'll probably do the same here too.

i saw the new tool today and have a new project where a lot of pointer are not in blocks. So it is also possible to dump a rom text via seperate pointer file?
I think what you're looking for here is Atlas' #PTRLIST command.

EDIT: Nevermind Got it.
Edit: Took me a while but I got it.
Hurray! Glad to see you were able to get things sorted out!