News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: abcde: Atlas + Cartographer + A Whole Lot More  (Read 17181 times)

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #20 on: April 22, 2020, 09:07:22 pm »
Hmm, yeah, 500+ * 500+ = 250000+, which is definitely a lot. Assuming your single table file currently looks like this (1024 entries + whatever else you've got):
Code: [Select]
1F00=[V1F00]\n
1F01=[V1F01]\n
...
20FF=[V20FF]\n

971000=971_000
971001=971_001
...
9711FF=971_1FF
you could make use of table switching to refactor the existing entries into multiple tables and then handle the combinations like this (805 entries over 5 tables + whatever else you've got):
Spoiler:
Code: [Select]
!1F=<[V1F>,@V:1
!20=<[V20>,@V:1

!%0001111100000000100101110001=<[V1F00]971_>,@nybble_01:1,@nybble:1,@nybble_newline:1
!%0001111100000001100101110001=<[V1F01]971_>,@nybble_01:1,@nybble:1,@nybble_newline:1
...
!%0010000011111111100101110001=<[V20FF]971_>,@nybble_01:1,@nybble:1,@nybble_newline:1

!%100101110001=<971_>,@nybble_01:1,@nybble:1
Code: [Select]
@V
00=00]\n
01=01]\n
...
FF=FF]\n
Code: [Select]
@nybble_01
# only matches 0 and 1
%0000=0
%0001=1
Code: [Select]
@nybble
%0000=0
%0001=1
...
%1111=F
Code: [Select]
@nybble_newline
%0000=0\n
%0001=1\n
...
%1111=F\n
(And wow that would be so much more readable if the left-hand side of table entries allowed nybbles instead of full bytes; I think I'll look into adding that in v0.0.8!)

This approach is obviously not perfect and has serious scaling issues since your table file needs to individually list all of the [V****] tokens before getting to the 971_* tokens, but 800+ entries is definitely better than 250000+ entries.

Risae

  • Jr. Member
  • **
  • Posts: 65
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #21 on: April 28, 2020, 09:34:42 am »
Hi abw,

thank you for your reply.
Hm, i see. I read about the function of using multiple table files before.
This is not a script file which i have to fix instantly, so i will take a look at it another time and see how it goes.
Thank you for your help!

_Ombra_

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #22 on: May 05, 2020, 10:46:36 pm »
Hey abw,

this looks great but i have a few questions. I'm trying to use it and i'm stumbling upon some stuff that i don't have with Cartographer for example.

If the tbl files contain a character assigned to < or > i get an error straight away. If i comment that out and i have letters i added (for example accents) it just says they aren't in the unicode table.

It that normal behavior? is it fixable?

Thanks

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #23 on: May 06, 2020, 09:34:55 pm »
If the tbl files contain a character assigned to < or > i get an error straight away.
Yup, that's expected. You'll have to update your table files to use something else like [], {}, or ‹› (U+2039/U+203A) instead.

If i comment that out and i have letters i added (for example accents) it just says they aren't in the unicode table.
My guess here is that your table files aren't encoded as UTF-8; simply saving them as UTF-8 instead of something else like ISO-8859-1 or Windows-1252 should solve that problem. Note that UTF-8 is expected for all text input, so if you have other pre-existing files in some other encoding, you'll need to update their encoding in order to have them work properly.

May 13, 2020, 07:28:43 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
v0.0.8 is up. This one's mostly a bugfix release for some regression errors in v0.0.7 and some funky #AUTOWRITE behaviour I recently noticed that was also present in Atlas. The only real improvement in this version is the ability to use hex table entries with odd lengths, so e.g. "248=foo" will match 12 bits of data instead of being flagged as an error.
« Last Edit: May 13, 2020, 07:28:43 pm by abw »

nejimakipiyo

  • Jr. Member
  • **
  • Posts: 42
    • View Profile
    • Demon Passion
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #24 on: July 24, 2020, 08:57:28 am »
Hello! This is my first time using this software. Attempting to do an extraction of all of the lists in Dragon Warrior Monsters (GBC). Chicken Knife was helping me get the extraction set up but we ran into a problem and he suggested I reach out to you after he couldn't figure it out. So here I am! And here's my problem:



When I try to run this command in the command line, it tells me it can't find Table.pm.
But that file is right here!



If it helps, here's the directory where my hacking stuff is organized.




This is all 100% new to me, so I'm at a total loss for how to fix this problem. Any help would be appreciated!

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #25 on: July 24, 2020, 08:39:45 pm »
Hmm, what's in that first "abcde" directory? It sort of looks like you've got an extra level of directory in there that shouldn't be... did you maybe originally have abcde.pl located under "DQM Hack\abcde\" and then move just that one file to "DQM Hack\" while leaving all of its supporting files under "DQM Hack\abcde\abcde\"?

Whatever ended up happening, you'll need to make it so that abcde.pl is located 2 levels up from Table.pm just like it is when unzipping the zip file normally, so you'll have e.g. "DQM Hack\abcde.pl" and "DQM Hack\abcde\Table\Table.pm".

Personally I'd probably put abcde in its own location e.g. C:\Users\wings\abcde\ and then call it from DQM Hack as ..\abcde\abcde.pl, but where you want to put it is up to you; just make sure not to separate individual files from the rest of their friends ;).

nejimakipiyo

  • Jr. Member
  • **
  • Posts: 42
    • View Profile
    • Demon Passion
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #26 on: July 24, 2020, 10:51:54 pm »
Thank you! That certainly helped me get one step forward, I think. I downloaded abcde back in January and haven't made a serious attempt to use it until now, so I couldn't begin to tell you what happened.

Now it's asking something else I'm not sure about. A Hash::Priority Queue Module?
Chicken Knife doesn't know this one either!

----

EDIT: We figured that one out, I had to shuffle my directories around more. This is where we're stumped now:



It doesn't seem to like my table file for some reason. It's accessing the table file, since it can see the first line. The tablefile.tbl is written that way in my cartographer text as well as in the command I'm putting into the command line. I'm lost.  :banghead:
« Last Edit: July 24, 2020, 11:39:41 pm by nejimakipiyo »

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #27 on: July 25, 2020, 10:23:34 am »
EDIT: We figured that one out, I had to shuffle my directories around more.
Yup, leaving the original directory structure intact is definitely the way to go!

This is where we're stumped now:
It looks like you're trying to pass a table file where a Cartographer command file is expected. What you want is probably something like
Code: [Select]
perl abcde.pl -cm abcde::Cartographer "Dragon Warrior Monsters.gbc" dwm_Cartographer_commands.txt dwm_Cartographer_out -s
with an appropriate dwm_Cartographer_commands.txt file that describes where in the ROM any text and pointers that you want to extract is located and what table file to use to translate the binary into text. Cartographer comes with example files for Final Fantasy (NES), and I'm assuming you already know where the text and pointers are located for Dragon Warrior Monsters since you've been editing it already, so plugging in the appropriate values shouldn't be too hard.

nejimakipiyo

  • Jr. Member
  • **
  • Posts: 42
    • View Profile
    • Demon Passion
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #28 on: July 25, 2020, 04:53:42 pm »
Thank you for your help again! I fixed the wrong command I was using in the command line with the one you wrote in your previous reply instead, and managed to get an output file created...



...except that the output file is blank. Completely empty. I tried it both with and without the "bin2text" part of the command from my previous attempts, to no result. Chicken Knife said it might have something to do with parts of the Cartographer sections that he doesn't understand, and I couldn't find anything elucidating in the documentation. Hope it's ok that I sent you a PM with a link to my files.  :angel:
« Last Edit: July 25, 2020, 05:35:11 pm by nejimakipiyo »

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #29 on: July 25, 2020, 11:21:22 pm »
It looks like your table file doesn't have any end tokens defined. Without some way of knowing when to stop, the extraction process starts from the first pointer and just keeps on running until it reaches the end of the ROM, then it starts from the second pointer and keeps on going until it reaches the end of the ROM. Lather, rinse, repeat until the final pointer, which again dumps everything until the end of the ROM. With over 1500 pointers starting about 1 MB from the end of the ROM, that's going to generate well over 1.5 GB of text; the disk I/O time alone for a file that large will be substantial, to say nothing of abcde's own processing time.

Fortunately the solution is simple: just add a "/" before your "F0=[F0]\n" to make it an end token "/F0=[F0]\n" :).

nejimakipiyo

  • Jr. Member
  • **
  • Posts: 42
    • View Profile
    • Demon Passion
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #30 on: July 26, 2020, 08:00:28 pm »
That worked! Thank you so much for all your help!  ;D

Chicken Knife

  • Sr. Member
  • ****
  • Posts: 456
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #31 on: July 27, 2020, 08:15:09 pm »
I'll give the next response since nejimakipiyo can't bump the thread.

We produced the big list dump, which is great, but when we reinsert it as an atlas file, we are now 233 characters over. nejimakipiyo got that result first, then I set up the file independently and got the same result as well. My other attempts to turn a cartographer out file into an atlas file have always been successful and didn't result in any kind of increase in bytes. I don't know how that would even be possible.

Anyway, nejimakipiyo is sending you a link to the most updated files. We appreciate your help as always!

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #32 on: July 27, 2020, 11:12:00 pm »
That's because some strings are pointed at by multiple pointers, resulting in those strings being extracted multiple times, and by default they appear multiple times in Cartographer's output, so unless you do something to remove duplicate strings, you'll be inserting multiple copies of the duplicate strings, which is where your size increase is coming from.

abcde adds a "#STRINGS END AT NEXT POINTER: Yes" option that will automatically remove duplicate strings for you, though I think I'll expand its documentation to make that side effect more obvious in the next release. In your case, it looks like the pointers to all the duplicate strings are adjacent, so #STRINGS END AT NEXT POINTER is probably enough, but in cases where the pointer values are not in ascending order, you might also want to set "#SORT OUTPUT BY STRING ADDRESS: Yes".

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #33 on: August 28, 2020, 07:07:19 pm »
v0.0.9 is now up. The most noteworthy part of this version is the introduction of the ability to perform more-or-less arbitrary user-supplied arithmetic calculations during insertion and write the result to ROM in a variety of ways; among other things, this makes it possible to insert all of the weird pointer formats that I've seen and a very large variety of formats that I haven't seen.

meunierd

  • RHDN Patreon Supporter!
  • Jr. Member
  • *****
  • Posts: 74
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #34 on: August 29, 2020, 09:01:33 pm »
Is there a git copy of this online somewhere? It's an amazing tool!

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #35 on: August 29, 2020, 09:57:31 pm »
No, just my private repo filled with all kinds of embarrassing commit messages :P. I'm glad you're finding it useful!

FlashPV

  • Full Member
  • ***
  • Posts: 175
    • View Profile
    • Terminus Traductions
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #36 on: September 05, 2020, 04:37:00 am »
Hi, looks like this tool may be what I've been looking for. Phantasy Star II uses some relative pointer. Each pointer is the sum of all previous pointers. Is there any way to extract and insert the script using abcde?

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #37 on: September 05, 2020, 11:40:37 am »
So, there's a base pointer somewhere and then the value for each of the individual pointers is the running total of the lengths of all the previous strings or the distance from the base pointer? E.g. if each string happens to be $10 long, you'd have pointer values like $00, $10, $20, etc.?

If so, then yes, you could extract that pretty normally with "#METHOD: POINTER_RELATIVE", setting "#BASE POINTER:" to whatever the address of the first string is. You'd then get pointer values of <first string address> + $00, <first string address> + $10, <first string address> + $20, etc. lining up with your string addresses.

For insertion, you could use the new CALCVAR features, filling in <> variables with your desired values:
Code: [Select]
// define some variables
#VAR(base, CALCVAR)
#VAR(length, CALCVAR)

// jump to insertion point, set an upper bound on insertion address
#JMP(<script start>, <script stop>)
// save current address into "base" variable
#CALC(base, "!CURRENT_ADDRESS")
// calculate "length" = current address - "base" (= $00 since they're currently the same value)
#CALC(length, "!CURRENT_ADDRESS - !base")
// use the current value of "length" ($00) as the value for the first pointer
#W8(length, <pointer 0 address>)
String #1 here.[end]

// calculate "length" = current address - "base" (= $10 since string 1 was $10 long)
#CALC(length, "!CURRENT_ADDRESS - !base")
// use the current value of "length" ($10) as the value for the second pointer
#W8(length, <pointer 1 address>)
String #12 here.[end]

// calculate "length" = current address - "base" (= $20 since both strings 1 and 2 were $10 long)
#CALC(length, "!CURRENT_ADDRESS - !base")
// use the current value of "length" ($20) as the value for the third pointer
#W8(length, <pointer 2 address>)
String #3 here.[end]

// etc.

FlashPV

  • Full Member
  • ***
  • Posts: 175
    • View Profile
    • Terminus Traductions
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #38 on: September 05, 2020, 01:37:27 pm »
Hum sorry, I didn't explain correctly.In fact each pointer indicate the distance from the previous pointer. If the first string is $07 long and the second is $10 long then the second pointer will be 07 and the third 10.

abw

  • Hero Member
  • *****
  • Posts: 554
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #39 on: September 05, 2020, 03:05:46 pm »
Ah, so then if each string happens to be $10 long, you'd have pointer values like $00, $10, $10, etc.? Yup, that is a slightly different setup.

Unfortunately, I haven't yet extended Cartographer's interface enough to make extracting text from pointer values like that easy. You can still do it by using a separate block for each pointer, but you'll have to calculate each string's start address yourself and use that value in the corresponding Cartographer block. So it is possible, but pretty annoying.

Inserting, on the other hand, is much easier since you can now control the pointer value calculations in your insert script:
Code: [Select]
// define a variable
#VAR(prev_pointer, CALCVAR)

// jump to insertion point, set an upper bound on insertion address
#JMP(<script start>, <script stop>)
// set "prev_pointer" variable to 0
#CALC(prev_pointer, "0")
// use the current value of "prev_pointer" ($00) as the value for the first pointer
#W8(prev_pointer, <pointer 0 address>)
String #1 here.[end]

// calculate "prev_pointer" = current address - "prev_pointer" (= $10 since string 1 was $10 long)
#CALC(prev_pointer, "!CURRENT_ADDRESS - !prev_pointer")
// use the current value of "prev_pointer" ($10) as the value for the second pointer
#W8(prev_pointer, <pointer 1 address>)
String #12 here.[end]

// calculate "prev_pointer" = current address - "prev_pointer" (= $10 since string 2 was $10 long)
#CALC(prev_pointer, "!CURRENT_ADDRESS - !prev_pointer")
// use the current value of "prev_pointer" ($10) as the value for the third pointer
#W8(prev_pointer, <pointer 2 address>)
String #3 here.[end]

// etc.