News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: abcde: Atlas + Cartographer + A Whole Lot More  (Read 24288 times)

Risae

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #40 on: October 03, 2020, 04:03:43 am »
Hi abw,

as always, thank you very much for this awesome tool.
I have been assisting someone with figuring out how the pointer table for the Growlanser 1 script works.
Didn't take long and i managed to figure out where its stored and make a Cartographer script to dump its contents.
But it seems like the end token \n issue still persists:

Spoiler:

Spoiler:

Spoiler:

Spoiler:

I am using the latest abcde version. Do you know what might cause this?

abw

  • Hero Member
  • *****
  • Posts: 585
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #41 on: October 07, 2020, 07:16:41 pm »
Can you remind me again what output you were expecting in this case? It looks like your table file says to add 1 newline after each [END-FF] and that's what you're getting; if you want 2 newlines instead, FFFF=[END-FF]\n\n should do the trick.

Risae

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #42 on: October 08, 2020, 02:24:45 am »
Hi abw, i apologize for the crappy table picture.
1 line is supported, multiple lines are not:

Spoiler:

Spoiler:

I hope this makes it more obvious.

Another thing:
Inside Growlanser 1 ELF we came across a very strange pointer table.
The table counts 2 bytes as 1, and we looked through the Cartographer and Atlas docs but couldn't find anything that might support these kind of pointers.
Our super MIPS hacker decided to hack this function into Atlas.pm and Cartographer.pm.
It seems to work so far and we would like to give you the code in case you want to implement it into abcde:

Code: [Select]
Cartographer:
#POINTER MULTIPLIER:        $XX

Atlas:
#MULTIPLIER($xx)
It gets added automatically with the new cartographer if #POINTER MULTIPLIER is provided

https://bitbucket.org/Risae/growlanser-6-english-translation/src/master/GL1/01-abcde%20scripts/Cartographer/MODIFIED%20FOR%20ELF%20Cartographer.pm
https://bitbucket.org/Risae/growlanser-6-english-translation/src/master/GL1/01-abcde%20scripts/Atlas/MODIFIED%20FOR%20ELF%20Atlas.pm

He doesn't have any Perl experience and apologizes if anything explodes :laugh:.
« Last Edit: October 08, 2020, 04:48:30 am by Risae »

abw

  • Hero Member
  • *****
  • Posts: 585
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #43 on: October 08, 2020, 07:38:33 pm »
Ah, I see what you mean now. Try adding "#TRIM TRAILING NEWLINES: No" (it defaults to Yes) to the Cartographer script and see if you like that any better; you should get something more like:
Quote
//POINTER #0 @ $1F24 - STRING #0 @ $20FA
#W16($1F24)
//[END-FF]
//
//
//
//
[END-FF]




//POINTER #1 @ $1F26 - STRING #1 @ $20FC

The table counts 2 bytes as 1, and we looked through the Cartographer and Atlas docs but couldn't find anything that might support these kind of pointers.
Huh, that is weird. I guess the game must expect all strings to start on an address that's a multiple of 2 since it wouldn't be able to store pointers to any odd addresses. For insertion, it's certainly wordier, but the new CALCVAR features should be able to handle this with e.g.
Code: [Select]
#VAR(addr, CALCVAR)
#CALC(addr, "!CURRENT_ADDRESS / 2")
#W16(addr, $1F2A)
For extraction, though, you're right that there wasn't any existing support for multiplying pointer values by 2, so I'm glad you were able to add it, hopefully without too much trouble!

Risae

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #44 on: October 09, 2020, 12:04:26 pm »
Hi abw,

thank you for your reply.

Quote
Ah, I see what you mean now. Try adding "#TRIM TRAILING NEWLINES: No" (it defaults to Yes) to the Cartographer script and see if you like that any better; you should get something more like:

Hm, that wouldn't really help in my case, since i only want those 4 spaces between each text block. Right now, using Comments: Both, it also does the spaces inside the commented lines.

Quote
Huh, that is weird. I guess the game must expect all strings to start on an address that's a multiple of 2 since it wouldn't be able to store pointers to any odd addresses. For insertion, it's certainly wordier, but the new CALCVAR features should be able to handle this with e.g.

Growlanser 1 uses a really shitty text encoding in which each character has to be 2 bytes.
It's similar to a shift-jis table that has all the ASCII/1 byte characters stripped out.
In the SCEN.DAT archive, which stores all the script files, the pointers count in 1 byte increments, but in the ELF its always 2 bytes. No idea why the devs decided to do that.

abw

  • Hero Member
  • *****
  • Posts: 585
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #45 on: October 10, 2020, 10:59:29 am »
Hm, that wouldn't really help in my case, since i only want those 4 spaces between each text block. Right now, using Comments: Both, it also does the spaces inside the commented lines.
Keep in mind that there's always the option of doing a little post-processing on your own; something like
Code: [Select]
sed -i 's/\(\/\/POINTER\)/\n\n\n\1/' Cartographer_G1_dumped_script*.txt
should get the job done nicely.

Everybody's got their own preferred formatting, and adding dozens of commands for separately controlling every place where whitespace is output is pretty annoying from a development perspective without providing much real benefit. I might look into adding some sort of customizable templating system for the Cartographer output, which could be much more flexible and could be used to address other formatting issues that take more than a single search-and-replace to resolve.

No idea why the devs decided to do that.
This is a common refrain throughout the ROM hacking community :D.

Risae

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #46 on: October 16, 2020, 10:02:07 am »
Hi abw,

thank you very much for your reply.

Keep in mind that there's always the option of doing a little post-processing on your own; something like
Code: [Select]
sed -i 's/\(\/\/POINTER\)/\n\n\n\1/' Cartographer_G1_dumped_script*.txt
should get the job done nicely.

Everybody's got their own preferred formatting, and adding dozens of commands for separately controlling every place where whitespace is output is pretty annoying from a development perspective without providing much real benefit. I might look into adding some sort of customizable templating system for the Cartographer output, which could be much more flexible and could be used to address other formatting issues that take more than a single search-and-replace to resolve.
This is a common refrain throughout the ROM hacking community :D.

Yup, i tried "\n\n\n\n//POINTER #" using Notepad++ and it perfectly does its job. But thank you for telling me about the option "#TRIM TRAILING NEWLINES: No", might come in Handy in the future

abw, i have another question regarding an issue that we came across.
Inside Growlanser 1 ELF there are textblocks stored. It is build up like this:

NOTHING
NOTHING
STUFF
STUFF
TEXTBLOCK 1
POINTERTABLE 1
TEXTBLOCK 2
POINTERTABLE 2
TEXTBLOCK 3
POINTERTABLE 3
TEXTBLOCK 4
POINTERTABLE 4
...
STUFF
STUFF
NOTHING
NOTHING

We have a size issue, because right after the end of the textblock 1/2/3... the pointer table starts.
So, i thought we can just move the textblock below the last STUFF and have an almost unlimited text space.
I found a spot that is close enough for 2 byte pointers that are devided by 2 to still reach (0xFFFF x 2).
Now here comes the problem that we came across.
When Atlas calculates the pointers, it cuts off everything after 0xFFFF, even though the pointers get divided by 2.

Spoiler:

0x13C610 - 0x12C4A4 = 0x1016C which would roll over to 0x016C
But 0x016C divided by two would be 0x00B6.

Do you know what we could do to make Atlas properly calculate the pointers?


Update:

our super MIPS Hacker and a buddy took a look at this issue in our Discord and suggested a modified part of the Atlas file:

https://bitbucket.org/Risae/growlanser-6-english-translation/commits/8fcbd02b7a51190ff583a7658ff4e330f55d1643

Seems to work like a charm now:

Spoiler:

Spoiler:
« Last Edit: October 16, 2020, 10:46:26 am by Risae »

abw

  • Hero Member
  • *****
  • Posts: 585
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #47 on: October 16, 2020, 05:30:13 pm »
Do you know what we could do to make Atlas properly calculate the pointers?
The previous extension to add a #MULTIPLIER command applied the division *after* the pointer had been reduced to 16 bits, so if your original pointer value was $12340, that would get reduced to $2340 and then divided by 2, resulting in $11A0. It looks like the new version divides first and then reduces to 16 bits, so now if your original pointer value was $12340, that would get divided by 2 to become $091A0 and then reduced to 16 bits, resulting in $91A0, which sounds like it matches up with the game's logic a bit better.

Risae

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #48 on: November 03, 2020, 08:50:36 am »
Hi abw,

i am once again in need of your assistance.
Currently i'm trying to dump so called "Bytecode" of Growlanser 6.
The bytecode sits right atop of the game's script:

Spoiler:

The Bytecode is right below the second pointer table.
At the bottom of the file sits the script:

Spoiler:

The first 4 byte pointertable points to a smaller 2 byte pointer table that holds all the bytecode stuff that i currently need.
Here comes the issue: The bytecode doesn't have proper ending tokens.
So, what i did is to use

Code: [Select]
#STRINGS END AT NEXT POINTER: Yes
to be able to have a proper dump of the code without it going to the end of the file.
But here comes the next issue: The 2 byte pointer table has values that are sometimes lower than the previous, which breaks some of the pointers:

Spoiler:

In bigger files it seems to be even worse:

Spoiler:

My question: Do you know if abcde's Cartographer supports a solution for this issue? Maybe something like only respond to pointers that are higher than the previous?

November 04, 2020, 01:14:52 am - (Auto Merged - Double Posts are not allowed before 7 days.)
Hi abw,

i have another question.
Is there a specific reason why abcde's Atlas doesn't allow dublicate table entries, even if the character is different?
For example, this here:

D901=者
D901=[CC.01FC]

The character that Atlas should relate to are completely different, but it doesn't accept it.
Do you know how to fix that?
« Last Edit: November 04, 2020, 01:14:52 am by Risae »

abw

  • Hero Member
  • *****
  • Posts: 585
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #49 on: November 04, 2020, 09:48:06 am »
My question: Do you know if abcde's Cartographer supports a solution for this issue? Maybe something like only respond to pointers that are higher than the previous?
The first option that comes to mind is adding "#SORT OUTPUT BY STRING ADDRESS: Yes", which in combination with "#STRINGS END AT NEXT POINTER: Yes" should handle this situation fairly well; internally, abcde sorts the pointers by the addresses of their strings rather than the addresses of the pointers and then extracts them in that order, with each string continuing until the start of the next string. I've also made #SCRIPT STOP available for pointer-based methods instead of being restricted to raw dumps, so you could try that too, particularly for the final string if the game doesn't store a separate pointer to indicate where the final string ends.

Is there a specific reason why abcde's Atlas doesn't allow dublicate table entries, even if the character is different?
For example, this here:

D901=者
D901=[CC.01FC]

The character that Atlas should relate to are completely different, but it doesn't accept it.
Do you know how to fix that?
If you want to have two different text values for the same hex value, I suspect you're setting up your table file incorrectly. Thinking about it from an extraction/display perspective, when the game is reading a string, is it going to treat D901 as "者" or as "[CC.01FC]"?

Risae

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #50 on: November 04, 2020, 10:01:32 am »
Hi abw,

thank you very much for your reply!


The first option that comes to mind is adding "#SORT OUTPUT BY STRING ADDRESS: Yes", which in combination with "#STRINGS END AT NEXT POINTER: Yes" should handle this situation fairly well; internally, abcde sorts the pointers by the addresses of their strings rather than the addresses of the pointers and then extracts them in that order, with each string continuing until the start of the next string. I've also made #SCRIPT STOP available for pointer-based methods instead of being restricted to raw dumps, so you could try that too, particularly for the final string if the game doesn't store a separate pointer to indicate where the final string ends.

Ok, i will give that a try.
And thank you for letting me know about #SCRIPT STOP!
I was looking for a way to let the Cartographer know the "real" end of the code, as currently it was writing all the hex until the end of the file after going to the last pointer.

If you want to have two different text values for the same hex value, I suspect you're setting up your table file incorrectly. Thinking about it from an extraction/display perspective, when the game is reading a string, is it going to treat D901 as "者" or as "[CC.01FC]"?

It should treat both as the same, reason:
A kind soul send us their VWF code for Growlanser 1.
He modified the original control codes and made them smaller.
In this case both "者" and "[CC.01FC]" share the same hex "D901".
What i could do, is to replace "者" with "<$D9><$01>" in the script.
But having Atlas not give me any errors would be a lot easier, since the table is specifying what Atlas should write (even if the hex is the same).

November 04, 2020, 11:21:49 am - (Auto Merged - Double Posts are not allowed before 7 days.)
Another example:

Growlanser 1 PSP doesn't have ASCII support (yet). It only supports full width characters.
But because we don't want to write in full width in our script files i would like to let the table file contain both the ASCII and the shift jis full width characters (using the same hex), so we can at least test the translation without breaking anything:

Spoiler:

https://bitbucket.org/Risae/growlanser-1-english-translation/src/master/GL1/01-abcde%20scripts/UTF-8%20GL1%20Table.tbl

November 08, 2020, 01:45:46 am - (Auto Merged - Double Posts are not allowed before 7 days.)








Hi abw,

i hope i don't bother you too much, but unrelated to the other issues that i talked about:

I have been tinkering around with multitable files yesterday.
I managed to properly create one, but i got an error message after i tried to let abcde do its magic:

Code: [Select]
too many table IDs at 'Bytecode test abcde.tbl' line 19! at abcde_v0_0_9/abcde/Table/Table.pm line 75, <TABLE> line 19.
I looked at the code and found the reason why it gave me that:



I would suggest changing the error message, because i didn't know that you need to have the switch

Code: [Select]
--multi-table-files
in order to use multitable files.
Maybe something like:

Code: [Select]
} else {
die ("too many table IDs at '$fileName' line $.!\nIn order to use multi-table files, you need to add the switch --multi-table-files to your abcde command\nCode error location");
}

It will look like this:

Code: [Select]
\abcde_v0_0_9>perl abcde.pl -m bin2text -cm abcde::Cartographer "00000004.SCEN" "abcde Cartographer 00000004.SCEN.txt" Dumped_bytecode -s
too many table IDs at '\Bytecode abcde.tbl' line 61!
In order to use multi-table files, you need to add the switch --multi-table-files to your abcde command
Code error location at \abcde_v0_0_9/abcde/Table/Table.pm line 75, <TABLE> line 61.

I also would propose to maybe add "\nCode error location" at the end of every error message.
It makes it easier to see where the error location is for either the script/table or the abcde file.
Right now it writes those information without any line breaks in between, makes it more difficult to understand in my opinion.
« Last Edit: November 08, 2020, 01:51:22 am by Risae »

RedScorpion

  • Full Member
  • ***
  • Posts: 112
    • View Profile
    • Snes-Projects
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #51 on: December 11, 2020, 08:05:41 am »
Hi guys,
i saw the new tool today and have a new project where a lot of pointer are not in blocks. So it is also possible to dump a rom text via seperate pointer file?

Thanks

red

Choppasmith

  • Full Member
  • ***
  • Posts: 209
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #52 on: December 16, 2020, 04:52:19 pm »
Hey abw, hope you're well. I actually have an issue with abcde that's NOT related to the DQ stuff I'm doing.

I'm trying to do a raw text dump of a PSP BIN file. The text is Japanese, in standard SJIS, downloaded a premade SJIS table from the site my cartogtapher.txt as so

Code: [Select]
#GAME NAME: BOOT.bin

// if you need to edit it, the dictionary for the main script is at $B49B-$B686, with the entry lengths stored at $B44B-$B49A
#BLOCK NAME: ISP Dump
#TYPE: NORMAL
#METHOD: RAW
#SCRIPT START:          $3C62B0
#SCRIPT STOP:           $412310
#TABLE: sjis.tbl
#COMMENTS: Both
#END BLOCK

I noticed the table file I downloaded was in Shift JIS and not UTF BOM. When I try to change the format to the latter using Notepad++, most of the text values turn into junk and running it either way in abcde I get the following error

Code: [Select]
error: UTF-8 "\x81" does not map to Unicode at C:/Perl64/lib/Encode.pm line 228, <TABLE> line 60.
 when reading sjis.tbl line 60

EDIT: Nevermind Got it. I overlooked the "Convert to UTF" in the Notepad++ Encoding menu.
« Last Edit: December 16, 2020, 05:39:33 pm by Choppasmith »

Choppasmith

  • Full Member
  • ***
  • Posts: 209
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #53 on: December 31, 2020, 09:42:30 am »
Okay I lied, having a problem with said game above. I can rip text RAW just fine but after discovering where the pointers were Cartographer just hangs. Text starts at 003C62B0, Pointer is at 00412320 with the value of 50 62 3C 00 so my cartographer.txt looks like this

Code: [Select]
#BLOCK NAME: ISP Dump
#TYPE: NORMAL
#METHOD: POINTER_RELATIVE
#POINTER ENDIAN: LITTLE
#POINTER TABLE START: $412320
#POINTER TABLE STOP: $41243F
#POINTER SIZE: $04
#POINTER SPACE: $00
#SCRIPT STOP: $41231F
#ATLAS PTRS: Yes
#BASE POINTER: $60
#SORT OUTPUT BY STRING ADDRESS: Yes
#STRINGS END AT NEXT POINTER: Yes
#TABLE: sjis.tbl
#COMMENTS: No
#END BLOCK

So with this it should be adding $60 to the 50 and looking at the text 3C62B0 but it's not. I tried limiting the dump to just the first couple of pointers and found it's just dumping the bytes at 3C6250 instead of 3C62B0. Am I just missing something obvious?

Edit: Took me a while but I got it. The pointers were 24 bit. Made the pointer size 3,the space 1 and removed Strings End at Next Pointer and it worked perfectly.
« Last Edit: January 16, 2021, 10:44:18 am by Choppasmith »

abw

  • Hero Member
  • *****
  • Posts: 585
    • View Profile
Re: abcde: Atlas + Cartographer + A Whole Lot More
« Reply #54 on: April 04, 2021, 07:24:27 pm »
Sorry for the delay in replying - Real Life has been sucking up all of my free time for the past several months.

It should treat both as the same, reason:
A kind soul send us their VWF code for Growlanser 1.
He modified the original control codes and made them smaller.
In this case both "者" and "[CC.01FC]" share the same hex "D901".
What i could do, is to replace "者" with "<$D9><$01>" in the script.
But having Atlas not give me any errors would be a lot easier, since the table is specifying what Atlas should write (even if the hex is the same).
Yup, I hear you. It would occasionally be useful to have different text map to the same hex for insertion, but as I alluded to earlier, that would break extraction.
Spoiler:
We already allow different hex to map to the same text, so also allowing different text to map to the same hex would make table files like
Code: [Select]
00=A
01=A
01=B
valid, and then you're in trouble. When you extract 01, do you get A or B? When you insert A, do you get 00 or 01? You wouldn't even be able to re-insert a script you just extracted with any confidence that the game would still display the same text.
If you have insert scripts from different sources that represent the same byte sequences with different characters, making the insert scripts consistent is definitely one option.

Another example:

Growlanser 1 PSP doesn't have ASCII support (yet). It only supports full width characters.
But because we don't want to write in full width in our script files i would like to let the table file contain both the ASCII and the shift jis full width characters (using the same hex), so we can at least test the translation without breaking anything:
I can also definitely understand not wanting to write in full width SJIS when the rest of your system is set up for ASCII :P. One of abcde's features is that you can use the same table file for insertion as extraction, but you don't have to. When you have specific special circumstances such as these "者" vs. "[CC.01FC]" and ASCII vs. full width SJIS cases where you have insert scripts using different encodings, it might very well make sense to use one or more different table files for insertion. E.g. when inserting a file that uses "者" you use a table file with "D901=者" and when inserting a file that uses "[CC.01FC]" you use a table file that has "D901=[CC.01FC]" instead.

I would suggest changing the error message, because i didn't know that you need to have the switch
Technically it is mentioned in the usage message ("--multi-table-files: allow multiple logical tables inside a single table file.") but yes, I agree that the actual error message should be more descriptive. I'll be sure to add that to the next release - thanks for pointing it out! :beer:

I also would propose to maybe add "\nCode error location" at the end of every error message.
It makes it easier to see where the error location is for either the script/table or the abcde file.
Right now it writes those information without any line breaks in between, makes it more difficult to understand in my opinion.
Hmm, I'm actually squelching perl's default behaviour of printing the error location for most errors, so I'll probably do the same here too.

i saw the new tool today and have a new project where a lot of pointer are not in blocks. So it is also possible to dump a rom text via seperate pointer file?
I think what you're looking for here is Atlas' #PTRLIST command.

EDIT: Nevermind Got it.
Edit: Took me a while but I got it.
Hurray! Glad to see you were able to get things sorted out!