News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: [SOLVED] [PS2] Using Atlas after Cartographer, confused  (Read 2817 times)

Risae

  • Jr. Member
  • **
  • Posts: 73
    • View Profile
[SOLVED] [PS2] Using Atlas after Cartographer, confused
« on: March 09, 2019, 10:31:31 am »
Edit: I created a little blog to document everything that i could figure out for Growlanser 6:
https://growlanser6english.blogspot.com/

Dear Community,

i am currently in the process of trying to translate the game "Growlanser 6" from Japanese to English.
through various means i managed to unpack the PS2 ISO, unpack the .DAT files within and locate the files that containt the text.

I managed to find the pointer table and sort of figure out how it works
(its located right above the text block, each pointer pointing to the first letter of the text).

I found various tools on this site to extract the text with the according text pointer associated.
Nothing really seemed to work and then i tried the tool "Cartographer".
After a bit of trial n error i actually managed to succesfully extract the text + pointers:

Code: [Select]
#GAME NAME: Growlanser 6

#BLOCK NAME: Dialogue Block (POINTER_RELATIVE)
#TYPE: NORMAL
#METHOD: POINTER_RELATIVE
#POINTER ENDIAN: LITTLE
#POINTER TABLE START: $FF24
#POINTER TABLE STOP: $10E93
#POINTER SIZE: $04
#POINTER SPACE: $00
#ATLAS PTRS: Yes
#BASE POINTER: $10EA0
#TABLE: sjis.tbl
#COMMENTS: Yes
#END BLOCK

The result was rather beautiful to look at:

Spoiler:

I have to figure out how to properly include Control Codes like

FF FC = next textline, same textwindow, same NPC
FF FD = next textwindow, same NPC
FF FE 00 = Closes textwindow, regaining movement for PC | Also seems to be used for next textline, choosing different options for npc, like A B C

but that is for another time.
I changed text from japanese to english in the file, and then i tried to re-insert the cartographer script file into the actual growlanser file.
I imagined it would be the same like using Cartographer, so i created a .BAT file with this code:

Code: [Select]
Atlas 00000005.flk GL6_script_000.txt

pause

and used it. This was the result:

Spoiler:

After looking at the Growlanser file, the whole pointer table was changed to "0000000".
The reinsert completely removed the pointer table above the text block and nothing in the text block was altered.

I am currently at a loss to what i have to do next with Atlas.
I read through the Atlas manual, but i am really confused to what i have to do with the Atlas script files.
Do i have to create an Atlas Script file to re-insert the Cartographer text file?



Edit: I just noticed that i might have made a huge mistake with the text extraction using Cartographer.
I wondered, why is the output file 32MB big?!
And i just noticed that the text in every line doesnt stop, and goes until the very end.
I think i need to input a control code so Cardographer knows where to stop


Edit2: Yup, because i didn't include the control codes for ending a text

Spoiler:

Cartographer always copied the whole text block until the very end of the file. Now it actually works properly:

Spoiler:

But, sadly the insert using Atlas still doesn't work:

Spoiler:

Spoiler:
« Last Edit: February 09, 2020, 11:11:42 pm by Risae »

abw

  • Hero Member
  • *****
  • Posts: 560
    • View Profile
Re: Using Atlas after Cartographer, confused
« Reply #1 on: March 09, 2019, 03:54:59 pm »
First off, congratulations on getting as far as you have! :thumbsup:

As for your insert problem, in your Cartographer script you specified "#COMMENTS: Yes", which makes Cartographer dump the script as Atlas comments (i.e. it prefaces every script line with "//"), so when you run Atlas, all it sees is a bunch of comments (which it ignores) and pointer writes (which it executes). If you delete the "//" at the start of the script lines, then your insert script should work much better.

Here are a couple of other tips:
- I've found that it is very useful to keep a copy of the original text as comments and have the translated text just below so that it looks like e.g.
Quote
//POINTER #0 @ $FF24 - STRING #0 @ $10EA3

#W32($FF24)
//original text
translated text

- Regarding your table file:
Quote
/FFFF00=<END>\n\n
/FFFE00=<END>\n\n
If the game uses separate control codes for things, there's usually a reason for that (and the reason is not always obvious), so unless you're absolutely certain the control codes really are completely interchangeable, you probably want to use different text to keep them separate, e.g.
Quote
/FFFF00=<END-FF>\n\n
/FFFE00=<END-FE>\n\n

Otherwise, it looks like you're well on your way - good luck!

Risae

  • Jr. Member
  • **
  • Posts: 73
    • View Profile
Re: Using Atlas after Cartographer, confused
« Reply #2 on: March 09, 2019, 10:52:46 pm »
Hi abw,

Thank you very much for your message.
You are the one who made the tool called "abcde", right?
I looked at abcde too, but it looked a little bit too complex for me so i chose to start with Cartographer and Atlas.
But thank you for creating another tool which supports these things, i think i will take another look at it once i understand Cartographer and Atlas better.

Quote
If the game uses separate control codes for things, there's usually a reason for that (and the reason is not always obvious), so unless you're absolutely certain the control codes really are completely interchangeable, you probably want to use different text to keep them separate, e.g.

I noticed that too, i included 2 diffent END Commands for the table file.
With that i made some nice progress, but i am currently having another issue with the text insertion using Atlas.
I searched a bit through the forum and i started to understand how Atlas works.
This is the script which i put in front of the Cartographer text:

Code: [Select]
#VAR(dialogue, TABLE)
#ADDTBL("sjis.tbl", dialogue)
#ACTIVETBL(dialogue)

#VAR(PTR, CUSTOMPOINTER)
#CREATEPTR(PTR, "LINEAR", $0, 32)

#VAR(PTRTBL, POINTERTABLE)
#PTRTBL(PTRTBL, $FF20, 4, PTR)

#JMP($10EA0)

#AUTOWRITE(PTRTBL, "<END-FF>")
#AUTOWRITE(PTRTBL, "<END-FE>")

//POINTER #0 @ $FF20 - STRING #0 @ $10EA0

#W32($FF20)
<END-FF>


//POINTER #1 @ $FF24 - STRING #1 @ $10EA5

#W32($FF24)
性<END-FF>
... and so on

It is actually creating a pointer table now, but it is not starting the table at 0:

Spoiler:

This is how it should look like:

Spoiler:

And an even bigger issue would be that the insert wrote the control codes as text, which is not good:

Spoiler:




Edit:
After a bit of trial n error i managed to fix the pointer issue, had to change
#W32($XXXX)
to
#WRITE(PTR, $XXXX)


Code: [Select]
#VAR(dialogue, TABLE)
#ADDTBL("sjis.tbl", dialogue)
#ACTIVETBL(dialogue)

#VAR(PTR, CUSTOMPOINTER)
#CREATEPTR(PTR, "LINEAR", $10EA0, 32)

#VAR(PTRTBL, POINTERTABLE)
#PTRTBL(PTRTBL, $FF20, 4, PTR)

#AUTOWRITE(PTRTBL, "<END-FF>")
#AUTOWRITE(PTRTBL, "<END-FE>")

#JMP($10EA0)

#WRITE(PTR, $FF20)
<END-FF>


//POINTER #1 @ $FF24 - STRING #1 @ $10EA3

#WRITE(PTR, $FF24)
女性<END-FF>


//POINTER #2 @ $FF28 - STRING #2 @ $10EAA

#WRITE(PTR, $FF28)
右スティック操作でマップ確認モード<END-FF>

The <END-FF> and <END-FE> issue still persists, but the pointers are working correctly now:

Spoiler:

Spoiler:

Anybody have an idea why the control codes are not properly inserted?
This is the table file i am using for the insertion:


Code: [Select]
20=
21=!
22="
23=#
24=$
25=%
26=&
27='
28=(
29=)
2A=*
2B=+
2C=,
2D=-
2E=.
2F=/
30=0
31=1
32=2
33=3
34=4
35=5
36=6
37=7
38=8
39=9
.......
EAA1=ê¡
EAA2=ê¢
/FFFF00=<END-FF>
/FFFE00=<END-FE>
« Last Edit: March 11, 2019, 03:25:04 am by Risae »

Risae

  • Jr. Member
  • **
  • Posts: 73
    • View Profile
Re: Using Atlas after Cartographer, confused
« Reply #3 on: March 10, 2019, 09:28:32 am »
I figured it out:

Atlas completely ignored the AUTOWRITE command.

Code: [Select]
#AUTOWRITE(PTRTBL, "<END-FF>")
#AUTOWRITE(PTRTBL, "<END-FE>")

were never used, Atlas wrote the text, calculated it and then put a pointer there and went to the next text.
Aaaaaah, no idea what i did wrong for it to completely ignore the autowrite commands, but when i changed all of

<END-FF>
to
<$FF><$FF><$00>

<END-FE>
to
<$FF><$FE><$00>

it worked without a flaw.
I think i programmed my script the wrong way, but at least it works for now.

Edit:
The proof of concept seems to work like a charm:

https://youtu.be/B4UrFwc07oA


This is the current Atlas script i am using:

Code: [Select]
#VAR(dialogue, TABLE)
#ADDTBL("sjis.tbl", dialogue)
#ACTIVETBL(dialogue)

#VAR(PTR, CUSTOMPOINTER)
#CREATEPTR(PTR, "LINEAR", $10EA0, 32)

#VAR(PTRTBL, POINTERTABLE)
#PTRTBL(PTRTBL, $FF20, 4, PTR)

#JMP($10EA0)

//POINTER #0 @ $FF20 - STRING #0 @ $10EA0
#WRITE(PTR, $FF20)
<$FF><$FF><$00>


//POINTER #1 @ $FF24 - STRING #1 @ $10EA3
#WRITE(PTR, $FF24)
//女性<$FF><$FF><$00>
Woman<$FF><$FF><$00>

Anybody got an idea how to get autowrite to actually work?
would be easier for me to just use autowrite instead of  replacing all of it in a file.
« Last Edit: March 10, 2019, 02:49:03 pm by Risae »

abw

  • Hero Member
  • *****
  • Posts: 560
    • View Profile
Re: Using Atlas after Cartographer, confused
« Reply #4 on: March 10, 2019, 04:50:06 pm »
You are the one who made the tool called "abcde", right?
I looked at abcde too, but it looked a little bit too complex for me so i chose to start with Cartographer and Atlas.
But thank you for creating another tool which supports these things, i think i will take another look at it once i understand Cartographer and Atlas better.
Yup, that's me. abcde is mostly intended to handle games with more complicated scripts than Cartographer/Atlas can cope with, but if you don't actually need abcde's extra power, Cartographer and Atlas should be fine.

#AUTOWRITE with a POINTERTABLE and a CUSTOMPOINTER is great if you're starting from an empty insert script, but with the #W32 commands Cartographer outputs for you, I find it's easier to leave the pointer commands alone and just add an appropriate #HDR command before the first pointer write in order to fix up the address calculation. If your script starts at $10EA0 and you want the first pointer to be $00000000, then a single #HDR($10EA0) command should do the trick.

Atlas not recognizing your end tokens is weird. Is there any chance your SJIS table file is using the 2-byte encoding for ASCII characters?

Anyway, the important thing is you got it to work!

Risae

  • Jr. Member
  • **
  • Posts: 73
    • View Profile
Re: Using Atlas after Cartographer, confused
« Reply #5 on: March 11, 2019, 12:41:34 pm »
Hello abw,

Thank you very much for your reply.
i just tested it with your suggestions and it seems to work now
(but i dont actually have to use the AUTOWRITE Command  :laugh:)

Atlas Script before:

Code: [Select]
#VAR(dialogue, TABLE)
#ADDTBL("sjis.tbl", dialogue)
#ACTIVETBL(dialogue)

#VAR(PTR, CUSTOMPOINTER)
#CREATEPTR(PTR, "LINEAR", $10EA0, 32)

#VAR(PTRTBL, POINTERTABLE)
#PTRTBL(PTRTBL, $FF20, 4, PTR)

#AUTOWRITE(PTRTBL, "<END-FF>")
#AUTOWRITE(PTRTBL, "<END-FE>")

#JMP($10EA0)

//POINTER #0 @ $FF20 - STRING #0 @ $10EA0

#W32($FF20)
<END-FF>


//POINTER #1 @ $FF24 - STRING #1 @ $10EA3

#W32($FF24)
This is a test to see if it works<END-FF>


//POINTER #2 @ $FF28 - STRING #2 @ $10EAA

#W32($FF28)
右スティック操作でマップ確認モード<END-FF>

Atlas Script after:

Code: [Select]
#VAR(dialogue, TABLE)
#ADDTBL("sjis.tbl", dialogue)
#ACTIVETBL(dialogue)

#VAR(PTR, CUSTOMPOINTER)
#CREATEPTR(PTR, "LINEAR", $10EA0, 32)

#VAR(PTRTBL, POINTERTABLE)
#PTRTBL(PTRTBL, $FF20, 4, PTR)

#JMP($10EA0)
#HDR($10EA0)

//POINTER #0 @ $FF20 - STRING #0 @ $10EA0

#W32($FF20)
<END-FF>


//POINTER #1 @ $FF24 - STRING #1 @ $10EA3

#W32($FF24)
This is a test to see if it works<END-FF>


//POINTER #2 @ $FF28 - STRING #2 @ $10EAA

#W32($FF28)
右スティック操作でマップ確認モード<END-FF>

Atlas Table before:

Code: [Select]
20=
21=!
22="
23=#
24=$
25=%
26=&
...
EAA0=槇
EAA1=遙
EAA2=瑤
/FFFF00=<END-FF>
/FFFE00=<END-FE>

Altas Table after:

Code: [Select]
20=
21=!
22="
23=#
24=$
25=%
26=&
...
EAA0=槇
EAA1=遙
EAA2=瑤
FFFC=[NLINE]
FFFD=[NWIN]
FFFF00=<END-FF>
FFFE00=<END-FE>

Pointer calculation works properly and starts at 0

Spoiler:

Spoiler:

I didn't test the insertion in game yet, but so far i didn't see any issues looking through the file in the hex editor.
Atlas and Cartopraph really are some awesome tools, after i included the control codes for NEW LINE and NEW WINDOW the dumped script looks a lot better:

before

Spoiler:

after

Spoiler:


Edit:
I just inserted the modified lines back into the game and checked if it works like intended - everything seems to work, the game didn't break.
The only thing that bothers me right now is the fact that, when i don't change anything and re-insert the script, the total length changed and the last line is at a different position.
The inserted text is shorter in total than the original file.
But, so far i didnt experience any crashes or anomalies, so it might be okay after all.
I changed the thread title to "SOLVED", should any issues arise again i will write a message here.
Thank you very much for the help!
« Last Edit: March 12, 2019, 12:48:44 pm by Risae »

abw

  • Hero Member
  • *****
  • Posts: 560
    • View Profile
Re: Using Atlas after Cartographer, confused
« Reply #6 on: March 12, 2019, 09:12:51 am »
i just tested it with your suggestions and it seems to work now
You're welcome, and hurray!

Edit:
I just inserted the modified lines back into the game and checked if it works like intended - everything seems to work, the game didn't break.
The only thing that bothers me right now is the fact that, when i don't change anything and re-insert the script, the total length changed and the last line is at a different position.
The inserted text is shorter in total than the original file.
But, so far i didnt experience any crashes or anomalies, so it might be okay after all.

That can happen in games that use some sort of simple compression scheme that you can represent in a table file (e.g. dictionary/DTE) where the original data was poorly compressed and the insertion utility (Atlas in this case) finds a shorter way of compressing the same text. For a purely Shift-JIS script, though, I don't think that should be happening. You can try diffing the original and modified files in a hex editor to see what the differences are. For a PS2 game, my guess is that the original strings were word-aligned (i.e. they all started at an address that's a multiple of 4 bytes) and the inserted strings are not; if that's the case, you can add a #STRINGALIGN(4) command and put the "/" back on your <END-FE> and <END-FF> table entries so that Atlas knows where each string stops.

Risae

  • Jr. Member
  • **
  • Posts: 73
    • View Profile
Re: [SOLVED] Using Atlas after Cartographer, confused
« Reply #7 on: March 12, 2019, 12:47:11 pm »
Hm, i think i understood what you mean.
After checking the before and after files again i saw that Atlas is not inputting the "dot" correctly:

Before text insertion



After text insertion



But i honestly don't know why, Cartographer was able to dump the script without an issue:



And the table file that i'm using to insert the text also has the correct hex code for the dot:



I will try to include the #STRINGALIGN(4) and put back the / so see if this might fix it

Edit:
Nope, still breaks the file, even with all the settings:










Gonna try to remove the dot code in the table and see what happens

Edit2:
Hmmm, when i put the hex value of the dot in the place where the dot was in the script, it will place it correctly.
But, after comparing both the original and the modified text file, there is more missing than i first thought:

http://www.framecompare.com/image-compare/screenshotcomparison/11EFNNNU

Well, i don't think i have to break my head over this, since i am translating the game to english and won't use any special symbols.
« Last Edit: March 12, 2019, 01:39:02 pm by Risae »

abw

  • Hero Member
  • *****
  • Posts: 560
    • View Profile
Re: [SOLVED] Using Atlas after Cartographer, confused
« Reply #8 on: March 13, 2019, 07:18:11 pm »
Hmm, which Shift-JIS table file are you using? Assuming it's this one, it looks like there are multiple table file entries for the "・" (5C and 8145) and "ッ" (7E and 8362) characters, so for those characters, Atlas is choosing to use the shorter one-byte table entry. In that case, I'd say this is an issue with the table file rather than Atlas.

As you said, since you're going to be translating the game to English, you probably don't need to worry too much about Japanese-specific insert issues, but I always like to try inserting the same data I've dumped just to test out my insert process. For games with non-standard encodings, this sometimes reveals subtleties that I would have otherwise overlooked, but for a Shift-JIS game, hopefully you'll be pretty safe.

Risae

  • Jr. Member
  • **
  • Posts: 73
    • View Profile
Re: [SOLVED] Using Atlas after Cartographer, confused
« Reply #9 on: March 14, 2019, 02:56:25 am »
Quote
Hmm, which Shift-JIS table file are you using? Assuming it's this one, it looks like there are multiple table file entries for the "・" (5C and 8145) and "ッ" (7E and 8362) characters, so for those characters, Atlas is choosing to use the shorter one-byte table entry.


Oooooh, this might actually be the reason. You are completely right, i used the SHIFT-JIS Table that you linked to, and i didn't notice that the small ツ and the dot had multiple entries.
I will try to remove the wrong ones and check the re-insert again.
Thank you very much!


Edit: Yup, that fixed the import.
I also found a bug in my Cartoprapher script, i used the wrong offset for the last pointer and it didn't dump that.
Thank you very much for your help!
« Last Edit: March 16, 2019, 03:33:53 am by Risae »