News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: [SOLVED][Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde  (Read 934 times)

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Hi!

I am currently trying to switch from using Atlas + Cartographer to abcde, but i am having issues with using my Shift JIS table that i have, which is currently encoded in Shift-JIS.
Does anybody know how i can transfer/convert my table from SJIS to UTF so i can use it in abcde?
« Last Edit: April 06, 2020, 10:44:57 am by Risae »

Disch

  • Hero Member
  • *****
  • Posts: 2809
  • NES Junkie
    • View Profile
Re: [Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde
« Reply #1 on: April 05, 2020, 04:16:49 pm »
Your table is just a text file, right?

I found this tool to do it online:

http://codepage-encoding.online-domain-tools.com/

1)  change input type to 'File'
2)  select SHIFT_JIS from source code page and UTF-8 as target code page
3)  'Browse' to upload your table file
4)  'Convert!'
5)  Click 'download as a binary file' beneath the text area (it's small -- easy to miss)

The binary file is actually a text file, you'll just have to change the extension and it should be good to go.


You can also change encodings with Notepad++ if you have that.  And I think there's a way to do it via commandline if you're on linux.

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: [Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde
« Reply #2 on: April 05, 2020, 05:21:20 pm »
Hi Disch!

after writing my thread i actually noticed that Notepad++ had the option to convert my S-JIS Table file to UTF-8.
It seemed to work without an issue but now abcde gives me a new error:

Code: [Select]
00-abcde>perl abcde.pl -cm abcde::Cartographer 00000080.flk GL6_cartographer_commands.txt GL6_dumped_script_000 -s
incorrect mode detected; switching to bin2text! at 00-abcde/abcde/Cartographer.pm line 57.
unrecognized table entry '3C=<' at '00-abcde\test.tbl' line 29 (invalid token format)

I am not entirely sure why abcde doesn't like this line, this is the converted table i'm using right now:

Code: [Select]
20=
21=!
22="
23=#
24=$
25=%
26=&
27='
28=(
29=)
2A=*
2B=+
2C=,
2D=-
2E=.
2F=/
30=0
31=1
32=2
33=3
34=4
35=5
36=6
37=7
38=8
39=9
3A=:
3B=;
3C=<
3D==
3E=>
3F=?
40=@
41=A
42=B
43=C
44=D
45=E
46=F
47=G
48=H
49=I
4A=J
4B=K
4C=L
4D=M
4E=N
4F=O
50=P
51=Q
52=R
53=S
54=T
55=U
56=V
57=W
58=X
59=Y
5A=Z
5B=[
5D=]
5E=^
5F=_
60=`
61=a
62=b
63=c
64=d
65=e
66=f
67=g
68=h
69=i
6A=j
6B=k
6C=l
6D=m
6E=n
6F=o
70=p
71=q
72=r
73=s
74=t
75=u
76=v
77=w
78=x
79=y
7A=z
7B={
7C=|
7D=}
7E=~
8140= 
8141=、
8142=。
8143=,
8144=.
8145=・
8146=:
8147=;
8148=?
8149=!
814A=゛
814B=゜
814C=´
814D=`
814E=¨
814F=^
8150= ̄
8151=_
8152=ヽ
8153=ヾ
8154=ゝ
8155=ゞ
8156=〃
8157=仝
8158=々
8159=〆
815A=〇
815B=ー
815C=―
815D=‐
815E=/
815F=\
8160=~
8161=∥
8162=|
8163=…
8164=‥
8165=‘
8166=’
8167=“
8168=”
8169=(
816A=)
816B=〔
816C=〕
816D=[
816E=]
816F={
8170=}
8171=〈
8172=〉
8173=《
8174=》
8175=「
8176=」
8177=『
8178=』
8179=【
817A=】
817B=+
817C=-
817D=±
817E=×
8180=÷
8181==
8182=≠
8183=<
8184=>
8185=≦
8186=≧
8187=∞
8188=∴
8189=♂
818A=♀
818B=°
818C=′
818D=″
818E=℃
818F=¥
8190=$
8191=¢
8192=£
8193=%
8194=#
8195=&
8196=*
8197=@
8198=§
8199=☆
819A=★
819B=○
819C=●
819D=◎
819E=◇
81A0=□
81A1=■
81A2=△
81A3=▲
81A4=▽
81A5=▼
81A6=※
81A7=〒
81A8=→
81A9=←
81AA=↑
81AB=↓
81AC=〓
824F=0
8250=1
8251=2
8252=3
8253=4
8254=5
8255=6
8256=7
8257=8
8258=9
8260=A
8261=B
8262=C
8263=D
8264=E
8265=F
8266=G
8267=H
8268=I
8269=J
826A=K
826B=L
826C=M
826D=N
826E=O
826F=P
8270=Q
8271=R
8272=S
8273=T
8274=U
8275=V
8276=W
8277=X
8278=Y
8279=Z
8281=a
8282=b
8283=c
8284=d
8285=e
8286=f
8287=g
8288=h
8289=i
828A=j
828B=k
828C=l
828D=m
828E=n
828F=o
8290=p
8291=q
8292=r
8293=s
8294=t
8295=u
8296=v
8297=w
8298=x
8299=y
829A=z
829F=ぁ
82A0=あ
82A1=ぃ
82A2=い
82A3=ぅ
82A4=う
82A5=ぇ
82A6=え
82A7=ぉ
82A8=お
82A9=か
82AA=が
82AB=き
82AC=ぎ
82AD=く
82AE=ぐ
82AF=け
82B0=げ
82B1=こ
82B2=ご
82B3=さ
82B4=ざ
82B5=し
82B6=じ
82B7=す
82B8=ず
82B9=せ
82BA=ぜ
82BB=そ
82BC=ぞ
82BD=た
82BE=だ
82BF=ち
82C0=ぢ
82C1=っ
82C2=つ
82C3=づ
82C4=て
82C5=で
82C6=と
82C7=ど
82C8=な
82C9=に
82CA=ぬ
82CB=ね
82CC=の
82CD=は
82CE=ば
82CF=ぱ
82D0=ひ
82D1=び
82D2=ぴ
82D3=ふ
82D4=ぶ
82D5=ぷ
82D6=へ
82D7=べ
82D8=ぺ
82D9=ほ
82DA=ぼ
82DB=ぽ
82DC=ま
82DD=み
82DE=む
82DF=め
82E0=も
82E1=ゃ
82E2=や
82E3=ゅ
82E4=ゆ
82E5=ょ
82E6=よ
82E7=ら
82E8=り
82E9=る
82EA=れ
82EB=ろ
82EC=ゎ
82ED=わ
82EE=ゐ
82EF=ゑ
82F0=を
82F1=ん
8340=ァ
8341=ア
8342=ィ
8343=イ
8344=ゥ
8345=ウ
8346=ェ
8347=エ
8348=ォ
8349=オ
834A=カ
834B=ガ
834C=キ
834D=ギ
834E=ク
834F=グ
8350=ケ
8351=ゲ
8352=コ
8353=ゴ
8354=サ
8355=ザ
8356=シ
8357=ジ
8358=ス
8359=ズ
835A=セ
835B=ゼ
835C=ソ
835D=ゾ
835E=タ
835F=ダ
8360=チ
8361=ヂ
8362=ッ
8363=ツ
8364=ヅ
8365=テ
8366=デ
8367=ト
8368=ド
8369=ナ
836A=ニ
836B=ヌ
836C=ネ
836D=ノ
836E=ハ
836F=バ
8370=パ
8371=ヒ
8372=ビ
8373=ピ
8374=フ
8375=ブ
8376=プ
8377=ヘ
8378=ベ
8379=ペ
837A=ホ
837B=ボ
837C=ポ
837D=マ
837E=ミ
8380=ム
8381=メ
8382=モ
8383=ャ
8384=ヤ
8385=ュ
8386=ユ
8387=ョ
8388=ヨ
8389=ラ
838A=リ
838B=ル
838C=レ
838D=ロ
838E=ヮ
838F=ワ
8390=ヰ
8391=ヱ
8392=ヲ
8393=ン
8394=ヴ
8395=ヵ
8396=ヶ
839E=Α
83A0=Β
83A1=Γ
83A2=Δ
83A3=Ε
83A4=Ζ
83A5=Η
83A6=Θ
83A7=Ι
83A8=Κ
83A9=Λ
83AA=Μ
83AB=Ν
83AC=Ξ
83AD=Ο
83AE=Π
83AF=Ρ
83B0=Σ
83B1=Τ
83B2=Υ
83B3=Φ
83B4=Χ
83B5=Ψ
83B6=Ω
83BF=α
83C0=β
83C1=γ
83C2=δ
83C3=ε
83C4=ζ
83C5=η
83C6=θ
83C7=ι
83C8=κ
83C9=λ
83CA=μ
83CB=ν
83CC=ξ
83CD=ο
83CE=π
83CF=ρ
83D0=σ
83D1=τ
83D2=υ
83D3=φ
83D4=χ
83D5=ψ
83D6=ω
8440=А
8441=Б
8442=В
8443=Г
8444=Д
8445=Е
8446=Ё
8447=Ж
8448=З
8449=И
844A=Й
844B=К
844C=Л
844D=М
844E=Н
844F=О
8450=П
8451=Р
8452=С
8453=Т
8454=У
8455=Ф
8456=Х
8457=Ц
8458=Ч
8459=Ш
845A=Щ
845B=Ъ
845C=Ы
845D=Ь
845E=Э
845F=Ю
8460=Я
8470=а
8471=б
8472=в
8473=г
8474=д
8475=е
8476=ё
8477=ж
8478=з
8479=и
847A=й
847B=к
847C=л
847D=м
847E=н
8480=о
8481=п
8482=р
8483=с
8484=т
8485=у
8486=ф
8487=х
8488=ц
8489=ч
848A=ш
848B=щ
848C=ъ
848D=ы
848E=ь
848F=э
8490=ю
8491=я
889F=亜
88A0=唖
88A1=娃
88A2=阿
88A3=哀
...bunch of kanji...
EA9F=堯
EAA0=槇
EAA1=遙
EAA2=瑤
FFCF0000=[V0000]\n//
FFCF0100=[V0100]\n//
FFCF0200=[V0200]\n//
FFCF0300=[V0300]\n//
FFCF0400=[V0400]\n//
FFCF0500=[V0500]\n//
FFCF0600=[V0600]\n//
FFCF0700=[V0700]\n//
FFCF0800=[V0800]\n//
FFCF0900=[V0900]\n//
... bunch of control codes...
FFCEDE03=[CHARDE03]
FFCEDF03=[CHARDF03]
FFCEE003=[CHARE003]
FFCEE103=[CHARE103]
FFCEE203=[CHARE203]
FFCEE303=[CHARE303]
FFCEE403=[CHARE403]
FFCEE503=[CHARE503]
FFCEE603=[CHARE603]
FFCEE703=[CHARE703]
FFCEE803=[CHARE803]
FFCEE903=[CHARE903]
FFCEEA03=[CHAREA03]
FFCEEB03=[CHAREB03]
FFCEEC03=[CHAREC03]
FFCEED03=[CHARED03]
FFCEEE03=[CHAREE03]
FFCEEF03=[CHAREF03]
FFCEF003=[CHARF003]
FFCEF103=[CHARF103]
FFCEF203=[CHARF203]
FFCEF303=[CHARF303]
FFCEF403=[CHARF403]
FFCEF503=[CHARF503]
FFCEF603=[CHARF603]
FFCEF703=[CHARF703]
FFCEF803=[CHARF803]
FFCEF903=[CHARF903]
FFCEFA03=[CHARFA03]
FFCEFB03=[CHARFB03]
FFCEFC03=[CHARFC03]
FFCEFD03=[CHARFD03]
FFCEFE03=[CHARFE03]
FFCEFF03=[CHARFF03]
FFB000=[CC.B000]
FFB100=[CC.B100]
FFB200=[CC.B200]
FFB300=[CC.B300]
FFB400=[CC.B400]
FFB500=[CC.B500]
FFB600=[CC.B600]
FFB700=[CC.B700]
FFB800=[CC.B800]
FFB900=[CC.B900]
FFBA00=[CC.BA00]
FFBB00=[CC.BB00]
FFBC00=[CC.BC00]
FFBD00=[CC.BD00]
FFBE00=[CC.BE00]
FFBF00=[CC.BF00]
FFEF00=[CC.EF00]
FFEF01=[CC.EF01]
FFEF02=[CC.EF02]
FFEF03=[CC.EF03]
FFEF04=[CC.EF04]
FFEF05=[CC.EF05]
FFEF06=[CC.EF06]
FFEF07=[CC.EF07]
FFEF08=[CC.EF08]
FFEF09=[CC.EF09]
FFEF0A=[CC.EF0A]
FFEF0B=[CC.EF0B]
FFEF0C=[CC.EF0C]
FFEF0D=[CC.EF0D]
FFEF0E=[CC.EF0E]
FFEF0F=[CC.EF0F]
FFEF10=[CC.EF10]
FFEF11=[CC.EF11]
FFEF12=[CC.EF12]
FFEF13=[CC.EF13]
FFEF14=[CC.EF14]
FFEF15=[CC.EF15]
FFEF16=[CC.EF16]
FFEF17=[CC.EF17]
FFEF18=[CC.EF18]
FFEF19=[CC.EF19]
FFEF1A=[CC.EF1A]
FFEF1B=[CC.EF1B]
FFEF1C=[CC.EF1C]
FFEF1D=[CC.EF1D]
FFEF1E=[CC.EF1E]
FFEF1F=[CC.EF1F]
FFAC00=[CC.AC00]
FFAC01=[CC.AC01]
FFAC02=[CC.AC02]
FFAC03=[CC.AC03]
FFAC04=[CC.AC04]
FFAC05=[CC.AC05]
FFAC06=[CC.AC06]
FFAC07=[CC.AC07]
FFAC08=[CC.AC08]
FFAC09=[CC.AC09]
FFAC0A=[CC.AC0A]
FFAC0B=[CC.AC0B]
FFAC0C=[CC.AC0C]
FFAC0D=[CC.AC0D]
FFAC0E=[CC.AC0E]
FFAC0F=[CC.AC0F]
FFD800=[CC.D800]
FFD801=[CC.D801]
FFD802=[CC.D802]
FFD803=[CC.D803]
FFD804=[CC.D804]
FFD805=[CC.D805]
FFD806=[CC.D806]
FFD807=[CC.D807]
FFD808=[CC.D808]
FFD809=[CC.D809]
FFD80A=[CC.D80A]
FFD80B=[CC.D80B]
FFD80C=[CC.D80C]
FFD80D=[CC.D80D]
FFD80E=[CC.D80E]
FFD80F=[CC.D80F]
FFD810=[CC.D810]
FFD811=[CC.D811]
FFD812=[CC.D812]
FFD813=[CC.D813]
FFD814=[CC.D814]
FFD815=[CC.D815]
FFD816=[CC.D816]
FFD817=[CC.D817]
FFD818=[CC.D818]
FFD819=[CC.D819]
FFD81A=[CC.D81A]
FFD81B=[CC.D81B]
FFD81C=[CC.D81C]
FFD81D=[CC.D81D]
FFD81E=[CC.D81E]
FFD81F=[CC.D81F]
FFAA0000=[CC.AA0000]
FFAA0100=[CC.AA0100]
FFAA0200=[CC.AA0200]
FFAA0300=[CC.AA0300]
FFAA0400=[CC.AA0400]
FFAA0500=[CC.AA0500]
FFAA0600=[CC.AA0600]
FFAA0700=[CC.AA0700]
FFAA0800=[CC.AA0800]
FFAA0900=[CC.AA0900]
FFAA0A00=[CC.AA0A00]
FFAA0B00=[CC.AA0B00]
FFAA0C00=[CC.AA0C00]
FFAA0D00=[CC.AA0D00]
FFAA0E00=[CC.AA0E00]
FFAA0F00=[CC.AA0F00]
FFAA0001=[CC.AA0001]
FFAA0002=[CC.AA0002]
FFAA0003=[CC.AA0003]
FFAA0004=[CC.AA0004]
FFAA0005=[CC.AA0005]
FFAA0006=[CC.AA0006]
FFAA0007=[CC.AA0007]
FFAA0008=[CC.AA0008]
FFAA0009=[CC.AA0009]
FFAA000A=[CC.AA000A]
FFAA000B=[CC.AA000B]
FFAA000C=[CC.AA000C]
FFAA000D=[CC.AA000D]
FFAA000E=[CC.AA000E]
FFAA000F=[CC.AA000F]
FFEA0000=[CC.EA0000]
FFEA0100=[CC.EA0100]
FFEA0200=[CC.EA0200]
FFEA0300=[CC.EA0300]
FFEA0400=[CC.EA0400]
FFEA0500=[CC.EA0500]
FFEA0600=[CC.EA0600]
FFEA0700=[CC.EA0700]
FFEA0800=[CC.EA0800]
FFEA0900=[CC.EA0900]
FFEA0A00=[CC.EA0A00]
FFEA0B00=[CC.EA0B00]
FFEA0C00=[CC.EA0C00]
FFEA0D00=[CC.EA0D00]
FFEA0E00=[CC.EA0E00]
FFEA0F00=[CC.EA0F00]
FFBE0000=[CC.BE0000]
FFBE0001=[CC.BE0001]
FFBE0002=[CC.BE0002]
FFBE0003=[CC.BE0003]
FFBE0004=[CC.BE0004]
FFBE0005=[CC.BE0005]
FFBE0006=[CC.BE0006]
FFBE0007=[CC.BE0007]
FFBE0008=[CC.BE0008]
FFBE0009=[CC.BE0009]
FFBE000A=[CC.BE000A]
FFBE000B=[CC.BE000B]
FFBE000C=[CC.BE000C]
FFBE000D=[CC.BE000D]
FFBE000E=[CC.BE000E]
FFBE000F=[CC.BE000F]
FFBE0100=[CC.BE0100]
FFBE0101=[CC.BE0101]
FFBE0102=[CC.BE0102]
FFBE0103=[CC.BE0103]
FFBE0104=[CC.BE0104]
FFBE0105=[CC.BE0105]
FFBE0106=[CC.BE0106]
FFBE0107=[CC.BE0107]
FFBE0108=[CC.BE0108]
FFBE0109=[CC.BE0109]
FFBE010A=[CC.BE010A]
FFBE010B=[CC.BE010B]
FFBE010C=[CC.BE010C]
FFBE010D=[CC.BE010D]
FFBE010E=[CC.BE010E]
FFBE010F=[CC.BE010F]
FFBF6000=[CC.BF6000]
FFBF6100=[CC.BF6100]
FFBF6200=[CC.BF6200]
FFBF6300=[CC.BF6300]
FFBF6400=[CC.BF6400]
FFBF6500=[CC.BF6500]
FFBF6600=[CC.BF6600]
FFBF6700=[CC.BF6700]
FFBF6800=[CC.BF6800]
FFBF6900=[CC.BF6900]
FFBF6A00=[CC.BF6A00]
FFBF6B00=[CC.BF6B00]
FFBF6C00=[CC.BF6C00]
FFBF6D00=[CC.BF6D00]
FFBF6E00=[CC.BF6E00]
FFBF6F00=[CC.BF6F00]
FFEAE0FF=[CC.EAE0FF]
FFEAE1FF=[CC.EAE1FF]
FFEAE2FF=[CC.EAE2FF]
FFEAE3FF=[CC.EAE3FF]
FFEAE4FF=[CC.EAE4FF]
FFEAE5FF=[CC.EAE5FF]
FFEAE6FF=[CC.EAE6FF]
FFEAE7FF=[CC.EAE7FF]
FFEAE8FF=[CC.EAE8FF]
FFEAE9FF=[CC.EAE9FF]
FFEAEAFF=[CC.EAEAFF]
FFEAEBFF=[CC.EAEBFF]
FFEAECFF=[CC.EAECFF]
FFEAEDFF=[CC.EAEDFF]
FFEAEEFF=[CC.EAEEFF]
FFEAEFFF=[CC.EAEFFF]
FFCD=[CC.FFCD]
FFFB=[CC.FFFB]
FFDF00=[COL00]
FFDF01=[COL01]
FFDF02=[COL02]
FFDF03=[COL03]
FFDF04=[COL04]
FFDF05=[COL05]
FFDF06=[COL06]
FFDF07=[COL07]
FFDF08=[COL08]
FFDF09=[COL09]
FFDF0A=[COL0A]
FFDF0B=[COL0B]
FFDF0C=[COL0C]
FFDF0D=[COL0D]
FFDF0E=[COL0E]
FFDF0F=[COL0F]
FFDF10=[COL10]
81CE=[TEAR-DROP]
81CD=[HEART]
81F4=[MUSIC-NOTE]
8740=[(1)]
8741=[(2)]
8742=[(3)]
8743=[(4)]
8744=[(5)]
8745=[(6)]
8746=[(7)]
8747=[(8)]
8748=[(9)]
8749=[(19)]
874A=[(11)]
874B=[(12)]
874C=[(13)]
874D=[(14)]
874E=[(15)]
874F=[(16)]
8750=[(17)]
8751=[(18)]
8752=[(19)]
8753=[(20)]
8754=[ONE]
8755=[TWO]
8756=[THREE]
8757=[FOUR]
8758=[FIVE]
8759=[SIX]
875A=[SEVEN]
875B=[EIGTH]
875C=[NINE]
875D=[TEN]
84A5=[CIRCLE]
84A6=[CROSS]
84A7=[SQUARE]
84A8=[TRIANGLE]
84A184A2=[L1]
84A384A4=[L2]
84AC84AD=[R1]
84AE84AF=[R2]
84B084B184B2=[START]
84B584B684B7=[SELECT]
FFFC=[NLINE]\n//
FFFD=[NWIN]\n//\n//
/FFFF00=<END-FF>\n\n\n\n\n
/FFFE00=<END-FE>\n\n\n\n\n

Full table can be found here: https://pastebin.com/pzqUDpMF
Anybody know why abcde could give this error?

Disch

  • Hero Member
  • *****
  • Posts: 2809
  • NES Junkie
    • View Profile
Re: [Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde
« Reply #3 on: April 05, 2020, 07:06:04 pm »
I can't really help very much with abcde specifics as I've never used it, but judging from the error message it doesn't like that one of the table entries is '<'.  Maybe it uses < for some internal thing and it needs to be escaped?

If there's nothing mentioned in abcde's documentation about escaping table file entries, then I would try changing it to '3C=\<' or just getting rid of the <> symbols entirely and replacing them with something else.

But I'm past the limit of my knowledge here so I can't really help any more than that.   :beer:

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: [Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde
« Reply #4 on: April 05, 2020, 07:59:13 pm »
Hi Disch,

thank you very much for your reply.
It seems like i overread that in the abdce ReadMe, "<" and ">" are used by the program:

Quote
# * "<" and ">" are reserved for various purposes including representing untranslated data and delimiting sections of table entries that need to contain multiple pieces of encoding information.

i commented them and had to change my ending text from <END-FE> to [END-FE].

Now it seems to work like a charm:

Spoiler:
//GAME NAME:      Growlanser 6

// Define required TABLE variables and load the corresponding tables
#VAR(Table_0, TABLE)
#ADDTBL("00-abcde\test.tbl", Table_0)

//BLOCK #000 NAME:      Dialogue Block (POINTER_RELATIVE)
#ACTIVETBL(Table_0) // Activate this block's starting TABLE

#JMP($2790, $5624) // Jump to insertion point
#HDR($2790) // Difference between ROM and RAM addresses for pointer value calculations

//POINTER #0 @ $2320 - STRING #0 @ $2790
#W32($2320)
//[END-FF]
[END-FF]
// current address: $2793

//POINTER #1 @ $2324 - STRING #1 @ $2793
#W32($2324)
//%u3010%u30C7%u30D0%u30C3%u30B0%u3011%u3059%u3079%u3066%u8868%u793A%u3059%u308B%uFF1F[END-FE]
%u3010%u30C7%u30D0%u30C3%u30B0%u3011%u3059%u3079%u3066%u8868%u793A%u3059%u308B%uFF1F[END-FE]
// current address: $27B2

//POINTER #2 @ $2328 - STRING #2 @ $27B2
#W32($2328)
//%u305F%u3060%u3044%u307E%u3001%u6B21%u56DE%u5927%u4F1A%u3078%u5411%u3051%u3066%u306E%u6E96%u5099%u4E2D%u3067%u3059%u3002[NLINE]
//%u958B%u50AC%u3055%u308C%u305F%u3089%u8155%u8A66%u3057%u306B%u6765%u3066%u304F%u3060%u3055%u3044%u306D%uFF01[END-FE]
%u305F%u3060%u3044%u307E%u3001%u6B21%u56DE%u5927%u4F1A%u3078%u5411%u3051%u3066%u306E%u6E96%u5099%u4E2D%u3067%u3059%u3002[NLINE]
%u958B%u50AC%u3055%u308C%u305F%u3089%u8155%u8A66%u3057%u306B%u6765%u3066%u304F%u3060%u3055%u3044%u306D%uFF01[END-FE]
// current address: $2803

But i have another question:

I configured my table file with End statements that are suppored to include new lines:

/FFFF00=[END-FF]\n\n\n\n\n\n\n
/FFFE00=[END-FE]\n\n\n\n\n\n\n

But its not writing those. Are new lines not possible in End statements?

And, is it possible to remove the writing of the lines "// current address: $27B2..."?

April 05, 2020, 08:11:56 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
But i have another question:

I configured my table file with End statements that are suppored to include new lines:

/FFFF00=[END-FF]\n\n\n\n\n\n\n
/FFFE00=[END-FE]\n\n\n\n\n\n\n

But its not writing those. Are new lines not possible in End statements?

I believe i found the reason for the new lines not working in abcde:

Quote
# Since newlines are reserved for delimiting table entries from each other, you can't make a multi-line table entry, but you can use "\n" anywhere in the right-hand side of a table entry to represent a newline:
FE=new\nline
# 0xFE would then be translated to text as:
# new
# line
# Newlines are ignored when translating text into binary, so in the absence of any other usable table entries, each of these texts would be translated to 0xFE:
# 1)
# new
# line
# 2)
# new
#
#
# line
# 3)
# newline
# 4)
# ne
# wli
# ne
# 5) etc., etc.

Thats kinda sad to see a good feature not working in abcde.
« Last Edit: April 05, 2020, 08:13:33 pm by Risae »

abw

  • Sr. Member
  • ****
  • Posts: 426
    • View Profile
Re: [Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde
« Reply #5 on: April 05, 2020, 10:54:31 pm »
It seems like i overread that in the abdce ReadMe, "<" and ">" are used by the program:

Quote
# * "<" and ">" are reserved for various purposes including representing untranslated data and delimiting sections of table entries that need to contain multiple pieces of encoding information.
i commented them and had to change my ending text from <END-FE> to [END-FE].
In table files, < and > are reserved partly for table switch syntax and partly to ensure separation between raw hexadecimal/binary and translated text (e.g. if your game actually contains the text "<$00>" you want that to stay encoded as 5 text characters and not become a $00 byte). It is kind of annoying when your text contains < characters, though; as a work around, you can substitute a different character/sequence, e.g. "&lt;" if you like HTML/XML or "‹" (U+2039) if you want something that looks like < (but you'll need to remember to use ‹ instead of < when editing your text) or "«" (U+00AB) if you just want to keep using a single character.

But i have another question:

I configured my table file with End statements that are suppored to include new lines:

/FFFF00=[END-FF]\n\n\n\n\n\n\n
/FFFE00=[END-FE]\n\n\n\n\n\n\n

But its not writing those. Are new lines not possible in End statements?
Hmm, it looks like I accidentally broke multiple newlines while adding the #COMMENTS: Both option and never noticed since I never use that many newlines at once :P. I'll get that fixed for v0.0.7.

I also saw your other thread about passing in multiple insert files at once instead of calling abcde multiple times. That's definitely useful in some situations (especially when you can specify the file names via shell globs/pattern matching/etc.) and it shouldn't be hard to add support for, so I'll look into adding it for v0.0.7.

And, is it possible to remove the writing of the lines "// current address: $27B2..."?
At the moment, no, but you can add your own post-processing step to strip them out, e.g. via search-and-replace in a text editor or an extra line in your batch file (something like "perl -i -pe 's/\/\/ current address:.*\R//' GL6_dumped_script_*" ought to work, but test that yourself before using it). As an abcde user, I like knowing what the end address of each string is since that information helps to identify gaps in the script (this is one way of discovering dummied content, for instance) or game-specific control codes that I haven't fully figured out yet, but I suppose it ought to be optional. One more thing for v0.0.7!

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: [Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde
« Reply #6 on: April 05, 2020, 11:23:22 pm »
Hello abw!

Thank you very much for your reply.

Quote
Hmm, it looks like I accidentally broke multiple newlines while adding the #COMMENTS: Both option and never noticed since I never use that many newlines at once :P. I'll get that fixed for v0.0.7.

I also saw your other thread about passing in multiple insert files at once instead of calling abcde multiple times. That's definitely useful in some situations (especially when you can specify the file names via shell globs/pattern matching/etc.) and it shouldn't be hard to add support for, so I'll look into adding it for v0.0.7.

Quote from: Risae on Today at 07:59:13 pm

    And, is it possible to remove the writing of the lines "// current address: $27B2..."?

At the moment, no, but you can add your own post-processing step to strip them out, e.g. via search-and-replace in a text editor or an extra line in your batch file (something like "perl -i -pe 's/\/\/ current address:.*\R//' GL6_dumped_script_*" ought to work, but test that yourself before using it). As an abcde user, I like knowing what the end address of each string is since that information helps to identify gaps in the script (this is one way of discovering dummied content, for instance) or game-specific control codes that I haven't fully figured out yet, but I suppose it ought to be optional. One more thing for v0.0.7!

That would be super super awesome if you could implement those features!!
The amount of script files in my translation will reach about 150 (at the very end), and having such a feature would be really neat.
Do you think it would be possible to implement a feature where you could let the program give you a warning if you insert too much text, causing the filesize to increase?
Or something like a function that prevents the program from inserting if the original file size changes.

Also, i have a quick question:
Using Atlas and Cartographer i had to use 2 different table files, because of the ending statements.
Is it possible to use one table file for both with abcde? Or are there still different ending statements needed for Cartographer vs Atlas?

Thank you very much for programming this neat tool!

abw

  • Sr. Member
  • ****
  • Posts: 426
    • View Profile
Re: [Technical][abcde] How to convert Shift JIS Table to UTF-8 for abcde
« Reply #7 on: April 06, 2020, 08:57:29 am »
Do you think it would be possible to implement a feature where you could let the program give you a warning if you insert too much text, causing the filesize to increase?
Or something like a function that prevents the program from inserting if the original file size changes.
That should already be working - setting an upper bound on the insert address with the second #JMP command parameter will will display an error message and prevent insertion beyond the given address if that section of your insert script takes up too much space, so e.g. with #JMP($2790, $5624) it won't insert script beyond $5624, which when you're working with a monolithic ROM like on the NES/SNES allows for even finer control than just checking file size; on platforms with a filesystem, if you're inserting into a specific file, you could set the upper insert bound to the size of the file.

Also, i have a quick question:
Using Atlas and Cartographer i had to use 2 different table files, because of the ending statements.
Is it possible to use one table file for both with abcde? Or are there still different ending statements needed for Cartographer vs Atlas?
Assuming you aren't making any changes to the game's text encoding, you should be able to use the exact same table file for both extracting and inserting text with abcde.

Thank you very much for programming this neat tool!
I'm just glad other people are finding it useful too :).

Risae

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
I believe i successfully made the switch to abcde!
Some cool things which i can now do:

1. Bitbucket for god knows why only encodes its files in UTF-8. You have no option to change it, anywhere.
Now i can dump all of the repository scripts into one folder and bulk insert them into the game files! So much time saved copy and pasting everything!
Still have to manually create a batch file that has the file names in it, and i have to change those whenever i changed them on the repository, but this is a lot less work than before.

2. I don't have to copy and paste the original script under the commented one anymore because abcde supports the function of "both", wish i had made the switch to abcde earlier hahaha

3. I don't have to maintain 2 shift jis tables anymore, one table file can work on both Atlas and Cartographer scripts!

Only downside i see is that i had to download perl in order to use abcde, it was easier to have a ready-to-work exe that you could use without any prerequisites. But i guess that is a first world problem, it works like a charm.

Quote
That should already be working - setting an upper bound on the insert address with the second #JMP command parameter will will display an error message and prevent insertion beyond the given address if that section of your insert script takes up too much space, so e.g. with #JMP($2790, $5624) it won't insert script beyond $5624, which when you're working with a monolithic ROM like on the NES/SNES allows for even finer control than just checking file size; on platforms with a filesystem, if you're inserting into a specific file, you could set the upper insert bound to the size of the file.

I see, i will test out some stuff after porting my scripts to abcde. Thanks for letting me know about it!

April 06, 2020, 12:07:54 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
I would also recommend updating the ReadMe, because this reads it like you can't do the things that are currently broken with the Both command:

Quote
# Since newlines are reserved for delimiting table entries from each other, you can't make a multi-line table entry, but you can use "\n" anywhere in the right-hand side of a table entry to represent a newline:
FE=new\nline
# 0xFE would then be translated to text as:
# new
# line
# Newlines are ignored when translating text into binary, so in the absence of any other usable table entries, each of these texts would be translated to 0xFE:
# 1)
# new
# line
# 2)
# new
#
#
# line
# 3)
# newline
# 4)
# ne
# wli
# ne
# 5) etc., etc.

In this example you didn't include an example where different pointers are further apart ( 1) 2)...)
« Last Edit: April 06, 2020, 12:07:54 pm by Risae »