News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Double Byte Text inserting / 16bit S-JIS  (Read 4409 times)

Jazz

  • Full Member
  • ***
  • Posts: 180
    • View Profile
    • Jetran
Double Byte Text inserting / 16bit S-JIS
« on: August 30, 2018, 03:39:02 am »
Hi Everyone, apologies for the 'whatever is easiest' type of question

I've always avoided games that use Double byte text (sticking to gameboy color type work) and have been playing with a game for GBA that uses 16bit text or DBCS.

What I wanted to know was what process you normally take to insert the text into the game. I've always used a crazy slow method of typing everything into Windhex/Thingy more for quality assurance and lack of trying new methods however with DBCS i'm forced to type the hex values in.

Is there a suggested insert tool or do you normally write your own.
« Last Edit: August 30, 2018, 03:45:25 am by Jazz »
Jetran Website
View for Gameboy english translation projects

Discord: Jazz#9202

FAST6191

  • Hero Member
  • *****
  • Posts: 2501
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #1 on: August 30, 2018, 04:49:21 am »
Depends what said double bytes are. Also for some things you can drop it down to 8 bits, also gain a lot more space and memory that way.

If said double bytes are something I can force with u16 unicode then OK. Same if you can do a find and replace later -- in plain ASCII ( http://www.asciitable.com/ ) you have things like numbers in hex are 3 followed by the number you want, another is lower case is 20 hex higher than upper (41=A and 61=a, 42=B and 62=b....) so if your chosen encoding is that.

For shiftJIS (note shiftJIS in the real world supports various types of unicode for the lower values, most games however only have the 8100 + ranges seen on http://www.rikai.com/library/kanjitables/kanji_codes.sjis.shtml , fortunately said encoding includes a copy of the Roman characters which can get you out of a pinch) then there are a few choices. Japanese text editors often have a "J-Ascii" type mode. JWPCE (I see also there is a fork in JWPxp now https://mijet.eludevisibility.org/JWPxp/JWPxp.html ) having such a thing. There are three mode buttons in the right hand side, one is plain ASCII, another is Japanese and the final will force it into the 8??? range that houses Roman characters in Japanese fonts. Save as shiftJIS and slice it out with your hex editor.
In actual Japanese (and Chinese and Korean) writing circles there is another in the form of NJstar https://www.njstar.com/cms/ . I am less familiar with this but it should also be able to do the job.

You could probably write some scripts as well (this sort of manipulation is what most of the likes of python, perl and such live for) but nobody has really shared those. Don't have any examples here, however might as well have a little tool I find handy in these situations
http://www.romhacking.net/utilities/504/

Jazz

  • Full Member
  • ***
  • Posts: 180
    • View Profile
    • Jetran
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #2 on: August 30, 2018, 06:27:24 am »
Really great feedback, thank you FAST.

When I say double bytes I just mean 2 hex values to produce a single character (which I'm sure you've understood) but for clarification i.e. 8273=T

I'm looking for different approaches on how translators best tackle this. I think it just might be time to stop being to manual and look at Extractor and Inserter script tools - abcde & atlas just turn me off with the technical hah.
Jetran Website
View for Gameboy english translation projects

Discord: Jazz#9202

FAST6191

  • Hero Member
  • *****
  • Posts: 2501
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #3 on: August 30, 2018, 10:21:58 am »
Yeah I probably pressed go a bit too soon there, hopefully the example still worked. For the sake of making it clearer still though. Various unicode editors will force a 00 in there as well for the simple ASCII stuff (0020 being space in forced 16 bit encodings sort of thing). In that case to turn it to shiftJIS (or at least the Roman character part of that) you could probably do a fairly simple addition to change it, or a if 41h<=[character]<=5Ah then add so much and do the same for lower case. You could even do all that in a spreadsheet -- give each 8bit (or 16 if you cram in the 00) section its own line, everything else is basic spreadsheet fiddling (hope you remember the fill command from school). Might need a tiny bit of fine tuning for punctuation but those can be a 10 line long find and replace really.
If playing with translations I tend not to work on the epic novel length RPGs or anything. If I only have 300 lines, and even if only in my head I can break that up further into menus, chapter start-end texts and whatever to make it and still have pointers to fiddle with after that is it not so bad using something like JWPCE. If the game has it in the font and encoding (most do as Japanese game devs do tend to chuck in an English word or two and thus they need it) then that makes for a nice workflow, far nicer than looking up things individually with what most hex editors pass off as 16 bit entry (I assume yours is like most and consider entering a 16 bit character a rarity, and thus gate it behind a bunch of menus. The combined sum of which can be said to suck a lot if you want to enter a 200 character sentence you could type in a few seconds normally).
You might end up transcribing a line or something but at that point it is just typing.

Proper tools are the best way in the end but you can put off having to build them for a little while. You can put it off longer if working with a translator and set things up to be easily inserted from the beginning.

theflyingzamboni

  • Jr. Member
  • **
  • Posts: 89
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #4 on: August 30, 2018, 06:09:57 pm »
If you can't find any existing solutions to the particular game you're working on, there's always the path of making your own text dumper/inserter. I'd make sure nothing else will serve your purpose first, but it's always an option, and the approach I had to take.

I'm working on text in The Legend of Dragoon, which uses 2-bytes per character in both the English and Japanese releases. Dealing with the characters themselves is easy. My dumper/inserter builds a codec at runtime from a customizable text document with the ordered list of characters (it uses its own non-ascii font table). It reads bytes two at a time and uses the codec to decode the 2-byte sequence into 16-bit Unicode, which it writes to a text file. When I want to insert text, it does the same thing in reverse, using the table to turn each Unicode character into a 2-byte pair and write it back to the binary file.

So if you end up taking this route, working with 2-byte characters isn't functionally any different than working with 1-byte ones.
ROM wasn't hacked in a day.

filler

  • RHDN Patreon Supporter!
  • Hero Member
  • *****
  • Posts: 876
  • "WINNERS DON'T SELL REPROS"
    • View Profile
    • Filler's Translation Projects
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #5 on: August 30, 2018, 10:01:32 pm »
If you can't find any existing solutions to the particular game you're working on, there's always the path of making your own text dumper/inserter. I'd make sure nothing else will serve your purpose first, but it's always an option, and the approach I had to take.

I'm not about to try to write an inserter since I know nothing about redirecting pointers, but I've just started writing my own dumpers in PHP to get scripts I can translate from games that have trickier text routines/formats. If you're already familiar with a syntax and coding environment you might do just as well to take a stab at writing something for your particular game and use case.

Jazz

  • Full Member
  • ***
  • Posts: 180
    • View Profile
    • Jetran
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #6 on: August 31, 2018, 11:55:58 pm »
Good point, filler - may look to build a script in PHP.

Jetran Website
View for Gameboy english translation projects

Discord: Jazz#9202

Disch

  • Hero Member
  • *****
  • Posts: 2710
  • NES Junkie
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #7 on: September 02, 2018, 11:08:32 am »
Not to be a wiseass or snob... but.... PHP?

I mean... that would work.  But why PHP over Python?  Or Perl?  Or Ruby?


Jazz

  • Full Member
  • ***
  • Posts: 180
    • View Profile
    • Jetran
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #8 on: September 02, 2018, 08:04:00 pm »
Using what you know best, I guess!?
Jetran Website
View for Gameboy english translation projects

Discord: Jazz#9202

theflyingzamboni

  • Jr. Member
  • **
  • Posts: 89
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #9 on: September 03, 2018, 09:30:57 am »
I'm not about to try to write an inserter since I know nothing about redirecting pointers, but I've just started writing my own dumpers in PHP to get scripts I can translate from games that have trickier text routines/formats. If you're already familiar with a syntax and coding environment you might do just as well to take a stab at writing something for your particular game and use case.

I didn't know anything about redirecting pointers either, until I wrote an inserter.  :laugh:

How do you get your translated texts back into the ROMs?
ROM wasn't hacked in a day.

filler

  • RHDN Patreon Supporter!
  • Hero Member
  • *****
  • Posts: 876
  • "WINNERS DON'T SELL REPROS"
    • View Profile
    • Filler's Translation Projects
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #10 on: September 03, 2018, 10:56:56 am »
I didn't know anything about redirecting pointers either, until I wrote an inserter.  :laugh:

How do you get your translated texts back into the ROMs?

I ask other people nicely to do it for me. :)

Normmatt

  • Full Member
  • ***
  • Posts: 139
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #11 on: September 06, 2018, 09:39:39 pm »
I personally (ab)use ARMIPS for inserting text these days.

Here's a few example projects where I use it:
https://github.com/Normmatt/F-Zero-Climax-GBA-Translation
https://github.com/Normmatt/Medarot-Navi-GBA-Translation

Some older projects use a combination of Cartographer to dump the script and Atlas to insert them:
https://github.com/Normmatt/Bomberman-Jetters-GBA-Translation
https://github.com/Normmatt/Gachasute--Dino-Device-2-Dragon-Translation


abw

  • Sr. Member
  • ****
  • Posts: 253
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #12 on: September 07, 2018, 05:12:18 pm »
When I say double bytes I just mean 2 hex values to produce a single character (which I'm sure you've understood) but for clarification i.e. 8273=T

I'm looking for different approaches on how translators best tackle this. I think it just might be time to stop being to manual and look at Extractor and Inserter script tools - abcde & atlas just turn me off with the technical hah.
FWIW, Atlas can definitely handle inserting multi-byte text. There is a bit of a learning curve, but the documentation is fairly helpful, and even if it takes you a couple of hours to figure out how to get it to do what you want, it will save you days of typing things in by hand.

abcde should also work, but its learning curve sort of starts where Atlas leaves off, and it sounds like you don't need to head that far down the rabbit hole quite yet.

Psyklax

  • Hero Member
  • *****
  • Posts: 1003
    • View Profile
    • Psyklax Translations
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #13 on: September 09, 2018, 04:17:04 pm »
I ask other people nicely to do it for me. :)

8)

Anyway, I really don't see why 8-bit and 16-bit text should be any different. Can't you just make a table file with two bytes per character? As FAST suggested, you might be able to convert 8 to 16 if you can write a little routine. A very helpful guy showed me a simple routine to do it on the PC-98.

As for inserting text, my preferred method is a combination of Pointer Tables (the program, dull name I know), making my own IPS files, a regular hex editor, and maybe a bit of WindHex 32 EX. It's an odd combo but works for me.

Jazz

  • Full Member
  • ***
  • Posts: 180
    • View Profile
    • Jetran
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #14 on: September 11, 2018, 02:23:46 am »
Thanks for all the suggestions and feedback everyone, it's greatly appreciated.

I know it may seem like a basic insertion adjustment but the way I've done things for 15+ years has been so manual I've never tried to look for alternatives until now working with dual byte text. It looks like my most obvious answer is using atlas which as you suggested abw will only take a few hours to probably get the hang of it to eventually save me days if not weeks on future projects.

@Normatt - great to see you still around buddy - been a long time since PN Network. Again, thanks for the references to those projects. Funny, I did a mostly completed Dino Devices 2 translation in 2005 with a guy named daMoose years ago however lost it on an old computer/may still have it but too lazy to find it.
Did you ever complete it?

September 11, 2018, 02:29:21 am - (Auto Merged - Double Posts are not allowed before 7 days.)
8)

Can't you just make a table file with two bytes per character? As FAST suggested, you might be able to convert 8 to 16 if you can write a little routine

Creating the table, dumping the text and pointer tables is no issue - it's how I normally go in with Windhex32 and manually type in the english which isn't something that is easily done when it's two bytes per character - or I just haven't found a successful way to do it.
« Last Edit: September 11, 2018, 02:29:21 am by Jazz »
Jetran Website
View for Gameboy english translation projects

Discord: Jazz#9202

Normmatt

  • Full Member
  • ***
  • Posts: 139
    • View Profile
Re: Double Byte Text inserting / 16bit S-JIS
« Reply #15 on: September 12, 2018, 11:02:42 pm »
@Normatt - great to see you still around buddy - been a long time since PN Network. Again, thanks for the references to those projects. Funny, I did a mostly completed Dino Devices 2 translation in 2005 with a guy named daMoose years ago however lost it on an old computer/may still have it but too lazy to find it.
Did you ever complete it?
Sadly, nope never finished it, haven't found anyone wanting to translate it  :'(