[Vorthod Wiler]

As recently stated in the Introduction sticky, I would love to try my hand at translating Shin Koihime Musou into English. I have some experience in both basic coding and Japanese language, but I am unable to find the information that can get me started. Some helpful people on the IRC channel pointed me towards the "Olly Debugger" and "MadEdit" tools claiming that they would be of use, but I find myself being unable to figure out how to use them for what I am attempting to accomplish. For now i don't need to do anything fancy like changing the graphics to make English buttons; I would just like to start by finding out how to locate the Japanese text and change it to an English script. Therefore, if anyone could give me (or post a link to) a useful tutorial, i would be extremely grateful.

[FAST6191]

Though all code, regardless of intended platform for running on, shares similar concepts the PC can be subtly different to consoles and have a do a lot of things that would be insane to do on consoles. Most notably the PC can,and frequently does, use heavy obfuscation, tweaks upon common formats, network and operating system interactions, runtimes and has no solid hardware spec to really work towards. This can leave a lot of people more used to console game hacking outside their comfort zone, I am certainly going to claim no great skill in this area.

Olly debugger and madedit are good tools. The former is a rather basic debugging program, but one that is easy enough to get to grips with, is free and works on programs that might not have their debugging functionality left in. Madedit is text editor favoured by a lot of coders as it also has serious hex editing capabilities too; with the possible exception of the notepad++ family most hex editors do not have great text editing abilities and more hex editors suck for text editing beyond changing the odd word here and there.

Just to make it even more fun the Japanese seem to do coding slightly differently to most people outside Japan. There seems to be various reasons for this and not all of them are without good reason, games are very much troubled by this though. My three rules of thumb for games tend to be images are often PCX format (a format most outside game dev/hacking will not have seen since the early 1990's) or even raw bitmap, audio can be raw wave (I have seen otherwise very small games have sizes multiplied seriously by this), compression tends not to be something other than zip/tgz/rar/7z (LZH family being the starting point), bonus for all this though is most of those formats can be worked with rather than descending into the madness that is a lot of directX type things in games made outside Japan. One more on pictures though -- look up the susie32 plugin system, most outside Japan I tend to find have not heard of it (no great reason, it works but is nothing particularly special) but it has a fair following inside Japan among game hackers and you may be able to find something for the game (or a similar one the same dev made) that helps you out.

Back on topic with Olly debugger then. We get to talk about assembly now and bonus it is X86 assembly. Assembly is the thing feared by most newcomers and is also what allows the better hackers to do what they can do where those just coming in would not even know where to start.
Links
http://www.plantation-productions.com/Webster/www.artofasm.com/index.html
http://stuff.pypt.lt/ggt80x86a/asm1.htm

Assembly is human readable version of what the processor runs in the computer and the X86 (aka what Intel and AMD make processors to support) are some of the most complex going, especially compared to the sorts of things we deal with on sites like this and few would call assembly for the processors used on those devices all that easy either. Nobody really writes in it any more, Microsoft have even mostly stopped supporting the ability to write in it in their dev tools and discourage its use outside it (good security reasons for this).
The trouble is where I can learn the hardware of a SNES and watch things work that way the PC does not typically interact with hardware like that. Here you have the operating system handling a lot of it and beyond that many games are made using scripting languages which typically do not get turned into assembly until moments before the code is run (python, lua, gamemaker, visual basic, java (ish), C# -- all things I have seen many PC games, especially Japanese PC exclusives, written in and all with serious scripting or otherwise nice to reverse engineer components). To that end general coding skills and operating system knowledge (from the coder perspective) will help you here, far more than it does for a lot of console hacking. My usual links for programming these days are http://programming-motherfucker.com/become.html and http://www.youtube.com/view_play_list?p=6B940F08B9773B9F

I mentioned it in passing earlier but you should also know many games obfuscate (scramble to make unreadable/uneditable to simple methods) their assets and code. On the PC it can be both modern and old things that do it, they vary in how they go about it but there is no real "before this date" type thing I can say here.

It is not all doom and gloom though and some of those things I was just griping about actually make life easier in certain ways.

[esperknight]

I can help with directing you   I was pointed to this thread by the same person who probably told you about those tools since those are what I use

I use one other tools as well, TSearch.  (My copy I use. Should be safe but never know...)

What I do usually is play up to a point where the text is shown and then load up TSearch and attach it to the game (Open Process I believe), click on Hex Editor, and then click on the search icon.  I then paste in whatever japanese text I see (I type it manually using JWPce or you could use Windows IME tool).  TSearch accepts SJIS input for searching so makes it easy   I do this to verify what I'm seeing is SJIS since they could easily use graphics for the text (Shikigami no Shiro *shudder*) or anything else they wish.  Now assuming that it's SJIS...  (SJIS table in case you want to use something that can take in tables : SJIS)

I do two things.  I'll do a search using MadEdit's find in files tool and search through all the files of the game (make sure to set text encoding to SJIS if using Japanese text or it may not find it if it's unencoded).  If I get lucy and it's not encrypted or compressed then I open that particular file and study it's format and write whatever tools are needed.

But if I can't find the text so easily...

I then fire up OllyDbg and then open the game with it.  I then go ahead and run it.  From there I'll try to pinpoint exactly when the text is loaded.  Usually your text either gets loaded when a stage loads or after the title screen.  Depends if they use separate chunks for text or one huge file.  Memory is cheap now a days so devs get lazy (I can vouch for this one

So once I pinpoint where it's roughly at I restart the game using OllyDbg, get to that point in the game or right before it (as close as you can), and then put breakpoints on all CreateFiles.  Right click on the code window and tell it to Search for All References or Imports... I forget which... it'll bring up a window with all the windows functions it calls into.  Search for CreateFile and breakpoint them all.  Have the game go and then...

It'll break on each file that's being opened.  From there I'll let each one run through and use TSearch to see if the text appears (forgot to mention you want to attach it at this point )  Once I see it I make a note of what file was used.

Then I restart OllyDbg again, get up to that point and then either if I'm feeling lucky I'll do it early or I'll wait till it's about to open the file and then put make breakpoints on all ReadFiles.  ReadFile is what's used to read the file of course so you'll see it get accessed A LOT.  So recommend it to only set it when needed if you can

So basically I do the same thing with readfile almost. Here you make note of where OllyDbg says it's going to be stored at.  Look for which register it says is the buffer pointer and then right click the memory view and tell it to use that buffer location or you can type the register it's using which is the same thing

Then hit F8 to step over it and go there, search for that chunk using the hex you see using MadEdit (set it to find hex on this one) and see what file it pulls up.  Then hit F9 and see if TSearch can find it.  If not, repeat ad naseum.

And voila!  You found the file it loads from and almost where it decrypts/uncompresses it from

Mind you this assumes the game isn't too crazy in how it does things.  As it doesn't need to use the windows functions directly but can use other ways of getting to them such as setting a pointer to that point that'll call that function and call that (something like that.. I forget how it looks but another game I'm working on does that...).  Or they can load up everything at the beginning and it's a pain digging through it all or they break when you attach a debugger immediately or.......... So hopefully this one isn't too bad

Also with TSearch attached you run the risk of the game throwing an exception and crashing so don't keep TSearch attached all the time.

Also, one thing you can do too if it's compressed and such, you can actually dump it all from memory.  Just jump to it in memory using OllyDbg and then tell it to dump that block out.  Then use MadEdit to pare it down and voila!  You have a nice unencrypted version of your file   This assume it doesn't decompress a line at a time or some such craziness (I've seen it before...).

If I get a chance (and I am very slow). I can help take a look at this one to provide you more specific guidance.

One day too I'll write a guide about this...

FAST6191 makes a great point too. PC can be a crazy beast due to how wide open it is.  For compression formats I agree with the most widely used one being LZSS (and for consoles as well).  It decompresses fast and also is free and open sourced as well (see here for the code).

Hope this all helps

Edit: Curious, are we talking the fighter or the visual novel?  Either works for me as I enjoy visual novels.  Fighters I suck at playing

[Vorthod Wiler]

Thanks you two, i should be able to at least experiment a bit with this information. As for your last question, Esper, I am looking at the visual novel. Though, since I love this series so far and the fighting game does look a bit interesting I might purchase a copy of that down the road and see what i think of it
March 18, 2014, 12:44:27 am - (Auto Merged - Double Posts are not allowed before 7 days.)Ok i have hit some roadblocks, but hopefully i can give enough information to make a solution clear. I'm not sure how severe the next couple of paragraphs are but i should probably include them since they could be the reason the issue in the last paragraph is happening. I've been playing with a text hooker and translator up until now, so its definitely got SJIS text over graphical text. nearly every file the game has is essentially unreadable in MadEdit no matter the encoding settings, so it's probably compressed or something. With that in mind, I have gone to the Olly Debugger step.

first off this game doesn't seem to like things attaching to its process. the text hooker i use will crash the game if it is attached when the game loads a save; it seems that behavior comes around to Olly as well since it wont start the game through Olly at all. It seems to work if i just attach Olly to it when it gets to the main menu though, so hopefully that's not a real issue.

Upon attaching, i get an error stating "Bad or unknown format of 32-bit executable file 'C:\Program Files (x86)\BaseSon\shin_koihime\shin_koihime.exe'" This is...worrying to say the least, but I wouldn't be surprised if that was a generic error that just meant something like "we cant track window and process names very well for this program" or something related to simple user-friendliness like that, so i continued on.

Anyway, my big problem is that i cannot find any instance of the phrase "CreateFile" or even "ReadFile." I have stopped the program at a multitude of points (Main menu, new game, first actual line of new game, text-free save point thing of loaded game, first line of dialogue from the loaded game), right clicked the code window and used the "Search for > all referenced text strings" (i think that's the command you were trying to remember) and searched the resulting window without finding any place where files are opened/created. I noticed the game has a disturbingly large number of threads (20 or so, one called main and the rest called by different 8-digit hex numbers) and i searched them for the command as well without any luck; though i only searched those extra threads in the "first line of dialogue" stage. If i could just get passed these first few steps i could at least start getting a feel for the flow of these programs. 

[esperknight]

You were close on what to search for   Strings can help too but what you want is Search For -> All Intermodular Calls.  This'll list all the calls being made out to external DLLs. Just make sure the EXE you are on is the proper one as a lot of times if you attach then it'll be NTLL.DLL (or whatever it is... I can't recall of the top of my head )  If that happens just click View -> Exes.  Threads I haven't really used but that could be useful as well.

The bigger problem you have though is the game uses DRM or at least anti piracy measures.  Doing a bit of searching, looks like it uses something called AlphaROM.  I've never heard of it till now so not sure how bad it is compared to SecuROM.  I'm surprised as most Japanese games just use anti debugger measures (that you've seen such as not allowing OllyDbg to start it which is part of AlphaROM).  Dealing with DRM such as this is a bit out of my league even but I wouldn't give up hope.  Not sure if you're using it... but if you find a NoCD patch sometimes the work is already done for you in decrypting the exe and such.

Otherwise one option is to find the entry point and somehow dump the executable (if it just loads one) but I think if it's like SecuROM it's probably much worse.  I'm not sure if I can post such things here since it's related to cracking and such...... but if you search for unpacking plus the type of DRM you may find something.

So curious then... interested in other games?   I'm slowly getting there on being able to look into this myself so I may be able to help you more.  Be a bit though.

One other option that could help (possibly) is using IDA Pro.  I prefer OllyDbg just because I'm use to it but IDA Pro is far more powerful.  They have a free version located here you can play with : IDA Pro Freeware.  The professional version costs quite a bit and not sure if it's for sale to individuals (it may be now...).  But definitely something to try.  If it's packed as I'm pretty sure it is though, IDA Pro most likely won't be able to help either.

[Vorthod Wiler]

It does appear to be attaching to ntdll.dll, but i cant get it to look at the actual exe since double clicking the exe name just shifts focus back to the CPU window without actually changing that window's target, so i will try and download that IDA debugger freeware you mentioned and see if it looks promising. If that doesn't work, then its off to google for the unpacking tips.

as for whether or not i am interested in other games. This is the only game I've played that still needs a translator, i think. the only other untranslated game I've played came out extremely recently and probably has a company working on it already (Lightning Warrior Raidy 3 if you are curious).  However, I won't go far as to say that i couldn't become interested in others, if i find an interesting game that needs translation, i may end up picking it up.

[esperknight]

Very interesting it's doing that... Wonder if it's due to it now knowing how to handle it?

I see what you mean about Raidy.  Appears to be picked up by JAST but no telling how long that'll take... At least S;G should be releasing this month (although there is that "leaked" patch of course).

Depending on your tastes I have a few that could be of interest.  I enjoy playing the older titles though so not sure.  Comic Party would be one and the other would be Kizuato, both by leaf.  Very much older titles with Kizuato I believe being one of there first ones or at least one of there first successful ones.  I have others on PC98/88, X68000, and PSX.  Even Saturn and DC   If you're interested my repo is here : transprojects  If you're not interested, no worries.

I still want to look into this one as well.  I'm always up for a challenge   I'm very surprised though this game uses any DRM at all.  The worst I've seen was a region check to make sure you were playing on an actual Japanese PC.  The others were more just a pain in how they were encrypted/compressed (looking at you Princess Lover and ShiniKiss).  That and some antidebugger methods akin to what you've seen but much more tame.  The worst I've seen is Sakura Wars 1 using SecuROM.  Very interesting.

One thing i forgot to mention that may be of help (I haven't checked yet) is whether or not Asmodean has gotten to it.  He writes utils for many, many games.  He makes my repo look like a tiny little thing in comparison : Asmodean's Tools.  I've used his tools as references for my own many a time (why recode the wheel if I don't have to?)  He doesn't right inserters though, only unpackers.  I can help with an inserter though.  He also doesn't usually right script extractors either so I can help there as well.

Hope this may help.  Soon enough I should be able to look at the game as well...

Edit: Yep, asmodean has written a tool for this one : extafarc  Not sure what it does just yet, whether or not the scripts will get decrypted/extracted or what.

Edit:  Seems they use XOR encryption and regular LZSS (my guess since Asmodean uses his own LZSS routine).  Not so bad at least

[Vorthod Wiler]

So with this tool from Asmodean, am i supposed to be running the exe files or the cpp? the exes just close immediately when i open them and my c++ IDE is acting up a bit since i havent used it in a while, so i would like to see if thats what Im supposed to be using before i go about trying to reinstall it. If im supposed to be using one of the exes, do i need to set a command line argument in their properties or something? I feel bad asking on this one, but i didnt find any instructions on the host site or in the readme.

also, Kizuato...i feel like I've heard that title somewhere before, but i can't remember where. Oh well, I enjoyed Utawarerumono and Tears to Tiara, so perhaps looking at some of Leaf's older works would be fun (its synopsis on vndb has potential as well). I will probably end up checking that out at some point.

[esperknight]

The exe file.  You can run it via command prompt (just run it via the accessories folder) or you should be able to just drag and drop the *.arc files on top of it.

I checked a bit on the game last night and yep, OllyDbg won't open it up due to not recognizing the CD.  Interesting enough, I tried using a packer identifier on it and it couldn't find anything.  Very odd.... I'll keep playing with it though.  Plus the packer identifier I used may not have that specific packer in it's database so never know.

Leaf makes some good games overall so I'd say it's worth it

Edit: Just tried the unpacker, you want v2.

Edit: And bin.arc has what looks like the scripts.  After it exctracts look at the Nov_SN* files with MadEdit.  Set it to SJIS encoding and you should see the text.

[Vorthod Wiler]

thanks a bunch. It looks like the Nov_SN* files contain scripts for each characters side stories while the SN#### ones contain the scripts for the main three storylines (the bonus storyline might be stored elsewhere since the file numbers only go to the 300s. So if i start editing these, how do i put them back into a patch so that i can see this in game?

[esperknight]

Putting them back together is the fun part

First we need to be able to extract the script properly.  I don't have access to the files just now but from what I glanced at this morning it looks like each scenario is scripted. The data before the scripts should point into them via pointers.  Depending on how the pointers are setup they oculd be easy to find.  Just search for the position of the string in hex but backwards.  So if it starts at 0xACC8, search for C8AC.  X86 is little endian so it stores numbers backwards but reads them in correctly.  If you get lucky it'll show up and you should be able to replicate it for others.

Another test if that doesn't work is offset the position with the beginning of the text block.  So if it starts at 0x2000 minus that off it's position and search for that.

There's other possibilities too but those are the easiest to test for.

So once extracted I prefer to use Atlas.  Atlas is a very nice insertion program that is quite customizable.  I thought I had code uploaded for examples on how I extract it for insertion with Atlas but apparently not... I'll upload some in a bit (I'm lazy so haven't been doing it I guess )

From there I setup a toolkit where I have a batch file run that inserts everything and then another to build the pac file.

The pac file is more fun.  That may be a bit harder as it seems v2 may uses a checksum (such as asmodean noted).  It may be we'd have to find out how it's calculated.  Other than that you'd have to setup the LZSS compressor which may just be a stock one and that's easy enough to integrate if so. I've also coded it many a time...  For a good example but very old one I did see this one for Comic Party : cp_pack Take a look at CompressAndBuild. This was my second game I hacked   Giten Megami Tensei being the first.  If you prefer C# I modified the orig LZSS algorithm to be in C# in one of the threads here...  I'll search for it later and link it here if I remember

Overall though for the pac file it's just repacking it via how Asmodean extracted it.  The hardest part really is just the compression and encryption pieces if existing as some can be a real pain.  This one to me isn't too bad but then again I've been doing it for quite a while

Hope this helps!

Edit: Here's an example script extractor I did for Corpse Party 2U for PSP.  To see what it extracts see here : Corpse Party 2U Scripts and Code

Got a chance to look more at the script files.  They do indeed use linear pointers so no offsetting. It appears that 0x540 always has the first pointer to the block. Whether or not it uses it I'm not sure but that's useful indeed.  I'm thinking that all the script stuff happens after that point as well as the beginning of the script files look the same (at least for the ones I sampled).  It also appears that the commands are 4 bytes each.

So what I would do for an extractor is jump to 0x540, grab the pointer there, set that as my text block beginning.  Then I'd loop through from 0x540 looking for locations that point beyond or equal to the beginning of that location.  Then I'd loop through and grab all the text and set it up for use with atlas

Edit:  This is how I'd extract it   SKM Text Extractor Code and Scripts.  It's not 100% complete as I'd spit out a bat with it too for insertion and also I'd not print out all the ogg and png and other files as no need to insert those twice.  I tend to use C# for this type of work when there's multiple files as C# makes it nice and easy to deal with bulk file extracts.  C++ I tend to use for packers and such as that can be easier with it since C++ I think works better with binaries.  But doesn't matter really as long as it gets the job done

Edit:  I'm confused.. searching more on the game... did MangaGamer make a release of it?  I'm seeing stuff on there blog about releasing an english version but I don't see on in the store?  Can't post links here for obvious reasons

Edit: And found this so it does seem that it's getting translated : SKM Project

[Vorthod Wiler]

Alright, that sounds like...fun. guess it's time to dust off my old c++ projects so i can remember how to use the language. Its been a lot longer than i realized now that i really think about the last time i used it. You sample code is certainly going to be a great help.

in regards to your most recent edit...ehem: HOW ON EARTH DO I ALWAYS MANAGE TO START THINGS LIKE THIS LESS THAN A MONTH AFTER A MAJOR DEVELOPMENT WAS ANNOUNCED! I swear the same thing happened when i finally gave up on the release of Raidy III (guess what i found a few weeks after that)...oh god these games are getting a disturbing number of things in common. But in all seriousness, this is creeping me out a little. Maybe that let's play/translation runthrough thing on youtube sparked their interest up in the project or something once they saw how it played out. That would at least give a couple explanations to their extraordinarily similar timing. Well, i probably wont be able to match up with their localization speed, but this will probably still be a good workout for me, so to speak.

[esperknight]

C# as well can't hurt since it's easy to pick up too   Python is another although I'm not as familiar coding in it.

And you should then start translating Kizuato or Comic Party   I admit though, I laughed pretty good reading you're reply.  It's crazy indeed No reason though not to offer your services if you're interested.  Multiple translators are the norm on PC VN projects since they're so large (and this game appears to be huge 0_0 )  I pointed them to this thread as well in case it may help them.

Also great news!  I forgot to mention one other trick you can try besides repacking it.  Just create a folder of the packaged files name (such as bin), stick the files in there and see if it loads   Works a charm for this one   Not always is that the case though but sometimes you get lucky   So yeah.... no need to compress or repackage.  Compress may still work but packaging as I said would be a pain due to the checksumming they most likely do.  Still can't hurt to try though for consistencies sake.

I'm assuming bin has the main script that starts up and that somehow it's not finding my renamed bin file as well.... so if you want to try verifying it, make a mod to the file with the beginning script and see what happens if you could   I'll try later as sadly I don't have much time now...

[Vorthod Wiler]

that is extremely weird. i cannot believe it actually loads when you do that. (at first i tried renaming the file bin.arc and that even more surprisingly failed). sadly, though, it does not seem to be loading in my modified dialogue. though that makes me wonder where its getting the text that it is actually displaying since i renamed the old bin file. I will mention that i tried this with both the first scene of the shoku (green) route and Chouhi's first character event, neither changes actually showed up.

[esperknight]

Not weird at all when you consider developers are lazy by nature . Would you want to rebuild the pac file just to test one change? . Normally such things should be removed before deployment but hey

Very odd on it not loading your dialogue though..... That makes ne wonder too.  Did you completely remove the bin file or just rrname it?  It be could smart and just looking for the name itself and not the extension part so renaming the extension may not cut it. Ill do some more testing at home as well to see whats up.

[Vorthod Wiler]

i Just renamed the file. I tried bin.arc.old and oldbin.arc and neither one changed the result. I could try moving it completely but I feel like the second renaming method should have hidden it well enough unless the data is duplicated somewhere else or the script is somehow landing in some sort of cache.

Edit: I am not entirely sure how it is doing it, but it is seems to be noticing that the old bin is still there even when i rename it. i moved it out and it blackscreens on me before the BaseSon logo even appears.

[esperknight]

You beat me to it before I saw your edit.  What it must be doing is scanning for all the bin files.  I wonder now if that one thing isn't a checksum but an ID of some sort?  Hrm....  As there's no way it should be able to tell unless it decompresses at least part of it to see what files are in there... which is quite possible too hrm...  This calls for some testing...

[Vorthod Wiler]

Since i didn't mention this yet, i did join up with that translation group and it sounds like we have a decent plan for translations going on, the only thing we are worried about is that we may not be able to do partial patches since the main plan is to send the finalized drafts over to mangagamer so they can look over them before they make their official patch. But all in all it looks like at this rate the project can actually be completed.

[esperknight]

Awesome that it'll get done.  So it seems like this'll be official then?  Great to hear as that'll help get it more exposure.

I'd say most likely no partial patches indeed but I guess it depends... have they said if Mangagamer will do an official release or is this more like they'll host the Japanese version of the game for download and then people can use an official patch?  Curious...

I sent them this way as well as I mentioned before so I may help out too but I don't think I'll be much use if it's official.  We'll see though.

[Vorthod Wiler]

Judging by other titles, they are likely to just do a full on english release instead of a patch-style thing, even if this is only half-official. But honestly I dont know too much about this, they told me there was a general plan and i left it at that since i was more focused on figuring out how to translate what i had been given. I've just been using the chat to figure out what some of the stranger kanji mean. 

However, depending on how things go, we might want to put together some test versions (the game is massive after all), so i am definitely keeping my links to this thread in case i need to look at some of your earlier comments. they were extremely informative, after all.