[NecrotizingFasciitis]

Hey there!

I'd like to dump the text of the game Touch Detective on DS. Ripping some sound effects and sprites would be an additional bonus.

I have unpacked the game using DS Lazy... and that's where I hit a brick wall. There is only one big (13 MB) data.bin file in the extracted data folder, and my googling around for what to do with it results in "Fuck if we know, .bin is a generic format, you have to figure stuff out on a case by case basis.". Welp!

I can open this file in Crystal Tile 2 and I can find some text, but it's pretty garbled.
For example,
"Here's your breakfast."
becomes
"He.re's..r b.reakfastÞ"
Dunno what's up with that.

So, basically, I've no idea what to read or search for to figure this out, and I'm a complete newbie at this. Thanks for any help you can provide!

[Ryusui]

Looks like the text is compressed. Probably an LZ scheme of some flavor. Hope you're a good programmer. :3

[FAST6191]

Ooh- looks like you found another of the "all game files packed into a single file" games, shockingly this one even has the music included in there which is rare indeed (though inside it all it appears to be straight SDAT style- in the US version's data.dat it is at 9ccb94 and using the SDAT internal size it runs for 0020B2C0 - it worked fine in VGMtrans).
Files seem to be somewhat end to end as well.

Back on topic Ryusui called compression and I am inclined to agree though much of it appears to be the type 10 compression aka GBA/DS BIOS compression which loads of tools can handle ( http://www.romhacking.net/utilities/826/ - that one is a great tool if you going to code something, some of the others might have a measure of detection but I have not never got on that well with tools that claim that ability though I still like them, another compression tool I like http://code.google.com/p/dsdecmp/ ). Among those I see loads of the common DS formats ( http://www.romhacking.net/documents/469/ http://llref.emutalk.net/docs/ ) which will probably take care of most sprites.

The packing format is the tricky one- given all the instances of compression I am likely to shy away a bit from some methods. Still SDAT is uncompressed so that will hopefully lead to something
Looking at the file the initial section might be a 8 byte header header section. Not sure what that would be though (location, size, uncompressed size and compressed flag I have seen before but that does not hold so much here and the numbers never quite get large enough to be standard location pointers).
The size of the SDAT file does appear to be in there though at 9110

Sample of the header section with a 32 bit byte flip done on it.

Code: [Select]

0000009108 0000 490C 0027 32E5 ..I..'2.
0000009110 0020 B2C0 002F 5F95 . .../_.
0000009118 0000 00E2 002F 5FCE ...../_.

The numbers do not add up but the large jump from the line before would suggest I have a 4 byte entry of some form a the start of the header/file itself (magic stamp, number of files within, size of section.... who knows at this point).
Pondering it for a few seconds it seems it is a common PC trick but one rarely seen on the DS (pretty much only assembly and 3d formats for me before this)- LSL2 aka logical shift left by 2 giving the other number as an address from the start of the whole data.bin

002732E5
0010 0111 0011 0010 1110 0101
lsl2 (or if you prefer just multiply by 4)
1001 1100 1100 1011 1001 0100
9CCB94

Given this info I could probably generate you a nice batch file to extract everything ( http://crackerscrap.com/ - click on projects and get "filecutter" if you need a command line file cutter) but it seems to rapidly approaching cheesy film of clock- there looks to be in the order of 5101 files contained within though the first few might be something else.
No idea on file names or extensions yet (I usually cheat and make one from the headers of the files I extract as they usually contain magic stamps which work well).

[Auryn]

Or just jump over all theories and go straight to the goal
HERE
This is just a proof of concept, a real translation still need much more work as the 30min i played with this :p

By the way FAST, the uncompressed "text stream" will contain control codes (same style of the AA series) for animations, fades, waits, progress flags and even conditional phrases and actions for inventory management and use (look second screen).

[NecrotizingFasciitis]

Ooooh, answers!

This is just a proof of concept, a real translation still need much more work as the 30min i played with this :p

I don't want to translate anything, actually. I was kinda toying with the idea of doing a Let's Play of this game, and I'd rather hang myself than have to transcribe everything by hand. So if you could just dump the whole script in a text file, quick and dirty, I'd be good to go. :p

Though I wish you'd explain what you did. I'm quite curious about how you magicked anything out of that data file.

[Auryn]

I changed the links to my pics in my last post so that you can go view them on the correct size by clicking on them.

Like i said in my last post, you have what i call a "stream of text" meaning you can not get a "clean text dump" especially not for your needs.
If you take a look at the first pic, you see the first text on the bottom screen, just after that you see "think", this is a function to put the next screen on the top screen. Later on you see:
actor_angle: probably rotate the character sprite to a position
NAMECO: unknow
se: play sound effect??
effect: some visual effect??
wait: pause before continue with the stream
actor_anim2: play sprite animation
RINA: ID for the next anim or text
actor_say: another sprite animation
then the second phrase of the text.

How did i do the video?? You gave me some text to work on with your "He.re's..r b.reakfast", found it in the ram (decompressed so that i see how it's looking like) and in the data.bin.
Started looking for the beginning of the sub-file in the hex editor but I could not find something really clear but I had an idea to where it was.
Then started look for the pointers at the beginning of the file discovering what FAST said that those pointers don't match the file size so i lost a while trying to find if there was a second pointer table in the middle of the file but without success.
Starting to get sleepy, I just jumped to the most logic thing:
I knew the subfile is compressed and where it starts more or less so let's search for compressions and there it was exactly where i was thinking it was especially because what i got after compression, was looking the same as in the ram.
The rest was just a child play from now on, decompressed the subfile to the format you see in my pics, inserted the new text trying to not move what i didn't understand, recompress it, insert the subfile in the data.bin and record the movie

[FAST6191]

Nice, I quite like these proto programming language/crazy markup style text engines.

As you asked for it the quickest and dirtiest script dump I have ever done (I assumed ASCII and ran a 3 character strings search across the entire decompressed file and then did not tidy it up at all- I did have a quick look and you do not appear to be missing any 2 character things floating around the place)
http://pastebin.com/qMDpMiPY

Also for giggles you might as well have my batch file for extraction-
http://pastebin.com/xbMnM5u1
Filecutter is just the program I mentioned in my last post.
I have not sorted out extensions (primarily as not all the files have magic stamps which is annoying) or compression support for it at this point (some of the files seemed to frustrate dsdecmp though crystaltile2 managed to cut through it which is shocking for something like this).

If you are serious about this I highly suggest a copy of astrogrep (it is what I used to find a few things) as it does well enough on binary files as well and you can search for things within it.
http://astrogrep.sourceforge.net/

For giggles and apologies for my laziness when it came to hunting down a palette (also I might have set the tile width too high)

http://trastindustries.com/touchdetectiveexample1.jpg

[NecrotizingFasciitis]

Like i said in my last post, you have what i call a "stream of text" meaning you can not get a "clean text dump" especially not for your needs.

No, actually, it would be good enough for my needs. The "function calls" are useful info, since they tell me who's speaking, show me dialogue trees, mention changes in background music, and other useful stuff.

NAMECO: unknown
se: play sound effect??
effect: some visual effect??
wait: pause before continue with the stream
actor_anim2: play sprite animation
RINA: ID for the next anim or text
actor_say: another sprite animation
then the second phrase of the text.

Just FYI, Nameco and Rina are the talking mushroom and the girl, respectively. In other words, these are actor IDs.
actor_say would pop up the background bubble for speech. The rest seem like good guesses.

How did i do the video?? You gave me some text to work on with your "He.re's..r b.reakfast", found it in the ram (decompressed so that i see how it's looking like) and in the data.bin.
(...)
Started looking for the beginning of the sub-file in the hex editor but I could not find something really clear but I had an idea to where it was.

I'm with you all the way through this.

I knew the subfile is compressed and where it starts more or less so let's search for compressions and there it was exactly where i was thinking it was especially because what i got after compression, was looking the same as in the ram.

Aaaaand... you lost me. How do you figure out what compression algorithm is used? And how do you reverse engineer it? I don't even know what I'm looking for so I'm finding exactly fuck all on Google.
 

The rest was just a child play from now on, decompressed the subfile to the format you see in my pics, inserted the new text trying to not move what i didn't understand, recompress it, insert the subfile in the data.bin and record the movie.

Yeah. I can change uncompressed text just fine. Replacing a bunch of hex values isn't the complicated part.

As you asked for it the quickest and dirtiest script dump I have ever done (I assumed ASCII and ran a 3 character strings search across the entire decompressed file and then did not tidy it up at all- I did have a quick look and you do not appear to be missing any 2 character things floating around the place)
http://pastebin.com/qMDpMiPY

Thanks mate! As I said, that format is more than good enough for me. Would you be willing to do this for the whole script? Or, tell me how so that I can do it myself?

Thanks again for putting up with my newbie arse, guys.

[FAST6191]

I made a kind of script dump and tidied it up a bit but I did it in the wrong order (decompressed later which does appear to have caught me out according to my checks in the meantime). Again it approaches cheesy film o clock but I can give you my method though, it is very crude even by my standards and likely to miss a few things out in several of the steps.

From the position of my batch file earlier (which I realise I messed up trying to be flash but hopefully not in a way that troubles things) I ended up with several files. I should have decompressed them here but did not (Cue's decompression tools I linked earlier have a program called LZSS.exe which the command "LZSS.exe -D *.bin" would decompress things for these files unlike the version of DSDecmp I had sitting around which was a first for me- note several files are not compressed right at the start so it will probably not do those)
I then grabbed Astrogrep and though there did not appear to be a header or some other magic stamp I could latch onto I figured RINA would come up in most of them.
Copied all those files with RINA to another directory (astrogrep will provide you a list- I made another batch file to copy just those to another directory), stuck them in a tar file* and ran a strings search on that.
I am not sure what we have for generic strings search these days (it is a feature of my chosen hex editor- hex workshop which is a paid editor) and not something I often see otherwise.
I did a quick test though with http://www.softpedia.com/get/Others/Miscellaneous/tiny-hexer.shtml (one of my big three freeware editors along with XVI32 and Hexplorer) and it has a passable string search though it will take a bit of prodding to get to a useful level for you; you need to add numbers to the "must start with" list and put the lines per page cap at 99999 (or do multiple pages I guess). From there it will spit out a text or html file of all the strings it found.

*bonus of using a tar file is it lists the names of the file you are looking at right at the start of where the files are within it.

Should I have time before I pass out for the night or in the morning I might see if I can refine the method a bit or indeed get you a reasonably full text dump (hex workshop method done in the proper order and with control strings is around 800KB or 75788 lines which might preclude pastebin). In the meantime though http://trastindustries.com/touchdetectivetextbetter.7z might get you somewhere. I can not promise that link will stay up for more than a couple of weeks though.

[NecrotizingFasciitis]

^^ I followed your instructions, and...


Hee hee!

FYI, you'll miss some script files by only searching for "RINA". There are some files that describe items where it wouldn't appear. I searched for "say" for those, as that's the command line for bubbles without a speaker, which would appear on those screens. I think I managed to miss some anyway, because I'm a few hundred lines short of your total. Derp. :S
Gonna redo it a bit later.

So, my initial problems are solved, but I still have a few questions:

a) About compression. I take it that if I encounter a compressed file, I should just use CUE's stuff (lzss here) and pray something works?

b) How do I generate the batch file for filecutter? With a file that size, I hope it wasn't written by hand. Once I know that, I'm gonna try cracking Touch Detective 2 on my own for practice.

c) What are these files with the text in them?
This stuff goes
bgm[mysterious bytes that are pointing to the music file?]some other command
Is it game code? It's clearly not just plain text, because "bgm" alone is completely useless as a command (which bgm?). Is there a way to know?

d) And lastly, how do I repack the ROM? That's just because I'm curious, I don't intend to do a hack.
Here's what I tried:
I changed the first few lines of in-game text in an uncompressed file.
Re-compressed it using lzss.
Opened the compressed sub-file and the original data.bin in a hex editor.
Looked for the beginning of my sub-file in the main file (hex values search).
Selected a block equal to the size of my sub-file from that point.
Copy-pasted.
Saved and repacked the files with ndstool.
Tried playing the ROM but it's broken (just a black screen).
Where am I going wrong? :/

[Auryn]

CT2 has a function to search for compressions inside a file.
The problems with that function are 2:
-it gives many fake positives
-it crashes CT2 is you try to decompress a false positive.
In clear, CT2 give you 7517 subfiles but data.bin has "only" 5101 subfiles (that's a lot of false positives).
Because i pingpointed already the start of that subfile, i could choose the right entry on the table on CT2 and extract only that file.
What make thing not more easy is the fact that the subfiles with text, are not one after the other but mixed with graphics.

What this meas for you:
- you need a program that take the right bytes at the beginning of data.bin and make that "shift left" thing FAST told you so that you get the actual offsets.

then you can go 2 ways:

1) use CT2 search function and delete manually the 2000+ false positives fron the list and hope that then it will decompress all files for you correctly.

2) use a file splitter + a decompressor like FAST told you in the last post.

here the 2 ways join again and you need to open all 5101 file to check if there is text inside.
When you got all text files (i estimate around 500 files), you need to clean them out like FAST did.
We don't know in what order the text will be either; so maybe you will need to order all those 500 file in the right order as in the game.

It a huge amount of work to do and i am not really sure if it's like this that you do "Let's play".

[henke37]

Heh, I got inspired by this and started implementing this custom archive format myself. I've got nothing new to say about it. Buy hey, have some code!

[FAST6191]

Nice work- on searching for RINA I figured it probably would not be total. Nothing stopping you from searching for multiple different strings and bundling those, I am still looking for a unique string to the text files though (other things have nice magic stamps like IPAL which makes the lack of a such a thing here all the odder).

a) Cue's stuff will still work for the files packed into archives- you just need to unpack them from the archives or otherwise get the start of the compressed section to the start of the file (select everything before it and press delete works well and by the nature of the compression everything after it is ignored) and run it again. As Auryn said though Crystaltile2 has a compression search feature which will look for compression (in reality it is barely more than a search for anything byte that reads 10 or whatever you stick in the little box) and as long as you do not click on the false positives you can use a batch extract to get them all at once.

b) No chance of me doing that by hand. This is where I start to call my methods crude- by and large I use a hex editor and spreadsheet rather than programming anything when playing around in the initial stages of pulling apart a ROM (despite python and such like probably being a far better method). My method went something like
get hex workshop to display everything in longs rather than shorts (options- preferences - layout tab and on the bottom right for the version I have).
set width to 2 longs (click and drag the bar between the hex and the ASCII readout) as that is all we have here (other formats have all sorts of things in them but that is a different discussion).
delete the initial bytes I called a magic stamp right at the start of the file to make everything line up properly
do a 32 bit byte flip to get everything into readable format.
select the header (I was working with the header by itself from the earlier posts but it does not really make a difference) and press "edit - copy as - text"
paste this into a spreadsheet (libreoffice calc here, for older versions of excel you need to have the engineering analysis toolpack installed and I have not idea about newer versions or gnumeric). Rather nicely it will offer to format it a bit and the spaces between the sections will slice it right up for you (though you might have to get rid of a few extra former ASCII columns as there will probably be a space in there somewhere- 20 hex (the ASCII for space) is quite likely to come up somewhere) and allow you to hide/remove columns.
From here I converted to decimal "=hex2dec(cellname)" for both sizes and location columns. I then multiplied the locations column by 4 (which is the same as doing lsl2 for this file seen as none of the numbers are close to going over the end- in binary multiplying by 2 is like multiplying by 10 in decimal in that you just shift the numbers/whack a zero on the end), I could have done the LSL2 in hex workshop but that would have meant I had to paste things twice (I would have broken the lengths and computing is all about being lazy. I assume you know about the fill command, if not the bottom right of a cell or selection of cells side by side will have a little square box- click and drag it and the spreadsheet will attempt to replicate a pattern it determines from the cells above.
As commands will be predictable I arranged the numbers in an appropriate order with the relevant columns also having fragments of the eventual command in them. This is where I hold I screwed up in my batch file as I used the locations for a name rather than just counting upwards like I usually do here (fill command is quite happy to count upwards) which means I only ended up with around 700 files, I also probably copied more as my header file went past the end of the header.
Copy and paste the relevant contents of the spreadsheet into a text editor, replace the tabs with space (at other points I do things like add a # or some other unique thing if I need to direct the replace all commands a bit more) and save as .bat). If I was thinking I would have added the decompress command at the end of the file to do that as well.
The rest is just getting the programs and files into the directory and running the batch file. By hopefully giving the files appropriate names you can then put them back in the right order later.

After finding the text files with astrogrep I did a similar thing though it was far simpler ("copy [astrogrep paste] .." to stick them all in the directory below).

Most of my batch files I ever post around hacking sites are made using variations on this theme, add in the ability to output the results of command line commands to text files ("dir /b *.* >>a.txt" being a nice one as it makes a plain text list of all the files in a directory- changing the *.* to changes what you search for), basic maths (I have seen files do things like just provide a list of sizes which means you get to add them up) and knowledge of boolean logic (probably want to do this in hex workshop) and typical maths operations (can be done in either- examples are things like ceiling, floor, mod, various types of rounding) and you have my entire bag of tricks. Though I call it crude, and it is, such things are most of what people end up doing when they program proper tools; such things will only take you so far though and you will need proper tools (be they plugins for things like tinke, scripts for a hex editor scripting language or actual standalone tools) eventually.
I will also use spreadsheets to figure out things- though here the pointers counted from the start of the file (give or take the whole logical shift thing) other times they do not- get a list of what the game calls pointers, what the sizes are and if you can where the locations of the sections "actually" are (why I like magic stamps) and then a few choice subtractions (even if they are pointing somewhere else entirely they probably will be the same distance apart or in the case of relative pointers out by say 4 bytes compared to the one before) will often see you have the answers be staring you right in the face. I do not consider this quite as crude as staring at a hex editor does not do as well for figuring out mathematical relationships in my experience.

Back on topic though I do encourage looking at prequels and games from the same developer when cracking a new game there is no guarantee at all that it will the same here. That said being less than a year between them you might get lucky.

c) I am afraid other than looking at them and assuming the obvious (always nice to have human readable scripting) I have not done a thing. Human readable gets you a long way and the rest I would probably do as a search for all the various would be commands and test them out somewhere- figuring out a scripting language from assembly level backwards is frequently annoying.

d) Your method is almost there- chances are when you selected "a block of equal size" you shifted all the data following it by however much bigger or smaller your new file was which is a very quick way to break a game (doubly so when you have a game that packs everything into a sub archive). You can fix the pointers or if your newly edited file is smaller overwrite the old data leaving everything in place/pad it out to the original size.
On top of that it might also be that this ROM is one of the ones that does not work with ndstool - I imagine Auryn probably did it in place or injected the file back in with tinke or crystaltile2.
henke37 looks to have started on a proper tool though which will be a better bet.
As for your let's play if I may be so bold you might actually want to do something like what Auryn did in the example hack video- a nice custom intro done in the game engine itself could well set you apart from others playing in that world (though going to the lengths of learning some hacking methods as part of it says a lot).

[Auryn]

You asked about the sizes:
data.bin:
first 4 bytes is number of subfiles

next 4 bytes are the first subfile offset (with the "shift left" thing)
next 4 bytes are the subfile size (the compressed file size)
repeat the last 2 lines 5100 times

data
EOF

Each subfile (not tested) start with 10 yy yy 00 where the 10 is the compression ID, yyyy is the decompressed file size.

Everything naturally in little endian

[FAST6191]

Yeah number of subfiles is a fairly common thing to start off with.

As for the 10YYYY stuff that is the tell-tale sign compression format but not all files have it- some of the smaller files skipped it when I ran the decompress command, others were packed without it (saving it for the subfiles) and naturally you have things like SDAT which is never compressed (aside from the ADPCM within it) on any ROM that I have seen.

[henke37]

Actually, there are uncompressed files in the master archive. The most obvious one being the SDAT. But there is also a bunch of IPAL ones there.

[NecrotizingFasciitis]

On b), I can follow the workflow. I can avoid some headaches, though, because...

Heh, I got inspired by this and started implementing this custom archive format myself. I've got nothing new to say about it. Buy hey, have some code!

This worked like a charm, so thanks very much for it!

d) Your method is almost there- chances are when you selected "a block of equal size" you shifted all the data following it by however much bigger or smaller your new file was which is a very quick way to break a game (doubly so when you have a game that packs everything into a sub archive). You can fix the pointers or if your newly edited file is smaller overwrite the old data leaving everything in place/pad it out to the original size.

Also got that working. I just needed to pad things out with some zeros.

You asked about the sizes:
data.bin:
first 4 bytes is number of subfiles

next 4 bytes are the first subfile offset (with the "shift left" thing)
next 4 bytes are the subfile size (the compressed file size)
repeat the last 2 lines 5100 times

data
EOF

Each subfile (not tested) start with 10 yy yy 00 where the 10 is the compression ID, yyyy is the decompressed file size.

Everything naturally in little endian

Gotcha!

And now that I figured a few things out, I also stumbled on more questions (hurr durr).

I can't find any way to make the ncgr files I have not look like garbage in crystal tile 2. I can read the file header just fine, but I've got no idea how to translate that to fixing up the image. On top of that, importing nclr files doesn't seem to have any effect.
I found an illustrated guide that explains how to fix up the Pokemon Black title screen, and I can follow along just fine. I can't find out what I'm doing wrong with the Touch Detective files. :/
It's not vital to me, but it annoys me when I don't understand.

Actually, there are uncompressed files in the master archive. The most obvious one being the SDAT. But there is also a bunch of IPAL ones there.

What's a IPAL file? Can't find anything in the DS docs about that file format.

[henke37]

I think that IPAL is predecessor to the NCLR format. It's obviously a palette. It's not the first game I've seen using them.