Re:CoM Translation Project

[Rhys]

I've recently been looking into making a file extractor for the .DAT format that Kingdom Hearts Re: Chain of Memories uses to pack all of its data together so that I can hopefully edit the cutscene text, I still have no idea how I'm going to reinsert the .DAT files since they're stored in a hidden section of the disk, but I'll come to that when I need to.

I've taken a look and everything seems pretty simple, there's two tables for each DAT file, the first table holds the information neccesary to calculate the offset, file size, etc, and the second table holds the file name, binary data, etc.

The main thing I'm confused over is the LBA part of table A, it's described below:

Table A

4 Byte = Table ID
4 Byte = Table LBA
4 Byte = Table Size (in LBA)
4 Byte = Unk1
4 Byte = Unk2 (not used in all tables)
12 Byte = dummy

Obviously someone way better at reverse engineering file formats than me figured this out, so I have no idea whatsoever what Table LBA and Table Size (in LBA) means, could someone help me out here?

[Killa B]

I think it might be logical block addressing. I have no idea wtf that is but it sounds possible.
http://en.wikipedia.org/wiki/Logical_block_addressing

[Tauwasser]

http://en.wikipedia.org/w.../Logical_block_addressing <-- ?

cYa,

Tauwasser

[Rhys]

I've skimmed over that article and it doesn't really help explain how that translates into working out an offset and size..

This is an example of a table entry for table a:

ID: 01 00 00 00
LBA: 02 00 00 00
Size: 29 02 00 00
Unk1: 01 00 00 00
Unk2: 00 00 00 00

now that size may be correct converted to decimal, but then I must have to do something with LBA to find out the position, and I have no idea what.

[EDIT] hmm now that I think about it, could it be that the offset is the LBA value + Size that provides the offset for the file, or something like that? I'm just thinking out loud here but I'm sure it's something like that.

[Tauwasser]

Usually LBA should be the offset in the file and LBA size the number of bytes it takes. Of course, you might find that the LBA is somewhat off-set.

Maybe post the first file's ID data and binary data from the file it's addressing.

cYa,

Tauwasser

[Rhys]

hmm well 02000000 is way beyond the end of the file, so I cut off the last two bytes which took me to offset 200 which is another table A entry, not a table B entry, if it was a table B entry it'd point to a 24-byte file name...

It's correctly positioned though so I'm not sure what it is, maybe the first entry sets the starting address to read all of the other values?

It points to ID 69, which has the following info:

ID: 69 00 00 00
LBA: 16 06 00 00
Size: 06 00 00 00
Unk1: 01 00 00 00
Unk2: 00 00 00 00

but again the LBA doesn't point to a file name, so I'm still confused

here's the original post by the guy that reversed the format:

there is no header in both tables, you must get the file count yourself in a simple loop.

Table A:

Loop start
4 Byte = Table ID
4 Byte = Table LBA
4 Byte = Table Size (in LBA)
4 Byte = Unk1
4 Byte = Unk2 (not used in all tables)
12 Byte = dummy
Loop end

with that data, you get the table positions and sizes, the next is TableB, this Table contains the informations for the files in this tables.

(simple extract Tables in TableA and get all data seperate or get all data together and read Table B)

Table B:

Loop Start:
24 Byte = Filename
4 Byte = Unk1
4 Byte = Unk2
4 Byte = dummy or rest of Unk2
4 Byte = Filesize (in LBA)
4 Byte = File LBA
4 Byte = Unk3
Loop end

[Tauwasser]

Umm dude, it's reversed. It should start at LBA = 0x00000002 and be 0x00000229 bytes long.

cYa,

Tauwasser

[Kiyoshi Aman]

Little-endian strikes again! >.>

[Rhys]

bah I thought it might be reversed but I didn't know whether it would be or not, I don't know much about ps2 endianess or if that even comes into effect with this file format.. but if it points to offset 02 then it's pointing to part of it's own table entry which is confusing...

if it is pointing to a table a like I originally thought then that makes the following data:

ID: 00 00 02 00
LBA: 00 00 29 02
Size: 00 00 01 00
Unk1: 00 00 00 00
Unk2: 00 00 00 00

there should be 12 bytes of padding however with this there's only 10 bytes so the table data's invalid, so I'm not sure what it's supposed to be...

[Tauwasser]

No. It points to LBA = 2, that means it points to the third sector on the disk. So if you have a disk dump, then you just need to know how big a cluster is I believe. That's how I understood the explanation at Wikipedia.

cYa,

Tauwasser

[Rhys]

hmm that's confusing, from the description that guy gave it points to information about the files.. shame he refused to help me

also this particular DAT file is 18MiB, therefore it has to contain the files, you can't have table data stretching out that far...

[Tauwasser]

http://www.romhacking.net/docs/playstation.txt <-- Data size.

M_Size   bit 5   0: 2048 byte   1: 2340 byte   
M_Size2   bit 4   0:-   1: 2328 byte
So try if you find a header B segment at 2*0x800, or one of the other sizes and see it it fits. You can probably find somewhere what cluster size this is. Also, do you mind posting the second Table A entry, too? That way you could make sure if it fits with the size of the first Table B thingy.

cYa,

Tauwasser

[Rhys]

It's just padding at offset 1600, at 2*2048, 2*2340 and 2*2328 it's just binary data, that document is for the PSX, does it still apply to the PS2?

The second entry in Table A is as follows:

ID: 02 00 00 0
LBA: 2C 02 00 00
Size: 88 00 00 00
Unk1: 01 00 00 00
Unk2: 00 00 00 00

[Rhys]

It worked! I got the LBA of the second entry in table A, converted it to decimal and multiplied it by 2048 then converted it back to hex which gave me 116000, when I went to this position in the DAT file it pointed to a filename, so it looks like the cluster size is 2048.

It's pointing to SY0002.BIN, so at least that's a good sign.

[EDIT] hmm looking at a screenshot of the extractor that guy made it looks like there's multiple table A's and multiple table B's, which means for each header table, there's more than one file, it makes sense I know, but the way he's talking it seems I have to do a loop to find out how many files there are, but I have no idea what the conditions should be.

Can anyone offer some suggestions?

[Rhys]

I'm nearly done, but it's way too early to be coding now

I've finished generating a list of file information based on both of the tables. The file information is dynamically loaded based on the information in the first table, whereas the first table is completely loaded into an array so that an entire list of tables can be accessed.

I solved the loop issue by investigating the files further myself, I found that there was always padding past the table information so I simply did a check to see if I hit the padding, and if I did the loop would stop.

Here's a screenshot of what I've accomplished so far, just extraction left to go now!