News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The Console Tool (by Low Lines)  (Read 254402 times)

henke37

  • Hero Member
  • *****
  • Posts: 643
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #360 on: November 28, 2011, 08:23:09 pm »
There is one thing that I hope that you are rewriting, the hexviewer. It's fucking slow for big files in 3.1a. Not to mention that the row highlighter seems to be off by one column.

I did some work on the professor Layton games a while back, here is the results.

Quote
File formats in the 3rd Professor Layton game

This document describes the file formats unique to the Proffessor Layton games.

Several of the files encountered in the games are compressed using the stock DS compression. This document describes the format of the uncompressed files.


Layton packages
Magic bytes: LPC2
Observed file extensions: *.cpck, *.cdat, *.cani
File type: Archive
Filenames: Yes
Endian: Little
Compression: Practically always compressed

The file begins with the 4 magic bytes. The fields in the file are all 4 bytes long.
The header structure can be summarized as the following c structure:
struct LPAC {
 u8 magic[4];
 u32 count;//Number of subfiles
 u32 dataStart;//The start of the file contents data
 u32 archiveFileSize;//The total size of the archive file, headers and padding included
 u32 fatOffset;//The position where the fat is located
 u32 fileNameBase;//The base value for file name pointers
 u32 fileContentsBase;//The base value for the file contents pointers
}

The fat is composed of as many records as the count field indicates and can be expressed as the following c structure:
struct FileRecord {
 u32 nameOffset;
 u32 contentsOffset;
 u32 fileSize;
}
In order to get the absolute file name and file contents offsets you have to add the base values from the archive file header. The file name is stored as a null terminated string and uses an 8 bit text encoding.


Layton images:
Magic bytes: LIMG
Observed file extensions: *.cimg
File type: Image/Animation?
Endian: Little
Compression: Practically always compressed

Layton script:
Magic bytes: LSCR
Observed file extensions: *.cbin, *.lbin
File type: Game script
Endian: Little
Compression: Practically always compressed

Credits:
* Henke37, author of this document and lead researcher
* HCS, generally nice guy and specifically a great help in this research

I also had a go at hotel dusk, but I never did figure out much about those tbl/wpf archives. It's a fixed size record format that should be obvious from a quick peek at it. But I never figured out how to unpack the actual file data.

As for generic requests, do you think that you could throw in a raw palette loader? Some games just dump the palette two bytes per color index instead of bothering with fancy stuff.

And of course, get those gk2 archives in please.

There is also that shared font format. Given how frequent it is it would be nice to have support for that. Pity that I couldn't find any documentation better than sourcecode.

Also, basic SDAT support. Just unpacking it is a good start. Playing the samples back is even better. But playing sequences back is just unreasonable.

There also seems to be a common text format that begins with the file header MESG (Brain Age among other titles).

Finally, add support for accessing those overlays and reading the banner.

Also, have you seen those dtb/hmm/mdc/phn/str/tree files, what is up with those?

Yes, that was a long list, but I know that most if not all of these are easy and are quite possible for you to do.
« Last Edit: November 28, 2011, 08:36:37 pm by henke37 »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #361 on: November 29, 2011, 07:13:06 am »
LPC2 archives are now supported.
I have a couple of suggestions on the format spec...

"dataStart" should be "headerSize" since it seems to match "fileContentsBase"
"fileNameBase" should be "fntOffset" just to keep the naming consistent
"fileContentsBase" should be "dataOffset" or "dataStart" going by your name conventions.

As for LIMGs
Code: [Select]
struct LIMG {
u8 magic[4];
u32 headerSize; //0x30
u16 unknown;
u16 unknown;
u16 unknown;
u16 unknown;
u16 unknown;
u16 constant?; //1024
u16 unknown;
u16 unknown;
u8 flags [24];
}
I'll have to check this one out in Tile Molester on my other laptop (working off a Mac so it makes things difficult), but the rest of the data looks like image data. Whether that's one image or several I'm not really sure yet.

For LSCRs...
Code: [Select]
struct LSCR {
u8 magic[4];
u16 count;
u16 headerSize;
u32 dataOffset;
u32 nameTableOffset;
}
This is followed by count * 8 byte blocks.
Code: [Select]
struct LSCRNode {
u16 attr1;
u16 attr2; //Possibly a count
u32 offset;
}
This is followed by data at the dataOffset.
And finally a name table at the end of the file. The names in the table look like file names, so maybe they specify which file to use with the script?

The hex viewer is a lot better now. I can scroll through a gamecube rom (1.4GB) no problem, and compressed files should be using a cache, so there shouldn't be any slowness their either.

Raw Palettes is a good idea, I'll have to look into that :p

As for NitroFonts I've been considering that myself, I just haven't got round to looking at them...yet. Same goes for MESG files.

SDATs are sort of openable as archives however it would appear there's still some bugs with it. As far playback and such goes, that's on my to-do list but I got to do up a GUI for that, which probably won't be done for the next release.

Overlays and the banner are accessible as raw files through the Archive viewer, though I haven't written a format spec for either yet (also wasn't aware overlays had a format spec).

Give me a quick refresh as to exactly which GK2 formats you are on about because I've forgotten :/
Same with "dtb/hmm/mdc/phn/str/tree" it's been a while... :/
I'll have a look at the Hotel Dusk stuff once I get a hold of the game.

henke37

  • Hero Member
  • *****
  • Posts: 643
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #362 on: November 29, 2011, 07:44:30 am »
Overlays have one simple table for each cpu core and can be compressed with their own variant pf LZ77, it is a documented setup. But as for the data in them, that is not standardized and completely game specific.

As for the gk2 files, I wrote a document on those.

The "dtb/hmm/mdc/phn/str/tree" stuff has been observed in Apollo justice as well as in Brain age. I have no idea what they do. My vague guess is that it's yet another archive file.

Interesting is that the MSEG files seem to be just one format of many with a similar setup. Kinda like the sectioned file format used by the graphics, but clearly a different structure. Check "Yoshi touch & go" for more files that seem to share it.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #363 on: November 29, 2011, 07:23:22 pm »
My guess is that the "dtb/hmm/mdc/phn/str/tree" are to do with game translation, considering they're all kept in an "english" file in the American version of Apollo Justice. If that is so, then any games that have them must of a similar file system, from which these files would redirect to the appropriate files etc. Again this is all assumption here but it sort of makes sense that it would be something like that.

Hotel Dusk tbl/wpf look like they work together to create a virtual sub folder. The tbl file contains the relative file path and what I'm guessing are offsets. At first glance it just looks like a list of files, but there could be a hierarchy going on there. The wpf file contains the files each starting with a header with a matching relative file path plus what could possibly be duplicate data as is seen in the tbl file.

Have just added support for format tagging. Basically what this means is now you can target a format spec to a specific game (or series of games) by specifying the appropriate Game Code (ie YB3E for Harvest Moon: Sunshine Islands). It will scan a file's hierarchy to see if any of it's ancestors have the specified tag(s). This was needed to deal with custom formats in particular ones that don't have any kind of magic number and/or rely on filename conventions and the like. This can obviously be extended for other platforms in the future however for now I'm just focusing on NDS.

I've also re-implemented previewing in the Archive Viewer alongside file linking.


I am aware that the linking process apparently wasn't very obvious the last time I had implemented it, but until I have a better way to implement it, people will just have to deal with little "link" icons next to filenames :p All you have to do select your assets in the reverse order (Palette > Image > Cell, etc) and you'll know right away if you've done it right by looking at the preview. By double-clicking on the top asset (in the case of this example, that would be the Cell), it will open that file with the selected linked assets already included, so you don't have to open each file individually, just select them in the viewer :p I'm probably making it sound more complicated than it is, but oh well...

Textures also can be previewed, and any files that have internal assets (such as Textures which can have palettes stored in them), can be accessed straight from the Archive Viewer, as once you click on a file, this effectively is opening/parsing the file which is when internal assets get added as child nodes to their parent file. This isn't done automatically upon opening a folder/archive because that can create a bit of overhead.

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #364 on: November 29, 2011, 11:12:28 pm »
I still eagerly await a 3D model animation viewer.
On another note, would it be possible to force-open files as a certain data type? Like, I know some games use non-standard file extensions but their contents could just be raw graphics data. Another file could be raw palette. It would be a hassle to rename and the program alone has no way of knowing what type of data it is. A force-open as function would be useful here.

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: The Console Tool (by Low Lines)
« Reply #365 on: November 29, 2011, 11:45:38 pm »
At the very least, it might be useful to have the program check for magic IDs. Death Note: The Kira Game stores its graphic resources in standard formats, but packed inside archive-like files.
In the event of a firestorm, the salad bar will remain open.

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #366 on: November 30, 2011, 01:28:20 am »
At the very least, it might be useful to have the program check for magic IDs. Death Note: The Kira Game stores its graphic resources in standard formats, but packed inside archive-like files.
Yeah, I was going to say something about this too. Some games have a generic archive format of 16-bit relative pointers. Unpacking these in-program to manipulate would be a very useful feature.

Auryn

  • Hero Member
  • *****
  • Posts: 650
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #367 on: November 30, 2011, 12:36:40 pm »
The Apollo Justice and the other 3 first games of the serie use almost the same archive format as the one descrived by Henke. The problem is that there are more than one archive in one file meaning that you have a pointer table, the data, a new pointer table, new data etc... in one single file.
From what I remember from the first game, there is no pointer table that point to those subarchive (at least not inside that file).

Lowlines, did you get my PM??

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #368 on: November 30, 2011, 03:39:45 pm »
@Auryn
That's great to know >_>
Can you guys provide me with a list of these Apollo games and a sample list of said files?

As far as your PM, I can't control how another program handles files. NCGRs are also not normally meant to be edited by themselves as they are normally used to for storing image data, not displaying/editing it directly. From your description (any a quick look at the files), I'd say they work in conjunction with an NCER file that doesn't use the most common method of storing image data. At this point in time ctool cannot edit files, so I can't help you much there.

As for your other questions, no idea, except for the last one which I've already stated the other day :p

@Koetsu & Ryusui
It's all good an well that having such features would be useful, the question is how you'd go about it. If I'm understanding it right, you're talking about an in file magic id scanner? What might be more useful is the ability to split files by manually specifying where the internal file(s) sit. A force-open feature sounds good too. I will see what I can do about implementing all three for the next version.

henke37

  • Hero Member
  • *****
  • Posts: 643
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #369 on: November 30, 2011, 05:36:59 pm »
Yeah, just give up if the file is called romdata.bin and the game is a port from a platform without a filesystem. All the root pointers are going to be in the executable.

Auryn

  • Hero Member
  • *****
  • Posts: 650
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #370 on: December 03, 2011, 07:29:32 pm »
Well, a year ago or so, I worked a bit on the first game only but I was checking the other games as well and that kind of archive is more or less common across the whole serie of the Ace attorney (even Gyakuten Saiban Jiten aka anthology)/ Ace attorney investigation just with little changes.
I stay at work and I don't have roms/data in my hands but what i remember is this variations:
-one pointer table/one data block for jiten
-one pointer table/one data block with compression flag in the pointer table for AAI games (with extracted files that are sub archives with their own pointer table/data block) Henke can tell you more about this.
-many pointer tables/many data blocks in one file (for PW:AA, AATT, AJAA and AAJfA)

To find those files should be easy by exclusion analizing the files in the game aka in PWAA, a big data.bin.
AAI games have 4+4 bytes as pointer table (4 bytes offset + 4 bytes compression flag).
All other games should have just a simple 4bytes for offset.

Like I said, this is all out of my memory so this could be not exact.

For my problem, I know you can't control other programs :p
I will try to create a standard NCGR/NCER to see what happens or I am condamned to use the 1:1 export function of CT2 :(

For the Wii things, you dont' have any idea where i can find something on the net either??
Sorry for the last question, latelly there was many post about "please support this/that" and I missed that important post you made.


For the "force open" of Ryusui, maybe a simple "open as.." for the many games that store NCGR/NCER/NSCR without header is a good idea as well.

henke37

  • Hero Member
  • *****
  • Posts: 643
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #371 on: December 03, 2011, 08:03:25 pm »
Didn't an old version of CT support some variants of the stock graphics formats that the latest released version doesn't? Check the stuff from Sim city.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #372 on: December 31, 2011, 02:17:04 pm »
http://llref.emutalk.net/projects/ctool/downloads/ctool31b_multi.zip

It is currently 4:30am 1/1/12 where I live, but technically its still 2011 in SOME parts of the world, so that still kinda counts. I was a bit more busy this holiday season including last night so I didn't end up spending as much time of ctool as I wanted. I was working towards adding COLLADA support as a surprise but I only got as far as parsing the file as I still need to sit down and break down all the 3D data so I can build my classes accordingly.

I will try to prepare another update within the next week or so, which will hopefully have anything that was missing not completely implemented, which I had intended to include in this release. This also marks the first multi-platform release, however bear in mind that MOST of the testing as of the last 6 or so months has been almost exclusively on the Mac for obvious reasons and the Linux build has had no testing what-so-ever. The only real difference between builds is the OpenGL support which is auto-generated by the compiler, and some minor native support functions exclusive to Mac.

Again report any bugs you find, but don't be surprised is x file opens but displays nothing...That generally means I is working on it :p
NCERs are also only half implemented right now, but as i said I will try to put out a fixed version later.

Happy New Year!

henke37

  • Hero Member
  • *****
  • Posts: 643
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #373 on: December 31, 2011, 04:17:30 pm »
Code: [Select]
C:\Users\Henrik\Desktop\ds reverse engineering\tools\ctool31b_multi\dist-windows
-i586>java -jar consoleTool3.jar
Exception in thread "main" java.lang.NoClassDefFoundError: com/apple/eawt/ApplicationEvent
        at ctool.Loader.init(Loader.java:23)
        at ctool.Main.main(Main.java:30)
Caused by: java.lang.ClassNotFoundException: com.apple.eawt.ApplicationEvent
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 2 more

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #374 on: December 31, 2011, 10:19:58 pm »
Code: [Select]
C:\Users\Henrik\Desktop\ds reverse engineering\tools\ctool31b_multi\dist-windows
-i586>java -jar consoleTool3.jar
Exception in thread "main"
*Blip* *Blip* *Blip* End of Cheese Error
*Blip* *Blip* *Blip* Can Not Find Drive Z:
*Blip* *Blip* *Blip* Unknown Application Error
*Blip* *Blip* *Blip* Please Reboot Universe
*Blip* *Blip* *Blip* Year Of The Sloth *Blip* *Blip* *Blip*

http://llref.emutalk.net/projects/ctool/downloads/ctool31b_windows-i586.zip

henke37

  • Hero Member
  • *****
  • Posts: 643
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #375 on: January 01, 2012, 10:50:18 am »
That runs. But I have some complaints.

The main issue to me is the directory tree and how it interacts with archives. The plus symbol only shows when you have selected the archive once. This is confusing and inconsistent with how the normal windows tree control operates.

There is also some performance issues. Some files hang the UI while being parsed.

Also, I suspect that you are using the wrong parser for the cpac_?d.bin files in AJ, those have three marginally different file formats. If I could figure them out in an hour so can you. Feel free to look at my code to save yourself the trouble of figuring them out.

The SDAT parser doesn't list streams.

Finally, do you think that you could change the default table the hex editor uses? It just doesn't work for files with non text data. It is impossible to tell the difference between the byte values. Talking about the hex editor, do you think that you could add some search and bookmarking functions?

And another file format that you should look into is the SRL file format. I strongly suspect that it is a chunk of dynamically loadable code, with a filesystem to go with it.

Celice

  • Hero Member
  • *****
  • Posts: 646
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #376 on: January 02, 2012, 12:14:20 am »
Last time I played with this tool was about two years ago. I tried the latest version but I don't know how to get it running? I click the java applet but nothing runs :/

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #377 on: January 02, 2012, 02:50:39 am »
Last time I played with this tool was about two years ago. I tried the latest version but I don't know how to get it running? I click the java applet but nothing runs :/
iirc you were a Linux user. The issue which was causing the Windows version to not run (and likely the Linux as well) was some Mac specific APIs that provide native functions for the Mac version. Since these APIs aren't part of the standard Java APIs they obviously aren't found on other platforms. One of the ups/downs of Java is that it doesn't actually store copies of the APIs in the application, it only references to them and expects the ClassLoader to find them when it starts up. The easy fix is including a stub for the APIs in other platform versions, which is what I have just done now. I'll have this properly fixed in future versions as its extra work to re-build the same thing twice, but hopefully this should resolve any startup issues.

Downloads:
Console Tool 3.1b (Windows)
Console Tool 3.1b (Linux)
Console Tool 3.1b (Mac OS X)
Console Tool 3.1b (All-Platforms *Updated*)

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #378 on: January 02, 2012, 03:51:19 pm »
Wow, just tried the latest release. I'm liking it so far. It's like the expectations of earlier releases, but better. I am particularly interested in the 3D model viewer and I'm glad to see it's working fine. It actually uses the correct texture if a model uses an external texture which I think the previous version didn't do. Overall it feels improved since more models look correct. Still a few errors though. {Odd distorted points} {Texturing error}

In an earlier release, model-viewing options were available. When would these features return? Also, it would be useful to have the tool display the 3D model's internal name somewhere since the texture already does this. And when would animation support be included? I eagerly await future release of this tool because it's looking great. Keep it up!
« Last Edit: January 03, 2012, 02:36:09 am by Koetsu »

Celice

  • Hero Member
  • *****
  • Posts: 646
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #379 on: January 05, 2012, 12:37:22 am »
Ah, I am actually Windows (then XP, now Win 7). I just had a nifty GUI wrap going on that made the system look... not like default Windows :P

I got it running up now. But I couldn't get it to open ANY tlp files which was weird (opened fine with other programs though). I was wanting to go spelunking in the GameCube Fire Emblem game, looking for some unused graphics. Worked fine with some DS games though... hum. Thanks for all the work you've put into this program, it really is spectacularly awesome :)