Romhacking.net

Romhacking => Personal Projects => Topic started by: Low Lines on June 18, 2009, 12:08:29 am

Title: The Console Tool (by Low Lines)
Post by: Low Lines on June 18, 2009, 12:08:29 am
For the sake of archiving, I'll post links to older versions here. Just be aware that Version 3.0 onwards is considered the main (non-demo) release.

---- Demo Releases ----
Version 0.1 24/01/10
Version 0.2 24/01/10
Version 2.0 25/08/10
Version 2.5 (Mini) 06/11/10

------- Releases -------
Version 3.0 (http://llref.emutalk.net/projects/ctool/downloads/consoleTool_v30.zip) 29/11/10
Version 3.1 (WIP) (http://llref.emutalk.net/projects/ctool/downloads/consoleTool_v31.zip) 04/12/2010

------- Latest Releases (2/1/2012) --------
Console Tool 3.1b (Windows) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_windows-i586.zip)
Console Tool 3.1b (Linux) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_linux-i586.zip)
Console Tool 3.1b (Mac OS X) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_macosx-universal.zip)
Console Tool 3.1b (All-Platforms *Updated*) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_multi.zip)

[update 7/3/2011]

This site now has an official home on my website (http://llref.emutalk.net/projects/ctool/ (http://llref.emutalk.net/projects/ctool/)). Be sure to check it out :p

Quote from: Original Message
Argh my 5 pages of rambling is gone...Oh wells :p
To be honest it's kind of good timing for me because lately my project has kind of warped into something bigger recently. For starters it is no longer just an NDS Tool, as I've been busily adding support for many systems which so far include PSX, Gamecube, Wii, PSP, and will have PS2 as well when I bother to get round and make a copy of a PS2 game and read that too.

What I've also started working on in the last couple of days, is something like the successor of Tile Molester :) This will mostly be useful for reading data from older systems which don't have a File Table (ie GBA), and you will be able create a custom file table for such games which will make it much easier to manage them :) It's all still a WIP though and I'm trying to rework the way my tool works so like I can have a temp/cache folder which would simplify reading compressed/encrypted data etc And like most things when you change something, you generally have to rework a few others and stuff.

So to start off the rebirth of my thread I have a few picys of all the above :p
Note: Ohs and I've been have fun rendering the console icons in Flash :) Some are just for show atm

Viewing a PSP Game (http://llref.emutalk.net/images/projects/console_tools/screen_23.png)
Hopefully the Successor of Tile Molester (http://llref.emutalk.net/images/projects/console_tools/screen_25.png)
New Menu w/ Icons (http://llref.emutalk.net/images/projects/console_tools/screen_24.png)

Edit: Will start posting my images as links now since I got growled *cries*

Title: Re: The Console Tool (by Low Lines)
Post by: Lilinda on June 18, 2009, 12:13:31 am
Low Lines, can you downsize that pic or make it into a URL?
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on June 18, 2009, 02:18:56 am
Friggin' awesome on all counts. 83
Title: Re: The Console Tool (by Low Lines)
Post by: Killa B on June 18, 2009, 02:23:03 am
You could always use an ImageShack thumbnail. :D

(http://img25.imageshack.us/img25/3383/screen24.th.png) (http://img25.imageshack.us/img25/3383/screen24.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on June 18, 2009, 03:39:57 am
Bah, that's work! Besides I upload all my images to my website (though I've been too lazy to update the project page). You all can suffer and actually have to click on my links *evil laugh*
Title: Re: The Console Tool (by Low Lines)
Post by: KaioShin on June 18, 2009, 08:59:54 am
Can your tool replace files inside gcm images (Gamecube ISOs)? If so, awesome :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on June 18, 2009, 10:37:34 am
Not yet, it only goes as far as opening them into a treeview. But that'll definitely be something I'll have working eventually. Modifying Wii Games/WADs might not happen though :p All the data is hashed and what not, which also include a signed check by Nintendo which can't be faked without knowing their private key. That being said, hacking them would be possible, just they wouldn't pass checks or anything, and doing that is kinda complex :p
Title: Re: The Console Tool (by Low Lines)
Post by: Vanya on June 18, 2009, 12:40:47 pm
Lookin' awesome! A successor to Tile Molester sounds really refreshing right about now! Thanks for all the effert on this I can't wait to try it. ^_^
Title: Re: The Console Tool (by Low Lines)
Post by: Klarth on August 29, 2009, 12:50:04 pm
Any progress on this?  I'm really interested in Console Tool and hope that it'll be available before the end of the year so I can include it in the doc I'm writing.  Also, what kind of code signing does Wii use?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 29, 2009, 11:46:16 pm
Meh...sorry, I started doing Uni since mid-year, so most of my projects are basically suspended indefinitely or until I am really bored :p

You should check out the info at WiiBrew (http://wiibrew.org/wiki/Wii_Disc) if you want to know more about the Wii. That's only one page, but if you use the search bar on the site you should find just about anything you want to know about the Wii. Am assuming you know everything on the Wii is also encrypted.

Signing wise, I know how it all signs stuff and that, but like I'm not smart enough to implement it. Was needing it to make a Wii Save Game editor (which I made specifically to edit Pokemon Ranch Data), and it's not rocket science modifying stuff, but the signing screws you up :S Save games for instance use Elliptic Curve Digital Signature Algorithm (ECDSA) (http://en.wikipedia.org/wiki/Elliptic_Curve_DSA), which is a real pain to try and code (and beyond me at the moment), though I understand how it works and all.

[edit]

Actually I'm kinda bored today :p I think I'll work on it a bit!!

[edit2]

This probably isn't very nice for those who want to see progress. But I've kind of started a new project (as in a fresh program), and begun with a solid browser tool. This is completely custom since Java doesn't natively support pretty much any of the typical Windows Explorer functions, and I'm using the FamFamFam Silk icons for all the files/folders etc.

I figured this was a better choice than having the button menu in the earlier version I did, since I can just specify a custom file filter to only display certain files etc The "Select" button would be like you copy button, you pick a file (say a modded file you want to import), and then you can import that into a container file such as an iso or whatever :p It loads much quicker than that crappy file chooser window too. You can also add directories to the drop down list at the top for quick and easy access.

Also it's not visible but the tree view only loads what is visible in the tree, since scanning your whole file system every time you load up the app would take forever, it was a pain in the arse to implement but I think it'll make accessing container files, especially ones with shitloads of files in them, load much quicker.

The file information is actually in the tree model except it gets clipped when it gets rendered (ie you have a folder "D:\Emulation\Games" but it gets clipped to "Games"). Makes referencing and stuff much easier as I don't have to have a separate array working alongside it.

Anyways here's a screenshot...And I'll get stuck on re-adding a lot of the stuff from the older versions soon :p Am really trying to stick to a solid GUI so I won't have to change it every time I find a new shortcut/trick to make my code better.

(http://llref.emutalk.net/images/projects/console_tools/browser.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 06, 2009, 07:27:54 am
Okay its been a while since I last updated this project. I've been rather busy with my University studies so I haven't had much time to work on it. That said, I'm not in my 3 month holidays and back working on it again!

The GUI has had a lot of improvements since, making it a lot more stable and less glitchy, and you should be able to open files that are inside files that are even compressed, without having to extract nothing.

For the moment I'm focusing on re-adding the common NDS formats to the library and doing up GUIs for each type. I am still not sure how I plan to add support for custom plugins, so I'll leave that alone for now.

I've also been working on some Gamecube formats as well, but I'll add those later.

Lastly if anyone can share with me some information on LZSS compression, it would be greatly appreciated! All the documentation I've found seems to only talk about the process, and not how to read the operation bits :/

More to come soon, will post some screenshots up later of NDS formats running.
Title: Re: The Console Tool (by Low Lines)
Post by: nonme345 on November 06, 2009, 08:34:15 am
Lastly if anyone can share with me some information on LZSS compression, it would be greatly appreciated!

Here's a possible source:
So You Want To Be a Hacker? Part IV: Compression Formats (http://web.archive.org/web/20080708183914/http://sekai.insani.org/archives/24)
Title: Re: The Console Tool (by Low Lines)
Post by: Tauwasser on November 06, 2009, 09:02:39 am
What platform do you need info for? NDS, Wii? I head those were pretty special cases.
The regular LZ77 compression mentioned in the handbooks already is LZSS, yet, there seem to be new files that contain "LZSS" as magic bytes and don't quite follow the traditional scheme.

cYa,

Tauwasser
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 06, 2009, 09:18:09 am
Wells in Pokemon Colosseum/XD there are these FSYS formats, which are a type of archive. I can open up those, but alot of files in those had the "LZSS" magic at the start of the file :p

I also recently stumbled across Thakis's work on Gamecube File Formats (http://www.emutalk.net/showthread.php?t=26062), and his documentation is clear enough for me to be able to interpret it :) His work includes Gamecube Model Files (BMD) with animations, Image Files (BTI), video files (THP), and Gamecube Archive Files (RARC). I've already implemented RARC along with Yaz0 compression, though I haven't added it to the GUI yet.

I've also been looking at some of the generic PSX formats as well among other things...

@nonme345
That links looks promising! I'll have a look at it later once I get these NDS formats added.

[edit]

Okay so adding those NDS formats hasn't quite happened yet :p I've been fixing up the GUI more. As you can see, the file browser has had a fair bit done to it, as it looks prettier and I've also added the icon scheme as well as highlighting the text blue to indicate a file is compressed (which is present in the Tree View Browser as well). It will also scan the start of every file to see if there are any Magic codes as well as compare file type extensions in an attempt to make it fairly compatible.

Currently you can view the hex of any file through the browser and open up NDS roms and NARC files. But it's just a matter of adding more formats really.

(http://llref.emutalk.net/images/projects/console_tools/ctool_preview.png)

[edit 2]

Going back to compression formats now, I noticed that there are LZ77 files where the Reserved Bits equal to 1 and not 0 like they normally do. I can say for certain that some of these files are definitely LZ77 compressed because some even have the "LZ77" magic id at the start of the file.

The problem is that sometimes comes across a Copy Operation that points to a negative offset. There are a lot of these files inside the Kingdom Hearts 358-2 Days game. They have a ".z" at the end of the file name, and if you open up some of them, there's an obvious Magic id near the start of the file of what's it is compressing.

Possibly this is a variant of LZ77 Compression, which manipulates the operation bits differently.

[edit 3]
Okay I've wasted a couple of hours trying to figure out how this variant of LZ77 works with little success. The only thing I can tell for certain is the the copy bits is "minus 1" as opposed to the type "minus 3" value. Was using a compressed NCGR file from kingdom hearts and trying to match up the constant data with the output.

If anyone can figure out how this variant works, I'm all ears. For now my program will detect it but will refer to it as "LZ??" and thus won't even try to decompress these files since all it will do is cause errors!

The ".z" files in Kingdom Hearts appear to be a combination of LZ77 and this variant as some of the files will extact no problem while others will just spit the dummy on me >_<

[edit 4]

Going back to the NDS formats now, I'm up to re-adding NSCR and NCER. NSCR is pretty straight forward, however NCER is a bit tricky.

From what I've gathered, there are two different NCER structure types, indicated by a byte flag I like to refer to as the "Bank Type".

Type 1 is the structure I figured out ages ago when I started this project, it groups cells (a cell being a block of tile data of varying size that makes up a part of a sprite) into sprite frames. And seems to include to extra sub sections in the file (LBAL and TXEU) that hold frame information such as it's name.

Type 0 seems to be radically different. First off the bank data indicates that each bank object is 0x10 bytes long, so it now includes another 8 bytes of data I'm not sure about what it does. Like type one, there is an offset that points to another section at the end of the Bank Area. In type 1 this area is the cell data area (0x8 bytes long), but in type 0 this data is different again appearing to be 0xC bytes long. Then after this area is ANOTHER area which looks more like tile offset data with varying lengths.

It's going to take a while to figure this one out :p

On a side note, while I was looking through the kingdom hearts files, I can say for certain that "D2KP" files are a grouped combination of several of these generic formats (ie palette, graphic and either cell & animation or map/screen). However as these files are also ".z" compressed, my decompressor won't always unpack them properly :(

[edit 5]

To help with decoding formats, I thought I'd try and add multi-file loading support, and I'm quite happy to say that it was very VERY easy to do with the file access structure I use in my program. Every time a particular file type (as generic not format specific; ie pal, tile, map, etc), it will remember this file as the last opened one of its type. Then when you load up a file that works in combination with this one it will check to see if this file exists and then load it as well. And given the way my program accesses files, it won't matter if the files are stored in two separate places or even compressed...as long as the access path is valid, it should load it :)

[edit 6]

(http://llref.emutalk.net/images/projects/console_tools/ctool_map.png)

A site for sore eyes! This was one of the first bunch of graphic files I tested way back when I started this project. It's a nice feeling to actually be catching up to where I was the first time :)

[edit 7]

(http://llref.emutalk.net/images/projects/console_tools/ctool_cells.png)

Okay reading sprite sheets (NCER files), is still a pain (mostly because it's a format I've worked out almost entirely on my own and obvious am no expert at it). I've got them working up to the same point I had it in my old version, and I'm just trying to fix all the problems there are with it to make it more compatible.

Aside from the main problem of there being two different file structures for NCER files, there is also the problem with handling non-tiled graphics such as those found in Super Princess Peach. When it references to graphic data, it assumes the data is already stored in memory in a 8x8 tile format (as in the first 64 pixels are tile 0), so I'm racking my brain at the moment trying to remember how to convert one between the other as I originally just had a function that takes all this information and outputs a buffered image! My brain doesn't want to work for me right now so I'm going to take a break from that one for the time being...

I'll start having a look at the animation files (NANR) instead :)

[edit 8]

Reading the animations are relatively easy. In another small project I was doing I had to read Wii Save Files and animate the icons in a browser...Using that knowledge, filling in the gaps an Arcnor's documentation is simple enough...there's only a few bits of data I'm not sure about yet, and can't really test them until I build the GUI to manage viewing animations.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 14, 2009, 01:10:03 am
http://llref.emutalk.net/images/projects/console_tools/ctool_cells_2.png (http://llref.emutalk.net/images/projects/console_tools/ctool_cells_2.png)
I've been looking at some graphic formats found in Kirby Squeak Squad, which were partially supported in Tahaxan, being files with the magic stamps BCLA, BCGA, BCEA or BRCA. They are simply raw data files with a custom header at the top.

The significant thing though, was that I figured out *most* of the data format for the other Sprite Sheet Type I mentioned in my earlier posts. Basically the format is very similar to the one I've worked out, it's just differs slightly in some aspects, such as lacking any header areas, or a Name Label section at the end of the file.

There's still a few things I have to figure out first, but the above picy is of a BCEA file (Sprite Sheet) that follows this other format, some kind of rock dude from the Kirby game. I haven't looked at the BRCA format yet, but all the other ones have been added.

[edit]

Just a quick thing...I finally figured out how sprite sheets store palette information. The picy below shows a single sprite using 3 different 16 color palettes for individual cells (aka, kirby, the cloud, the cake).

http://llref.emutalk.net/images/projects/console_tools/ctool_cells_3.png (http://llref.emutalk.net/images/projects/console_tools/ctool_cells_3.png)

I've also got to figure out a what a few of the unknown bytes do as for instance going through the file above, about 15 out of 133 sprites aren't positioning the cells properly, so there must be something different about them I am yet to find out...

[edit 2]
http://llref.emutalk.net/images/projects/console_tools/ctool_sdat.png (http://llref.emutalk.net/images/projects/console_tools/ctool_sdat.png)

I have been adding a few more file formats, though I also spent I bit of time cleaning up my file accessing code, which I had found out didn't do all of what I expected it to do -_- This also included some speed enhancements with scanning files to detect their format.

The latest thing I've been working on is adding support for NDS sound files...starting with the very common SDAT format, which is actually more of an archive than an actual sound file. From what I understand there are 8 sub sections in these files, 5 of which represent actual files, while the other 3 contain data for combining these files etc. So what I've done is group each type of data in sub folders, using a fake file object to reference to non-sub file data that is found in the SDAT. I've tested it against a couple of SDAT files from different games, and it isn't too bad...except some files for whatever reason reference to nowhere etc. Once I fix up the pile of bugs in my code on this, I'll start working on the actual sound file formats (SSEQ, SSAR, STRM, SWAR, SBNK), which are contained inside SDAT files :)

[edit 3]

Wells I've just spent the last half of a day learning all about MIDI files...why? Because SSEQ files are a custom type of them. I thought it would be wiser learning how these little suckers work before trying to write some code to decode SSEQ files on the fly, but man...MIDI files are really quite complex!! But actually once you understand them, they're a lot easier to understand than some of the much more simpler file formats. That being said, the next thing on my list of to do things is obviously write a SSEQ decoder now :) Also Java has (wells I think) a much nicer MIDI sound library than the default one Windows. What would be nice though, is to learn where in the NDS file, it actually stores the sounds/algorithms for playing back sequences. I'm assuming it'd be some kind of generic file/data maybe even stored in the BIOS? Another thought was that they might be in the Utility.bin file, but that isn't always present in some NDS games so I dunno :/

[edit 4]

Okay it would appear the SBNK files may contain the sounds/instruments needed by the SSEQ files (Seems kind of obvious when you think about it), though there is very little documentation on them, so it might take a while to learn how they work. I haven't had a lot of success in porting SSEQ2MIDI to Java, simply because of the excessive amount of data these files contain, but I'll get there.

[edit 5]

Okay I've been adding all of the sound formats today. I found Kiwi.DS's SDAT documentation and it helped get a better understanding of it all :) However, Java has a somewhat limited Native Sound support, and it can only manipulate data in PCM format. Apparently it can play all sorts of different data formats but it only seems to do if said formats are in their proper file format, where as I'm trying to read stuff on the fly.

From what I understand, NDS sound files come in three types. PCM 8-Bit, PCM 16-Bit and IMA ADPCM. I can play the first two types directly out of the file without having to converting anything. Unfortunately the most common one is the last one (ADPCM), since it has better compression than the other two (from what I've researched). Now I don't have a lot of documentation on it, and I doubt it will be simple to covert it to PCM.

With luck there'll be a nice Java add-on I can use rather than the limited default package (as was with OpenGL), otherwise I'll have to learn about ADPCM >_<

Oh and the reason why I'm posting this, is because I have spent the last couple of hours listening to horrible static every time I try playing a STRM file, and I finally came across one in PCM 8-Bit format. By coincidence it was the Title Screen music for Kingdom Hearts 358/2 Days, and it sounded so wonderful to my ears!!! (Well what with all that static, anything was better lol)

[edit 6]

I've been having a go a writing a ADPCM decoder...and I've had a bit of success. Playback has gone from horrible static to clearly understanding but still fuzzy :p I've been using this site as a reference (http://wiki.multimedia.cx/index.php?title=IMA_ADPCM) (it's actually quite hard to find solid documentation on ADPCM!!)

The real pain is that Java only works with signed variables so I've got to add extra code to translate stuff.

Here's a basic breakdown of my code...
Code: [Select]
outputBuffer.size(buffer.size * 4);
stepIndex = 0;
predictor = 0;
step = stepTable[stepIndex];
for (i=0; i<buffer.size; i++) {
for (j=0; j<2; j++) {
nibble = (buffer[i] >> (4*j)) & 0xFF;
stepIndex += indexTable[nibble];
if (stepIndex < 0) {
stepIndex = 0;
}
if (stepIndex > 88) {
stepIndex = 88;
}
signedNibble = nibble;
if (signedNibble > 7) {
signeNibble -= 16;
}
diff = (2 * signedNibble + 1) * step / 8;
predictor += diff;
if (predictor > 0x7FFF) {
predictor = 0x7FFF;
}
if (predictor < -0x8000) {
predictor = -0x8000;
}
step = stepTable[stepIndex];
outputBuffer[pos++] = predictor & 0xFF;
outputBuffer[pos++] = (predictor >> 8) & 0xFF;
}
}

I've been playing around with the variables a bit *hoping* I'd strike gold and it would just be perfect...but no....Swapping between lower and upper nibble first doesn't do nothing, changing the nibble shift to 5 seems to make the sample smoother but more muffled. I figure that there either if something wrong in my code (bare in mind this was ported to Java by me), or there's some Nintendo proprietary thing going on in there. Btw if it wasn't obvious, the reference table I used are the same ones I found on the website.

Luckily I've been doing some digging and game across VGMStream, which apparently has support for most Nintendo DS audio formats, so I'll check their source code and hopefully that will sort out my problem. But if anyone is an A class expert on audio decoding, be my guest :)

[edit 7]

Still not much luck for ADPCM sound :(
I've compared my code to various implementations of it and even found out that GBATek (http://nocash.emubase.de/gbatek.htm#dssoundnotes) has some info about it also.
I'm still getting a heck of a lot of noise and/or jumpy playback.

I have however ascertained that the data is stored in blocks (size is specified in the file header), and that the number of samples in each block indicates 4 "unused" bytes as such means that at the start of every block it updates the 3 variables with the first 4 bytes of data.

Heck for all I know it could be simply Java and how it handles data (unsigned/signed, etc), I don't have enough understanding of sound playback to know myself >_<
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 13, 2009, 09:30:06 pm
Being the holidays for me, I've been doing all the stuff I've been wanting to do, which included a Stargate marathon ;D

Nonetheless I've still been working on this project off and on, except it's been slow because I'm trying to make sure the GUIs work properly with no bugs :D

Lately my focus has been in re-adding the modeling functionality, which was present in my older versions, thus making the current build finally beyond the point in which it's origins ended up. Textures are up and running with a working GUI, and I've started on the modelling GUI, which is going to be more user friendly than my old one :)

Here's a screeny of some dumb tree from Pokemon from a texture file. I'm planning on adding exporting functionality later (as in to PNG) since its just a matter of exporting a buffered image.

(http://img502.imageshack.us/img502/1492/ctooltexture.png)

I've also started adding preliminary ISO capabilities, however currently my sector reading code runs a bit slow (especially for PSX games in raw BIN format).

[edit]

I forgot how much data modelling files have :-\
So far I've got partial implementation done for display 3D models. That said, only some of the ones I've tried are displaying properly, though I'm also trying to construct a genericish model viewer at the same time :p

To get a better understanding of the NDS generic model format, I've been figuring out how data is stored in "BMD" files from Super Mario 64 DS. They are basically a simplified version of the NSBMD format, which I figure means they were possibly make with either a custom program or an older version of Nintendo's SDK. I still haven't figured everything out yet, but below is the model for fat mario :)

(http://img502.imageshack.us/img502/3417/ctoolmodel.png)

[edit2]

There are several bugs I had worked out in my older version which I have not yet been able to work out yet, and for whatever reason, I seem to have lost my solution since...In other words, I have to re-figure-out what I did last time from scratch.

It's good in a way though because I seem to have a lot better understanding now than I did then, simply because I've spent a lot more time working on it, like the fact that my app renders stuff a lot quicker now than it did before, particularly with large 3D maps.

I took a screenshot of Peach's Castle (Exterior) from Super Mario 64 DS. The holes in the textures is a combination of an error in my 4x4 Compressed Texture code, and a bug in my vector code centered around the scaling factor which I recall tormented me the last time I worked on it.

(http://img502.imageshack.us/img502/1457/ctoolpeachcastle.png)

Hopefully, I'll figure out the problems so I can get started on Animations :)
Title: Re: The Console Tool (by Low Lines)
Post by: Tauwasser on December 19, 2009, 03:49:06 pm
It's looking pretty good again :D
Have you made any additional changes to the browser? What other stuff is not working that was working in the old version?

cYa,

Tauwasser
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 20, 2009, 12:41:59 am
Wells once I get these couple of fixes in the 3D done, everything will be working from the old version, but it's already far more advanced in its own respect. I suppose the only other thing I'd have to add to make it truly surpass the old version is adding the ability to export graphics to PNG etc :p

The model browser itself can disable/enable Objects, Polygons and Materials along with partial highlighting functionality by changing the vertex color to red. I'm using Maya as a basis for the UI though personally I barely know how to use Maya -_- You can also zoom in/out and perform a simple rotate of the model with the mouse, but they both need a little more work. I want to add some sort of camera control later rather than just editing the raw translate/rotation/scale variables.

The browser has had very little changed lately since I've been working mostly on specific file types, but I'm working towards adding a live preview thumbnail which would display graphics, video, sound, models and maybe more.

So as of right now supported formats are...

Archive Formats:
RARC Archive (Gamecube)
ISO Image (Partial Support)
Nintendo DS Roms
NARC Archive (Nintendo DS)
WARC Archive (Wii - Initial Support)

Graphics/Palettes:
BCGA (Kirby Games
BCEA (Kirby Games)
BCLA (Kirby Games)
NANR (Initial Support)
NCER (Partial Support)
NCGR
NCLR
NSCR

Sound/Video:
MIDI
SDAT and its sub-types (Partial Support)

3D Modeling:
BMD0 (commonly known as NSBMD)
BTX0 (commonly known as NSBTX)
Super Mario 64 DS BMD files

Along with file scanning, you should be able to see a decent amount of file support in most Nintendo DS games, unless they use a lot of custom formats.

I still need to update parts of the GUI, with some updates to graphic handling I came up with while working on the 3D stuff, along with a much needed overhaul of the hex viewer, which I have been neglecting for ages, but possibly after I've done all that as well as solidify support the all the above formats (and probably a few more by then), it might be worth releasing so other people can have a play around with it :) Keep in mind though, it would only support viewing stuff and exporting it, because I'm trying do a bunch of different things at once, I'm leaving actual file editing to later, since it will require a massive amount of work to build GUIs for it :p I've got at least two months of holidays left, so hopefully I can achieve most of the above in the time I have before I get back to Uni :)

My current goal at the moment, is to get one of the Pokemon maps fully rendered, which would include loading multiple models at once and positioning them on the area map, and adding collision mapping :) Though I also want to get Model Animations working too!

In case anyone is interested in reading up on some Pokemon ROM Editing (http://projectpokemon.org/wiki/ROM_Editing_and_Research) notes, here's the page I'm using as a reference.

[edit]

Okay I figured out what I was doing wrong...and as soon as I did, I remembered I had done it before too (déjà vu? lol). In NSBMD files, each object has a Transform Flag, and basically I was reading only one byte instead of two (hence causing some flags to result in zero). Changing this fixes the pivot/rotation factors, however there are still some other things as well. The below is old Petey, now looking much better than he did this morning, however a bit of his face is missing. Now I know I have fixed this before because I've still got the original screenshot I took a year ago, so now its all a matter of figuring out what I did to fix it. On another note, there are also still a bunch of variables I haven't implemented (mostly because I don't know what they do), which affect models from other games, such as materials for models from Phantom Hourglass. Link sadly also looks like a flag pole at the moment too -_-

(http://llref.emutalk.net/images/projects/console_tools/ctool_packun.png)

[edit 2]

The problem with models like petey is that each polygon also seems to have an initial scale factor as well, which is why the first chunk of the mouth appears as though its missing, it actually is there, just not scaled properly. So far I haven't been able to isolate the exact bit of data which specifies the initial scale, but by not resetting the overall base scale value to 1 it seems to fix it on the second refresh of the screen. Not a proper fix though, and would no doubt be prone to errors once more things are added.

[edit 3]

Taking a break from models, I've started looking at model animations :) Although kiwi.ds's code for reading NSBCA files is far from incomplete, it's given me a start. It's formatted in very much the same way as NSBMD and NSBTX files.

The basic structure up to where I am is as follows...

BCA0 Header (0x10 bytes)
 - Section Offsets (since there is only one section, this is only a single 4 byte offset)
 - JNT0 Section (contains 3 sub sections which are present in both NSBMD and NSBTX files, namely UNKNOWN, DATA, NAMES)
------------------------------------------
Following this each animation jumps to its own section that begins with a J_AC (0x4A004143) magic stamp.
Each J_AC store the initial settings for each animation as well as points to places in the latter part of the file relative to the start of the J_AC. The offsets for every J_AC point to the same two places in the file.

kiwi.ds only got as far as the J_AC sections, which I figure is because they have a really weird way storing information.

For each object in a J_AC section, it seems to hold the base settings for Translation, Rotation and Scaling, as well as the Object # and a currently unknown byte (which always seems to be zero).

The settings a referenced using a 2 byte flag which is as follows...

xxZYXBSCxRZYXBTx

if T, R, or S equal to 0, data is present for each of their corresponding types (ie Translate, Rotation, Scale).
Both T and S control the following 4 bits to their left, being B, X, Y, Z.
R controls C.

if the corresponding flag equals to 1 a DWORD follows (except for Scale XYZ which use two DWORDs).
otherwise two WORDs and a DWORD follow.

This is how I've interpreted the data, which seems to fill in all the blanks in kiwi.ds's code, and so far has all data that follows the flag present and accounted for. So now I just have to figure out what the hell it all does :p

This isn't a very good explanation, but hopefully it sheds some light on the currently unknown NSBCA file :p

More to come once I figure that out.

[edit 4]

Animations are a pain.
When a flag is set, the data that follows are the values that don't change throughout the animation, essentially if say a character is walking to the left, the Y value may remain constant for the whole animation. But when it's not set, 8 bytes follow which reference to what happens for each frame in the animation. Breaking it down, the first two bytes always seem to be zero, and the last four bytes are an offset relative to the start of the J_AC section, which seem to always point to somewhere after the second offset found in the J_AC header.

The first offset section is made up of 6 byte chunks, the first two bytes are some kind of id or number and the last 4 bytes make up two word size signed values or a single signed dword. So I figure this is some kind of matrix table.

The second offset section seems to hold a variety of different data. Most often the first chunk seems to hold multiple two byte chunks, the first byte incrementing by 1, but this varies.

The problem I am having is that there doesn't seem to be a constant size for data referenced from the second section. Assuming references don't overlap, it seems that the third byte in the above mentioned reference data specifies how the data is stored, and more importantly right now...how big it is.

Say the animation is 30 frames and the Translation X value changes throughout it? So assuming the first section is the matrix table, you would need at least 30 bytes to properly reference which matrix from the table needs to be loaded into the current matrix. However I've seen data which which is 32 bytes, or even less than 30 bytes which is somehow meant to represent the entire 30 frames for that particular variable. Then there's also the possibility that the CAN be a zero sized first section as well (meaning there is no matrix table). There's at least 7 different types which so far have made no sense to me at all :(

So like I said, animations are a pain, but I'll figure it out eventually :)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 31, 2009, 10:16:48 am
Lately I've been doing a bit of GUI cleanup as wells as updating the code with some of the new stuff I've been doing with the 3D Modeling section. Right now I am determined to get Cell-Based Sprites 100% working. I've managed to fix a few problems I've been having with them, getting most sprites to display properly now, but there's still a few bugs with non-tiled sprites. While adding a decent grid layout to the GUI I've realized what the rest of the data in cells actually do, I'm just not quite sure how they work yet :p

(http://img502.imageshack.us/img502/9901/ctoolcellsnew.png)

The above is probably one of the most complex sprites I've come across yet (having a whooping 38 cells in it), and as you can see, there are errors. The problem lies in the last bit of unknown data found in each cell. For this sprite, the value is 2, which somehow is meant to shift everything to the left more. The grid is made up of 4x4 64x64 blocks, but with bigger sprites such as this one, cells can be positioned to stick outside this range, meaning the maximum size for a sprite of this format is 384x384, 192 pixels in any given direction from the center point (0,0). Once you take into account the size and position of the grid, alot of sprites I originally thought were displaying properly are actually wrong because they aren't being centered, which is what the last bit of data is somehow supposed to do.

So once I figure out how to position the cells properly and fix the bugs with non-tiled sprites, Cell-Based Sprites should work 100%

[edit]

(http://img502.imageshack.us/img502/779/ctoolcellsfixed.png)

Fixed. The unknown data is basically 4 flags, two of them describe flipping horizontally and vertically and the other two flags  specify whether their respective axis uses a different shift table.

Each cell has three variables that make up the amount of shift is added from the base grid position, normally these are just multiplied by 16, 4 and 1 respectively, however when the X or Y flag is set, the first shift variable uses a different set of positions instead. It's rather hard to explain, but basically whatever I'm doing seems to fix all problems with cell positioning in every Cell Sprite I've tried. There are probably still a few bits missing here and there, but until I find a file that uses them, I can't really do much.

The only other major problem I'm having is with referencing tiles from a non-tiled image. There are 4 types of image files. Firstly there is the 8-bit image, which I'm pretty sure is always non-tiled. Then there is there 4-bit tiled image which is the most common one. The last two are 4-bit non-tiled. One is basically treated as a single image (typically with a width of 256), where smaller images are all packed in neatly into a rectangle.

The last type, which is obviously the one I'm having trouble with is basically multiple images that sequentially follow each other. For example, the first image is 32x16, the second 16x16, and the third 16x32.

So far I've only come across images of this type in the game Digimon Championship. The NCER files that use them are also interestingly use the other banking method, which has alot more unknown data. Hopefully I'll figure it out eventually :p
Title: Re: The Console Tool (by Low Lines)
Post by: Spikeman on January 02, 2010, 07:42:57 am
Just want to say keep up the good work man. This is one of the coolest projects being worked on in my opinion. :)
Title: Re: The Console Tool (by Low Lines)
Post by: Klarth on January 02, 2010, 04:29:39 pm
Agreed.  Though to be honest, 95% of my interest is geared towards PSX CD-image building and extraction so I won't have to code it myself.
Title: Re: The Console Tool (by Low Lines)
Post by: Gemini on January 02, 2010, 05:21:05 pm
Can't you already do that with pixel's Cd-Tool? I'm pretty sure it can even manage correctly virtual file systems (if they aren't too complicated, that is).
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 02, 2010, 05:33:03 pm
Agreed.  Though to be honest, 95% of my interest is geared towards PSX CD-image building and extraction so I won't have to code it myself.
Yeah I had started working on that, except the code I came up with for reading PSX BIN files was really slow (because you have to jump to a sector, read part of it and then read the next sector until you have your file), which is really bad for my automatic file detection because the only way I know of identifying an iso is by reading sector 16 which is the primary header or whatever, so any files bigger than 16 sectors will get checked and relying on the ".bin" extension is useless because its a VERY popular file extension for MOST files in games -_-

What I've found is that its much quicker to load load large files into Java's memory than to be constantly accessing it through file access methods, however Java also gradually builds up wasted memory space, so over time especially when accessing large files, it will run out of memory. I've already increased said maximum memory to 256MB which seems to be enough even for accessing some of the larger files from NDS games, if anyones used Tile Molester, they would know it's limited to files at a maximum size of 64 MB, thus making it less useful for direct editing of a lot of the new game images.
Title: Re: The Console Tool (by Low Lines)
Post by: Klarth on January 03, 2010, 12:33:25 am
Can't you already do that with pixel's Cd-Tool? I'm pretty sure it can even manage correctly virtual file systems (if they aren't too complicated, that is).

Hmm...I suppose I could write a general purpose script to dump and rebuild PSX games via Cd-Tool.  Just that I'd hate to have to use LUA.  On the other hand, it could save me a lot of time.  I'll probably go this route.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 03, 2010, 04:09:27 pm
Hmm wells I was "hoping" I would have a nice little preview of sprite animations working by the end of the day, but no...GIF images, I never knew how much of a pain it is to read them (let alone write them from scratch). I spent all night trying to make head or tail of the Wiki documentation on LZW compression, before finally understanding how it works!! It's very vague I think and the examples have horrible paragraphing which makes it even harder to line stuff up!!

So tomorrow, I'm going to try and get GIFs supported, and THEN write an exporter so I can show off animations :)
Also after doing some research, I think that maybe adding support for APNGs might be an idea too...I mean they aren't heavily supported yet but wells...look at PNG :p

The last thing I'd like to make note about today, is correcting myself on some of the notes I've been making on the NCER format. For starters the mappable range is actually 512x512 and not 384x384, and also the so called "fix" I came up with actually did nothing ... increasing the mappable range was what fixed the crocodile sprite I showed earlier. All NCERs that follow the offical format work fine, it's just the custom Kirby BCEA ones that are playing up, where like the bottom half of a character is positioned above its head, with no obvious errors in my code -_-

Ohs and I've been doing a bunch for fixes on the GUI, such as properly implementing Byte Order swapping/detecting (so you can reading both types without having to worry too much about it), and I have made a start on the Hex Viewer, starting with some dumb bugs that have haunted it for the past year etc.

Time for sleep...and I shall dream about GIFs laughing at me :'(
Title: Re: The Console Tool (by Low Lines)
Post by: creaothceann on January 03, 2010, 06:25:24 pm
I spent all night trying to make head or tail of the Wiki documentation on LZW compression, before finally understanding how it works!! It's very vague I think and the examples have horrible paragraphing which makes it even harder to line stuff up!!
Yeah, I tried to understand it at one point and gave up because of the documents. :P
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 04, 2010, 12:30:10 am
Wells LZW, from what I understand is basically a command based compression method, as opposed to say LZ77. Data is broken up into 9 bit lengths and read one at a time. The first code in any data should always be 0x100 which is the command called CLR. Codes that are between 0x00-0xFF are uncompressed colors, and codes between 0x100-0x107 are commands. I'm still in the process of understanding how each command works but even my crappy explanation makes better sense than the "proper" ones I've found on the internet. Once have it figured out, I'll try and explain how I did it :)

[edit]

I think I'm starting to understand LZW better now. For a gif, the initial table of values is from 0x0-0x101...0x0-0xFF obviously being your 256 colors, and 0x100 and 0x101 are reserved for "CLR" and "END" commands respectively. Any code greater than 0x101 points to the extended part of your table which gets updated as you go through the file. A code value of 0x102 basically means the first color 0 repeated twice, a value of 0x103 is the first color and the second color (ie 0, 1) and so on.
The really confusing part in all the documentation is all this "string" crap they go on about, like one author claimed that every code could be translated into a string, even though string character don't start until 0x30 in terms of bytes, which makes no sense to me. So instead I'm using a double byte array for my "string table". I'm no expert, but honestly how can comparing strings be reliable when doing something like this? Anyways with luck the route I'm taking will just work and I can move on and get GIFs out of the way...

I also attempted at adding PNG support (in which to further add APNG support), but again I got stuck on the compression part, DEFLATE being significantly harder for me to make sense of compared to horrid little LZW.

However I really like the file structure of the PNG, it stores each piece of data in its own chunk, allowing for custom chunks to be added without crippling support with earlier versions. In the future I think making up my own custom formats using this chunk structure may be a much better way at manipulating files such as grouped ones like NCLR, NCGR, NSCR, NCER, NANR. My app works in a similar way when handling multiple types of data, it checks if combinable files are presently stored in memory and then loads them together. Also having a custom format, would allow me to retain a lot of the data that often gets lost when converting custom formats to pc based ones, especially with ones like mappers and sprite cells. It's just an idea at the moment though.
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on January 04, 2010, 02:31:54 pm
http://en.wikipedia.org/wiki/DEFLATE

Doesn't seem that complicated...if you've ever worked with LZ compression and Huffman coding, combining the two makes perfect sense.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 04, 2010, 11:10:57 pm
http://en.wikipedia.org/wiki/DEFLATE

Doesn't seem that complicated...if you've ever worked with LZ compression and Huffman coding, combining the two makes perfect sense.

It's mostly that I get stuck at a particular point in the process because I don't understand. I haven't implemented any form of Huffman yet, so that's all new to me, but I do understand how the string table works, but what I don't understand is how I generate it and read from it in the decompression process. I probably should be doing it the other way around (compression then decompression), but wells :p

Here's the data of a PNG IDAT chunk representing a one pixel image with an RGB of (237, 28, 36).

Code: [Select]
0x185763782BA302000327012E
The first two bytes are ZLIB flags and the last 4 bytes are a custom CRC of the decompressed data.

The ZLIB flags tell me 3 things. Compression mode is DEFLATE, Window Size is 2048, and there is no "preset dictionary" present before the actual data starts. So therefore I move to byte 3 (0x63) which is the start of my DEFLATE block.

The first three 3 bits are the block header. The first bit indicates whether this is the last block (which it is), and the other two specify how the data is stored (which is method 1). After this point I am stuck. I know I'm supposed to read each bit one at a time and travel down the tree until I get a value, but I don't know how to start it.

I'm using this document (http://www.gzip.org/zlib/rfc-deflate.html#block-details) as my reference. I have been using the Wiki reference too, but some of their notes on this stuff is a pain to understand (like with LZW).
Assuming my process is right up to this point, compression type 1 uses fixed Huffman codes (see link). I see the information in the documentation (Section 3.2.6) but I'm not sure how I'm supposed to use it etc. Like am I already supposed to have a partial table generated? It may seem simple to some (or sadly most), but it's new to me and I haven't looked at it long enough to go crossed-eyed and see the hidden picture :p
Title: Re: The Console Tool (by Low Lines)
Post by: KaioShin on January 05, 2010, 04:39:02 am
To decompress Huffman you need the Huffman tree. It must be stored with the compressed data somewhere, probably in some sort of header. If you have the tree the rest will be easy.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 12, 2010, 05:20:55 am
I managed to get GIFs mostly decompressed, just there's a bug in my code that stuffs up LZW decompression at some point and that's as far as I got with it :/

Putting aside my war with the compression formats...I've started working on some gamecube stuff. GCM files should now work and Yaz0 compression has been added along with RARC archive support. My main focus however is adding support for gamecube models. I figure that in order to understand NDS models better, I should see how a different model format works (a more complex one), and then build my model viewer based on that.

So far I've added support for most texture types, S3TC, I4, I8, IA4, IA8, CI4, CI8 and partial support for RGB5A3 and RGB565. My next step is understanding all the model data (which there is ALOT of...). Expect more to come :)

(http://img72.imageshack.us/img72/6198/ngctexture.png)

[edit]

Okay...this is my first attempt at drawing gamecube models...It's a hat!! Can't you tell??

(http://img72.imageshack.us/img72/6056/gccap1.png)

I'm using what notes thakis (http://www.emutalk.net/member.php?u=21602) has posted on the formats, though I'm pretty much at the point where I have to figure things out on my own. Having taken hours just writing up the code to interpret about 80% of the file, I just wanted to see my progress...but obviously there's still a long way to go ;D Here's what the model above is SUPPOSED to look like if anyones interested in comparing :p

http://www.emutalk.net/attachment.php?attachmentid=21789&d=1109088267 (http://www.emutalk.net/attachment.php?attachmentid=21789&d=1109088267)
Title: Re: The Console Tool (by Low Lines)
Post by: BRPXQZME on January 13, 2010, 10:24:23 am
Mmm... polygon goulash....
Title: .
Post by: creeperton on January 13, 2010, 12:49:52 pm
.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 14, 2010, 07:56:00 pm
Okay not so meshy anymore :p

(http://img72.imageshack.us/img72/2542/gccap2.png)

I've been updating the model GUI a little as well, and reworked my model viewer to operate in a chunk like style instead of the piles of custom code I had for viewing NDS models specifically -_- Hopefully I can figure out why these gamecube models are being stretched out :)
Title: Re: The Console Tool (by Low Lines)
Post by: Vanya on January 15, 2010, 12:16:03 am
Have you released a demo yet?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 16, 2010, 02:19:47 am
Yay progress!

(http://img72.imageshack.us/img72/2629/gccap3.png)

99% of the problem was me not understanding the documentation :p I found an easy to interpret Floating Point Calculator on Google and from that I realized that "**" apparently stands for "to the power of". I also looked through Thakis' bmdview2 source and finally understand how he reads Fixed Point Values as well. So basically now simple models (that don't require Joints etc) display properly and I am feeling relieved to not be staring at meshies all day.

I actually only spent like 15 minutes on figuring out my problem with the Gamecube models, as I've been overhauling my NDS model code more and gradually improving the model GUI as well.
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on January 16, 2010, 02:43:35 am
This is going to be fricking incredible when it's finally released.
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 16, 2010, 03:43:18 pm
Awwwweeeeeessssoooommmeeee.
Title: Re: The Console Tool (by Low Lines)
Post by: Penance on January 16, 2010, 03:59:29 pm
Wow, completely astounded. Can't wait to play around with this.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 17, 2010, 09:28:38 am
Sorting through all the data in these Gamecube Files is seriously hectic, mostly because a lot of things point to another offset which then points to ANOTHER offset before you "might" even get the data you want. Before you know it, you have 100 lines of code purely for seeing this data so you can get your head around it.

Anyways, I've halted work on them for the moment because I figured if I could get my Plugin Concept operational, it would make decoding files with massive amounts of data like this much quicker...or at least I hope it will...

(http://img72.imageshack.us/img72/5126/plugintool1.png)

This is VERY much still a work in progress, so bare with me here. Now the idea is that plugins are stored in XML files made up of 5 or more sections. Firstly the header, which has all the information needed to recognize the file and give it a custom icon etc. The second part is a file structure tree that maps the contents of the file. Variables that rely on other variables will use a simple byte code to reference, test, jump to offsets and whatever.
The next section is for extended code, and/or I may put all byte code here for neatness or whatever later.
The fourth section is for external variables. Sometimes the data you want has to be decoded before you can actually make use of it etc.
The last section is simply the read/output function. It gathers up the data needed to load the file using my generic data structures into the app.

Now the idea is that once I have this up and working, you'd be able to write your plugins via the GUI using the window above. So far my plugin tool can load up the NCLR plugin, and seems to actually check for the PMCP section (which isn't always present) and omits it from the tree. But it's buggy as hell and my byte code decoder is VERY basic atm so there's still work to do on it.
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on January 17, 2010, 03:34:44 pm
That sounds awesome, will you be providing any inbuilt decompression for plugins for the standard compression types like LZ77?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 19, 2010, 03:35:21 pm
If anyone belittles how much work it takes to make GUIs, seriously...shoot them! It's taken me a fair while to think up / build and design all the GUIs I believe I need for the plugin editor, and that's not including my perfectionist like manner :p

Anways here are the preliminary GUIs that will make up the Plugin Editor.

(http://img502.imageshack.us/img502/436/plugingui.png) (http://img502.imageshack.us/img502/9739/pluginicon.png)

So far only the Plugin Editor/Info dialogs are actually part implemented. If anyone suggests or ideas that I could include, feel free to say so. The idea of this whole project is making something like Photoshop/Flash which is so easy and I suppose "fun" to use once you know how they work. So when I design the GUIs, I'm building as to how I would like a GUI to work, and not simply a quick and easy way to get the job done.

Oh and in case anyones wondering, the icons I'm using are from FAMFAMFAM (http://www.famfamfam.com/lab/icons/silk/) but I'm also looking at adding Fugue and some others which I think go well together (as the FAMFAMFAM icon set is limited somewhat), thus increasing the variety I'll have for file icons AND the GUI. I'll probably also make up some custom ones after I get past the goals I already have with this project.

Title: Re: The Console Tool (by Low Lines)
Post by: BRPXQZME on January 19, 2010, 03:41:25 pm
If anyone belittles how much work it takes to make GUIs, seriously...shoot them! It's taken me a fair while to think up / build and design all the GUIs I believe I need for the plugin editor, and that's not including my perfectionist like manner :p
While there are planning strategies that make this kind of thing easier, most good GUI coders I know code drunk....

They are different people from the good UI designers I know, mind you
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 19, 2010, 03:48:55 pm
While there are planning strategies that make this kind of thing easier, most good GUI coders I know code drunk....

Lol wells, I don't even drink alcohol, so I dunno what that says about me :p I get to cheat a little though, I am using Netbeans to build the GUIs, it's more or less a drag and drop for the most part, but I have to go in and make all the fine details myself, and custom modifications like the tree, list and the icon picker are coded from scratch. It also takes time to make sure everything reacts properly, like the Info/Hierarchy/Code dialogs are children of the Plugin Editor dialog, so if you minimize it or the main window, it has to remember which dialogs were open etc. Then there's interaction between dialogs etc, the list goes on, but they're kick ass once they're done, which is the whole point :)
Title: Re: The Console Tool (by Low Lines)
Post by: Nightcrawler on January 20, 2010, 09:06:21 am
Use a design pattern that separates out the view domain such as an MVC/MVP (http://en.wikipedia.org/wiki/Model-view-presenter) variant. You can then rearrange, tweak, or even change display technologies and your code hardly changes at all. Even better, you can then have some test code to test your software for you in an automated fashion.

A smart design pattern such as the one mentioned will make nearly all of the things you mentioned much simpler and not much of a problem. I wouldn't do a large scale GUI heavy project without one. I prefer Passive View (http://www.martinfowler.com/eaaDev/PassiveScreen.html) which is an MVP variant that basically takes the concepts to the extreme for maximum benefit.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 21, 2010, 01:18:57 am
Use a design pattern that separates out the view domain...
I don't quite have a very good understanding of all that, but I think I'm already partly doing it anyways :p

I've been spending most of my time being nit-picky about the look of parts of the GUI so I haven't gotten too much more done on it, however I have an urge to get back to working on the formats again :p The plugin tree is about part way to where I want it. Essentially, the aim was to render the XML plugin into a tree made up of tables, basically that it can function as the documentation for that format at the same time. Below is my start on that...
(http://img88.imageshack.us/img88/9750/plugingui2.png)

The grayed rows pop up when the next node has skipped some bytes, which ought to help with error checking etc (I did it deliberately here though). Generally when I put something aside for a bit and then go back to it, it turns out better anyways :p

[major edit]

Okay, I'll be honest and say that I've been considering it for a long time now, I'm going to release a demo version. However as of right now I'm on a Battlestar Galactica marathon so I have no clue as to what is working and what isn't, and if there are any major bugs, I'll fix them up ASAP. If the program stops working, simply close it and reopen it and all problems should be resolved. Also when reading a large number of files (or a fairly chunky archive table such as Gamecube ones), the program WILL appear as though it's not doing anything, but this is because it's checking all files for specific formats and I haven't added any processing dialog yet. The only time you can know for sure if you've come across an error is if the GUI doesn't display properly.

You should be able to export files (which includes automatic decompression) which should be handy for those who need a quicker way to extract stuff and don't want to use 5 programs to do it. It will decompress stuff to the same folder with "_extract" added for decompressed files. You can also bookmark folders which are saved to the settings.ini which will work with future versions unless I choose to change it which is unlikely as of this moment.

Other than that...happy hunting!
consoleTool_v01b.zip (http://llref.emutalk.net/downloads/consoleTool_v01b.zip)

[edit2]

There was a problem with the OpenGL extensions, causing Model stuff to not work...Update with fix...
consoleTool_v02b.zip (http://llref.emutalk.net/downloads/consoleTool_v02b.zip)
Title: Re: The Console Tool (by Low Lines)
Post by: samurai goroh on January 25, 2010, 12:15:03 pm
Cool to finally see a release (even if it's a demo), I'll check it out right away :D
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 26, 2010, 04:48:28 am
Since posting the demo, I've been steadily working on alot of the bugs...That noted here's a list of them and my progress...
Code: [Select]
Browser
 - Loading folder bookmarks (which my cause an error) *fixed

Sprite Cells/Animations
 - Overall GUI problems *fixed (as far as I can tell)
 - Sprite Palette Indexing *partially fixed
 - Tile Mirroring *fixed
 - Screen/Maps Problems

3D Models
 - Material Size Bug *fixed

ISO Loader
 - Doesn't work

Also note that the Plugin Editor is disabled, and the Hex Viewer only displays files (meaning you can't select chunks or anything atm).
I've also updated the SM64DS model plugin to work with the new data structure I'm using.

I was also randomly looking at model files of one of the Nintendogs games and had a quiet giggle at this...

(http://img88.imageshack.us/img88/6927/ndogschi.png)

Reminds me of something from Resident Evil XD...Not so innocent now huh Nintendo?
Title: Re: The Console Tool (by Low Lines)
Post by: Gemini on January 26, 2010, 06:27:32 am
Is that a NintenB.O.W.? :D
Title: Re: The Console Tool (by Low Lines)
Post by: BRPXQZME on January 26, 2010, 08:24:22 am
Can’t... unsee....
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 26, 2010, 08:38:40 am
Can’t... unsee....
If you are talking about toggling visibility of model stuff, it wasn't readded (alot of stuff had to be disabled for debugging purposes) when I released the demo. It will be back in future releases...

I think the reason why the poor puppy dog is mutated is because the initial stack id for polygon0 is wrong, which is set in the Bones section. I've managed to fix a some of the NSMB models (such as goomba) which used Bone Command #5, which seems to set the initial stack to the last one in memory.

Atm I'm looking at NSBTP files, which I am almost certain are texture animations...

[edit]

Ohs and I've also fixed A3I5/A5I3 textures which display wrong (bright green means errors).
Title: Re: The Console Tool (by Low Lines)
Post by: BRPXQZME on January 26, 2010, 11:16:25 am
Yeah, I was talking about the cute puppydog gone horribly wrong ;_;
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 26, 2010, 07:43:01 pm
Okay, I shot my way through the Umbrella Corporation and found the antidote to return ninpuppy to normal 8)
(http://img88.imageshack.us/img88/2651/ndogschifix.png)
Cause...one bone command had the wrong length, causing it to skip/break on the next command, and also the stack initilizer was using the object's id instead of the stack id for that object.
Title: Re: The Console Tool (by Low Lines)
Post by: Killa B on January 26, 2010, 08:46:11 pm
Not sure if I've already said this or not, but this project is really fucking cool.
Title: Re: The Console Tool (by Low Lines)
Post by: DarknessSavior on January 26, 2010, 10:13:45 pm
Aww...I liked the zombie-puppy.

~DS
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 27, 2010, 06:56:19 am
First off, I've fixed a bug where opening an archive within an archive (ie in a NDS ROM) that contains folders, caused the tree to stuff up. Yay! Also, in case you missed my post, from now on, any releases will be added to my signature...

Atm I'm focusing on Zelda Hourglass, and am trying to get most models to work properly, as well as examining the four Model Animation formats, they being, NSBCA, NSBMA, NSBTP and NSBTA. Though I'm not certain, here's what each animation format does...

NSBCA...animates models by manipulating the geometry via Rotating/Scaling/Translating etc
NSBMA...seems to manipulate materials somehow, perhaps via transparency, or positioning. Examples I've seen generally are a form of lighting such as an object that glows.
NSBTP...animates materials by swapping textures much in the same fashion as sprite animation. An example would be link's eyes blinking.
NSBTA...seems to manipulate textures, most likely via positioning. An example would be the waves in zelda.

The file structures of all 4 are fairly similar, I just haven't determined how all the data is stored quite yet, but I'm working on it....
Title: Re: The Console Tool (by Low Lines)
Post by: Penance on January 27, 2010, 10:38:14 am
Such an amazing tool. I can't fucking wait to try it.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 28, 2010, 01:23:46 am
I think a lot of bugs in the models is because of poor interpretation of the Bone Commands. Kiwi.DS's implementation generally works well with most models, but it tends to stuff up on weird ones. I think this is due to variables not being inherited properly etc. I've been trying to fix all problems, but it's more or less fixing one model breaks another so there's still lots of figuring out to do.

An interesting result of my testing produced this...

(http://img88.imageshack.us/img88/7076/ndogschi2.png)

I call it Nindumbopuppy XD

[edit]

Fixed texture mirroring.
Added texture scaling (mostly works).
Added texture clipping (though I'm not totally sure where the flags are stored in the material data...).
Title: Re: The Console Tool (by Low Lines)
Post by: Spikeman on January 28, 2010, 09:34:35 pm
Do you want us to post any buggy models we find? I was looking through Custom Robo DS and some of them displayed correctly but a lot were completely broken.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 28, 2010, 10:20:59 pm
Do you want us to post any buggy models we find? I was looking through Custom Robo DS and some of them displayed correctly but a lot were completely broken.
No point really, generally the models that are broken don't point to stacks in a sequential order (ie obj 1 = stack 1, obj 2 = stack 6), plus the so called "dummy ID" seems to also be used somehow too as some models store numbers in there too. I'm not just aiming for models to display properly, but for objects to register the right parts to them...For instance, with the dog model, when it is displaying properly, "tail1" selects parts of the dogs back leg etc.

Atm I'm trying to get one of the phantoms for Hourglass to display properly.

(http://img88.imageshack.us/img88/9208/phantom.png)

Kiwi.DS implementation sets the matrix stack using the stack id, if you check out this model (BattleChaser), you'll see all the limbs display properly, but the chest but breaks. The implementation I used here uses the object id when setting the matrix stack, the result being that the chest displays properly but the right leg is broken. The most annoying part is that for some reason the leg and the chest are sharing the same object/stack when they shouldn't be!! Another thing to note is that the "dummy id" also shows values of 0-3 with this model too which complicates things.

Here's a print out of the bones commands...
Code: [Select]
============================================================================================================
00 | 0x26 | [Connect to Object] Object ID: 00 | Parent ID: 00 | Dummy ID: 02 | Stack ID: 00 [Restore ID: -1]
01 | 0x26 | [Connect to Object] Object ID: 01 | Parent ID: 00 | Dummy ID: 03 | Stack ID: 01 [Restore ID: -1]
02 | 0x26 | [Connect to Object] Object ID: 02 | Parent ID: 01 | Dummy ID: 03 | Stack ID: 02 [Restore ID: -1]
03 | 0x26 | [Connect to Object] Object ID: 03 | Parent ID: 02 | Dummy ID: 01 | Stack ID: 03 [Restore ID: -1]
04 | 0x66 | [Connect to Object] Object ID: 04 | Parent ID: 01 | Dummy ID: 03 | Stack ID: 04 | Restore ID: 01
05 | 0x26 | [Connect to Object] Object ID: 05 | Parent ID: 04 | Dummy ID: 01 | Stack ID: 05 [Restore ID: -1]
06 | 0x66 | [Connect to Object] Object ID: 06 | Parent ID: 01 | Dummy ID: 01 | Stack ID: 06 | Restore ID: 01
07 | 0x66 | [Connect to Object] Object ID: 07 | Parent ID: 00 | Dummy ID: 03 | Stack ID: 07 | Restore ID: 00
08 | 0x26 | [Connect to Object] Object ID: 08 | Parent ID: 07 | Dummy ID: 03 | Stack ID: 00 [Restore ID: -1]
09 | 0x26 | [Connect to Object] Object ID: 09 | Parent ID: 08 | Dummy ID: 01 | Stack ID: 08 [Restore ID: -1]
10 | 0x66 | [Connect to Object] Object ID: 10 | Parent ID: 07 | Dummy ID: 03 | Stack ID: 09 | Restore ID: 07
============================================================================================================
11 | 0x02 | [Object ID: 0] Visibility: 1
============================================================================================================
12 | 0x03 | [Set First ID: 1]
============================================================================================================
13 | 0x04 | [Material Pairing]  | Material ID: 01 | Check: 5 | Polygon ID: 01 [First ID: 1]
============================================================================================================
14 | 0x66 | [Connect to Object] Object ID: 11 | Parent ID: 10 | Dummy ID: 01 | Stack ID: 01 | Restore ID: 09
============================================================================================================
15 | 0x03 | [Set First ID: 3]
============================================================================================================
16 | 0x04 | [Material Pairing]  | Material ID: 00 | Check: 5 | Polygon ID: 00 [First ID: 3]
============================================================================================================
17 | 0x01 | [End of Bones Section]
18 | 0x00 | [Padding]
19 | 0x00 | [Padding]

The data in brackets on the right hand side mean that the command itself doesn't specify the data, and it relies on whatever values are loaded into the respective variables.

Also note that if you test out this model yourself, you'll get a different result as I've been tweaking the model viewer since the demo release...

[edit]

Ok I think I understand what the problem is...The reason why the phantom's chest and leg are breaking is because they ARE using the same stack. I've been treating the matrix stack as multiple 4x4 matrices, when it's really more like a buffer. I was looking at some of the Debugging tools in DeSmuME and I noticed that there were like 32 matrices. What I've been doing (what Kiwi.DS's code did) was parse all the stacks before rendering the models, thus any stack used by several objects would only retain the last stack loaded into it, and this is why the phantom's chest and right leg breaks when swapping between object id and stack id.
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 29, 2010, 12:12:31 pm
(http://i5.photobucket.com/albums/y154/Diadora/brokenslimes.png)
A new species of the Slime!

EDIT:  Whoa, this thing can get to be a real power hog.  Both of my processors are being used to the max :O
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 29, 2010, 10:32:34 pm
Yeah, that's probably because it's updating constantly...For whatever reason a simple "render once and not unless it changes" hasn't been my friend (it displays it once then is blank). I'll sort out that problem eventually, for now...it's a beast and that's just how it is :p My laptop is fairly decent though, so i don't really notice it, however I've noticed it also creates a lag with windows explorer.

Are you using it on Linux or something?

[edit]

Looky looky :laugh:

(http://img72.imageshack.us/img72/5065/pkmntwinleaf.png)

Now this is just a quick implementation I did partly because I wanted to and partly because loading multiple models (and a map), on the scene is something I'll have to implement eventually anyways. As you can see I am yet to figure out what determines transparency with textures (aka paletted images will always have color 0 as transparent) which is why some of the ground is see-through, but it's a WIP.

I was also looking at using NSBMD courses as a secondary model for implementing 3d maps, has anyone used the NSMB Editor (http://code.google.com/p/nsmb-editor/) before? It just crashes for me when I try to load the NSMB rom :banghead:

[edit 2]

Okay I think I've mostly figured out NSBTP files...

 - Each NSBTP can store multiple texture/palette swapping animations
  - Each animation contains all materials that use this type of animation
   - Each material contains a number of frames which makes up the animation for that material
    - Each frame contains the first frame number, texture id and palette id

There's still some data I'm not too sure about, which mostly likely controls how the animation plays back, as this type of animation can also be used for stills like a Thwomp (in NSMB) that changes his facial expression depending on how close mario is to it.
Also an annoying fact about NDS models is that all textures/palettes are referenced by their actual name! Though I suppose it makes some sense in a way :-\
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 30, 2010, 12:08:44 pm
Quote
Are you using it on Linux or something?
I'm on XP (the thing that never dies), I just have a front-end GUI running in place of the standard look.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 31, 2010, 11:22:49 am
NSBTP files have only 2 bytes per animation that I haven't figured out now. I can successfully open them up with loaded texture/material info and can manually swap frames no problem, but it took a really ugly quick gui do up and it's just a mess -_- So I'm adjusting the GUI a bit so I can add all the different animations into it without clogging up the workspace :p

I've added support for BRRES archives which contain a collection of 3D data files that make up a character or scene. Should be most well known by those who hack Super Smash Bros Brawl, but the format is also found in New Super Mario Bros Wii (though apparently different somewhat). The reason why I'm looking at some of the wii stuff too is because I've noticed that some of the magic stamps for formats are very similar to some of the NDS/GC ones I've been looking at, and there's a better chance I'll figure those ones out if there's documentation I can reference to since Nintendo doesn't seem to change their formats very much between consoles.

Also for those who are interested in Super Smash Bros Brawl models check out this project (http://www.smashboards.com/showthread.php?t=238861) I came across. It is seriously awesome and it's given me some ideas to work into my own model viewer.

[edit]

Still in the process of adding more support for gamecube models. The entire mesh is drawn now, though I've had to modify the implementation a little to make it work, so I'll have to make those changes to my NDS code too...

(http://llref.emutalk.net/images/projects/console_tools/sms_airport.png)

Also now because I can load full maps such as ones from Super Mario Sunshine, there was a clear lag and CPU overkill, so I looked around and set it up so it won't update the screen unless something changes. What was most sad is that I discovered a thread which was updating the values of the translation slider guis was using a whopping 50% of my CPU...so I killed that too, and now everything is much faster and isn't constantly putting a strain on the processor. At worse large models take up to about 60% of the CPU when refreshing going by my laptop (before it was actually using like 80% plus *constantly* so its quite significant). I'm still a newbie to threads and such so I'm bound to make more dumb mistakes in the near future :p

Kryal, the author of that SSBB tool is letting me use his documentation on the Brawl models formats, so I'll add them later once I get past the current hurtles I'm facing right now.

[edit]

For those of you who are geared toward PSX hacking. Can you send me some valid TMD models and/or other model formats with documentation preferably. Of all the games I have, only Digimon World provided me with some valid TMDs, but I would like a few more from other games to make sure I have good compatibility. Thanks :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on February 18, 2010, 09:28:06 pm
Over the last couple of weeks I've been working at basically overhauling my whole project. I did this because it was getting really messy (with code) and I was finding that the GUI format just wasn't working all that well with every new feature added (such as the multiple Model Animations). Another major reason was my lack of understanding on how classes work, as I now think I am better utilizing them than I was before. The code behind the GUI also now (or wells I hope so) follows the standard as mentioned by Nightcrawler. I've also found that some of the built in UIs visually break (such as the Vista one I use), on things such as InternalFrames, so I'm making my own custom skinnable ones using the original classes as a basis.

Major changes, is that the tab system will be restricted to the browser and archive files and everything else will use a Workspace Layout like photoshop (where each file is a separate internalframe, and there are toolbars etc). I've also tried to simulate some of the workspace aspects of photoshop since I think they work really well for this kind of thing. The basic functions are done, I just need to start adding the appropriate toolbars/panels for each type of file and the functionality that goes with it :)

Atm I'm updating my file loader code to use classes (instead of nasty objects), and add preview/thumbnails to the browser. I start my Uni studies back up in about a week and I'll probably spend less time on this from then on...But hopefully this project will look better and better once I have everything up and running :)

(http://llref.emutalk.net/images/projects/console_tools/feb19_update.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Rocket Science on April 05, 2010, 07:14:03 pm
Can't wait to see what the final version will be like  :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 18, 2010, 03:16:37 am
Finally after x amount of months I have a few days free where I'm awake enough and not being hassled with all the assessments I have to do at Uni.

Of what I've managed to do over this weekend, I think my new interface is coming along nicely. Most of the work has been improving my file management classes as well as customized GUIs, but as I've had very little time to work on it, I only have a Palette viewer working, which is still very basic in terms of stuff you can do with it.

(http://llref.emutalk.net/images/projects/console_tools/workpace_1.png)

I'm using the Photoshop interface as a base which I'll customize later to look unique to my app. I'm trying to make sure I keep a clear separation between the interface and the data management, because in one of my classes we were taught so with Web Development, and its honestly a good thing to be doing because it makes it easier to go back and change things without having to locate stuff all over again which I think has been one of my ongoing problems. All data manipulation will be handled by the individual Plugin classes themselves which implement abstract classes for each unique type of file (ie palette, image, sound, etc), that way the interface only need to call something like getColors() and it will parse the file into say an array of 32bit colors etc. The tricky part has mostly been getting the GUIs to display properly which is what burns most of my time. However pretty much once I've made the custom object classes, the latter GUIs will take much less time to make.

Unfortunately starting tomorrow I'll be really busy again with studies so I might not make much more progress on this for a little while yet, but I've actually been using a much older and more stable version to rip stuff from games, which has resulted in how I chose to spend this weekend. I might try and make it more of a habit :)

[edit]

Oh and I forgot to mention that I've also figured out how to add drag and drop functionality to my application so you can drag any file from your desktop (or where ever) and it will try to load it :)
Title: Re: The Console Tool (by Low Lines)
Post by: drowssap14 on June 07, 2010, 05:07:07 am
Will it ever be possible to edit the model and collision files from SM64DS or any of the other DS games? Because it would be good to be able to create custom levels, edit characters etc.

Project looking really good so far.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on June 07, 2010, 09:25:03 am
Wells at the moment I've been trying to get the GUIs to work right. Having some trouble putting a 3d render panel inside an internal frame, which just seems to vanish when I refresh it. But once I can get the GUIs working I can start adding functionality.

If you've ever used 3D editors before, you should know they are quite complex to use, and most likely even more complex to code. I think basic mesh editing is a nice goal, however more complex stuff would have to be done in one of the proper programs which are built for it. So you would have to export models and edit them and then import them back in.

My studies only finished last week, so I've now got a month off. Don't expect any like overnight updates or anything though, still have lots to do...But with luck I can get some of the annoying GUI problems sorted and get stuck into working on formats again...

[edit]

Figured out what the problem was. Was trying to put a heavyweight component into an internal frame which is a lightweight. It kept disappearing because heavyweights ignore parent components.

[edit2]

(http://llref.emutalk.net/images/projects/console_tools/workspace_2.png)

Right...now here's a screenshot of where the current interface is at. I totally rewrote my InternalFrame code, making it much cleaner and it also does everything I want it to now! At the moment I am working on building the Toolbar interface which will allow multiple docking on all sides of the workspace as well as floating toolbars. As you can see that's mostly implemented already at least for the layout part as I might leave drag and drop functionality to later since I'm much more keen to get back to adding file format support. I still have to setup a method that will allow actions to be performed when buttons are clicked however I've already got it figured out in my mind so that shouldn't take too long. Once I can start adding actions, it should start to get interesting.

At this point I have several format type frames working including the palette viewer, graphic viewer, model viewer and an initial start on my hex viewer. The model viewer uses a lightweight component as opposed to the heavyweight one that was used in the old implementation, so its works a lot faster and I don't have to do much in terms screen updating since it does it all by itself (mostly), making it less cpu hungry. I haven't updated my NDS and GAMECUBE model codes to my new file handling structure, but I did up basic PLAYSTATION tmd models and they display fine :) I've also started making my own GUI icons (see the bottom of the screenshot), since I can't find any they show what I want these buttons to do.

Will post more once I get actions up and working.
Title: Re: The Console Tool (by Low Lines)
Post by: justin3009 on June 13, 2010, 12:06:52 pm
This is just exceptional work.  Fantastic job on this, absolutely loving it.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on June 17, 2010, 09:41:47 pm
GUI Actions seem to be working alright now, they can be called from pretty much anywhere and don't require much work to change etc. Non-generic toolbars such as the one in the screenshot below affect each separate opened file, so if you say want on Model file to display wireframe, it will only set wireframe for the currently selected window. Also toolbars that aren't used by the window type are hidden every time you switch windows, so it should help keep stuff neat and tidy :)
I've fixed a bunch of bugs in the file browser, which actually was loading a folder several times every time it was refreshed -_-
The archive window is now fully working, and loads inner archives really really fast since all the file scanning is only done once each time the source file if opened.

(http://llref.emutalk.net/images/projects/console_tools/workspace_3.png)

My main focus at the moment is coming up with a decent model structure class that I can use across as many different formats as possible like I've been doing with image formats. The trouble is there is quite a bit of variation in how people like to store their 3d models and stuff, which means I have to study them more to get a better understanding :(
So I've been looking at Autodesk's 3DS format, which I have a Zelda TP pack for to experiment on. With that, basic support for TGA images has been added too :)

The last thing I've been doing was making a reboot on my documentation. Since I did a Web class this semester at uni, I've gotten a bit more wise with HTML & CSS, so I've used that to build an interface for my docs. It works like a Directory tree with inner nodes able to be expanded and hidden which is much easier to navigate than with crappy tables a mile long. The docs themselves are kept in XML format and loaded in via Javascript though only because I don't need to upload the files to view them, will make a PHP version once I decide to do so. It makes use of a bunch of neat CSS3 properties and displays alright on Safari and Firefox (sorry IE is a loser). Much easier to maintain and I think looks cool :)

(http://llref.emutalk.net/images/projects/console_tools/doc_format.png)

[edit]

I've come up with a fairly generic Model Data Structure which is based on how gamecube models are packed. Basically in terms for NDS models, the polygon data is broken into four levels...
 - polygon
  - shape (ie quad, quadstrip, triangles, etc)
   - vertices (just a group name for the attributes below)
    - attributes (ie position, normals, scale, restoreId, color, etc)
Not all vertices will have every type of attribute, which can be easily checked upon rendering using struct styled classes (this is java after all).

(http://llref.emutalk.net/images/projects/console_tools/pakkun_plant.png)

Each time I reimplement my NDS Model code, it always seems to be a little different from the last time >_<
The above was one particular model that really hated me in all my older implementations, and now it just works...A lot of other funky models now also display properly too however some of the ones I had gotten perfect now have some spastic vertices.

[edit 2]

Found the problem. VTX_DIFF wasn't being added to the vertex list.

[edit 3]

(http://llref.emutalk.net/images/projects/console_tools/model_gui_new.png)

I am now able to get nearly every single NSMB character model displaying properly now! Models with Joint Scaling and Rotation, or funky stack IDs are the only ones that don't display properly, which covers pretty much all Zelda game models -_-

Gamecube models have also been updated to the new format structure, however they still require a lot of work.

The Model GUI has gone through a bunch of improvements, including basic lighting (or at least what is supposed to be lighting) and Perspective Transformations using the mouse buttons (ROTATE, MOVE, SCALE). Toggles for most of the interface have been added too.

A preliminary Key Listener has been added as well to allow for shortcuts etc in the future :)

NDS Roms are back and files can be exported from archives.
Fixed a bug that caused multiple reads per file while checking for compression, folders load quicker now.

I still need to build some properties windows to display stuff and create a resource manager so files can be linked with external files, such as external textures. Currently it will load the last texture selected by default if no texture is found within the model file.

[edit 4]

(http://llref.emutalk.net/images/projects/console_tools/model_joints.png)

While trying to figure out joint rotation in NDS models, I decided to have a go at drawing the joints on screen to help with debugging. I was kind of surprised at how easy it was to do and so now the model viewer is one step closer to total awesomeness :)

The joint rotation data is made up of 8 signed decimal values, which I figure sit in top right hand corner of the 4x4 matrix stack. Looking at "bomb_menbo", I've managed to get his left legs looking kinda right, but I'll have to do more trial and error to figure out how exactly these values are stored...

Although I haven't looked at them yet, the joint scaling is made up of 6 signed decimal values, however I have seen some impossible data (ie really really big values) pop up in these slots so I'm not completely sure on them yet...

[edit 5]

(http://llref.emutalk.net/images/projects/console_tools/model_rotate.png)

I figured it out!!! Each joint had an used word which turns out to be a type of scaling factor. I only figured this out because I came across a model in NSBMD where the bird's wings had been obviously flipped in the model editor and hadn't had their transformations reset (for those not familiar with modelling...sorry). Each wing had an inverse scale factor so when I inputted that into the Rotation matrix...It displays properly :)

So in simple terms the 8 rotation values + the scaling factor are put into a 4x4 matrix like so...Note that for whatever reason when people represent a 4x4 matrix it is shown as top-bottom, left-right.

Code: [Select]
s = scale factor (signed)
r = rotation[8]

[    s, r[2], r[5], 0]
[ r[0], r[3], r[6], 0]
[ r[1], r[4], r[7], 0]
[    0,   0,    0,  1]

I'll assume that this scaling factor affects all joint data, hence why alot of models aren't sitting in their bounding boxes. Going to test that assumption out now...

[edit 6]

(http://llref.emutalk.net/images/projects/console_tools/ben10.png)

Since figuring out how joint rotation data is stored, a very large number of models now display properly. The above is a Ben 10 model, which I've also pasted what comes up in KiwiDS's NSBMD model viewer so you can see the difference in accuracy :)

I haven't figured out how the joint scaling data is used. It's made up of 6 values, which is twice as many as any typical scaling (aka one for each axis)...however it seems to have little visual impact on how accurate the models look so I'm not that worried.

Pretty much all issues remaining lie in the bones data...There's still a few commands that aren't properly implemented if at all, plus there's still the issue with objects overwriting used stack locations due to wacky stack IDs...The funny thing is that KiwiDS's model viewer doesn't seem to face this problem and yet I've gone over his source code and cannot see how he implemented it so I'm guessing I have an older version of the code or something :/

[edit 7]

(http://llref.emutalk.net/images/projects/console_tools/battlechaser.png)

Lol this is turning into a long post...Anyways figured out how to get models with wacky stack ids to display right. You keep a copy of each joint matrix in a separate stack and when each polygon is loaded you "restore" that joint matrix to the main stack, thus allowing multiple joints to use the same area in the stack...It was kind of obvious now that I think about it, however the fact was that all restore calls from vertices reference to the main stack and not the joints themselves. Zelda models now display well too now, though ironically Link is among the few that DON'T display properly yet -_-
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on June 27, 2010, 02:51:01 pm
7 edits is enough for one post I think :p

(http://llref.emutalk.net/images/projects/console_tools/hm_dog.png)

Most of the wacky models have a very unordered bones list, often jumping between joints and polygons and all what not. So its kind of obvious that the ordering needs to be retained for rendering. With that pretty much every model displays with correct geometry now, except a few with unknown bones commands such as 0x9. The "lamb.nsbmd" model from Harvest Moon: Sunshine Islands is one example that has like a dozen 0x9 commands in its bone list.

Most models from the Zelda games work too now :)

(http://llref.emutalk.net/images/projects/console_tools/link.png)

I still have to figure out what's contained within material data, along with figuring out how to get alpha blending working right. See with the dog's cheeks, if you turn on transparency, you'll see they have a gradual softened edge, however it will totally disregard polygons behind it and only display the background and the grid in the transparent area. Its more than likely that each material has their own alpha blending options.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on July 01, 2010, 02:15:22 am
This tool is really fascinating. I have had trouble using it, though. Like, how is the model rotated? The actions are slow for me.

But I would really like it if the nsbmd files contained in the ZIP download could show up properly. It's coming really close but a huge texture gets in the way most of the time.

Here are the files.
http://www.mediafire.com/?uvk1e2oy2wq

This is what it's looking like for me.
http://i55.photobucket.com/albums/g144/mega_rock_exe/lostbn5netmapct.png
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 01, 2010, 02:51:55 am
First of all that demo link in my signature is over 6 months old. The build I'm working on at the moment is like completely re-written and way better than that shitty demo I put up. You should be able to tell the interface in my newer screenshots looks totally different to the one you are using.

Interestingly though those models you linked me (which you should remove due to copyright) have some scaling issues when I view them, with some objects and platforms being drawn outside the map (assuming its not supposed to do that of course).

(http://llref.emutalk.net/images/projects/console_tools/lost27.png)

My uni studies start back up in 3 weeks so I'm considering posting another demo so people can test out the new interface and give me feedback on the model viewer, however there are still a few things I want done before then.

As can be seen in the above screeny, I've been slowly working on interfaces that will allow for more manipulation options and hopefully a start on actual EDITING of model stuff. The upper window is basically your Selector and Show/Hide Toggler...Selecting stuff makes wireframe and vertex points and joints change color, and hiding stuff makes them very transparent. In future this interface would also be done using mouse input on screen however I'm not too worried about doing that as of yet.

The second window is my start on an Attribute Editor. It's based on the Netbeans GUI property interface which I thought was a really nice clean way to display information, and is very easy to manually edit values via keyboard input, since it uses the JTable interface (something I didn't know existed until late last night -_-)

Aside from a handful of bug fixes, the last thing I want to have in place is an interface for animations, which has been the main thing holding me back from making a start on adding support for animations. Since figuring out the rotation data for joints I've got a much better understanding of the NSBCA format, I just need a way to load them into the model viewer so I can start debugging.

I'll probably have to start writing a User Manual with in all the controls, since there's starting to be quite a lot of them :/

[edit]

(http://llref.emutalk.net/images/projects/console_tools/megaman.png)

Here's a screenshot of the character model from those files (466.bmd) showing selection/show/hide in action :)
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on July 01, 2010, 05:34:08 am
Just wanted to stop in and say, this is still quite fucking awesome :)
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on July 01, 2010, 12:24:07 pm
Well this is really nice then. I would have liked to try this out myself but the demo link is all there is.
I'm assuming those blue lines mean all the buildings overlap each other. It probably shouldn't do that but the map might be incomplete. None of these should really exist so I wouldn't be surprised if anything is missing from them. Just out of curiosity, can I see what the 7-11 building looks like?
The character model looks great. Just like I expected it to look. I'll be waiting for the animation support; that is, if these models even have any animations.
Title: Re: The Console Tool (by Low Lines)
Post by: RetroHelix on July 01, 2010, 01:18:23 pm
Awesome work!

Over the time I lost track of what this tool  does and what not. Could you make a featurelist or something? :laugh:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 01, 2010, 02:43:21 pm
...Just out of curiosity, can I see what the 7-11 building looks like?

I'm not sure which one you are talking about...The first model is that green one you took a screenshot of, then there's that suburb map I was looking at, then some kind of in house map, and the last 3 models were pieces of the character including his suit. Not really sure where the 7-11 thing comes in?

The blue lines are joints, each joint is connected to a parent etc, think of it like a really simple skeleton. Yes it is kind of weird how maps can have joints too, but that's what they are, and you can easily hide/show joint display if you don't like it.

...Could you make a featurelist or something? :laugh:

The overall aim of this project is to create a tool that does pretty anything you could want to do in terms of hacking/modding or hopefully even creating stuff for games. I've been mostly focusing on the model viewer as of late however, and specifically the NDS formats.

First off, its made up of two parts, you have the file browser, and the workspace which will house all file manipulation and stuff. Any file can be dragged and dropped from your desktop (or wherever) into the Workspace tab and will be loaded automatically. Also the browser features live previewing of displayable formats (such as images), which is really handy when you have those huge folders with numbered files such as in pokemon.

In terms of archiving, NDS Roms, NARC, BRESS/ARC (Wii & Brawl), RARC and GC Iso are currently supported, with LZ77, YaZ0 and RLUncomp supported though still a little buggy in some cases.
Compressed and/or files contained in archives can be exported, though at the moment it will always export to the same folder as the source. You can also export folders, however I think there's a slight bug, which I will have to fix.

I haven't had the chance to work on all the NDS sprite data formats yet, however that will be next on my list once I've done enough on the model viewer.
Textures open up in their own interface, though all you can do is toggle the transparency and adjust the scale at the moment.

Lastly there's the model viewer which has been my main focus lately which has very strong support for NDS models, and basic support for Gamecube and Playstation formats.

Eventually I want to have interfaces for each and every type of file with heaps of options and stuff you can do however that takes time and I also have other commitments such as my uni studies that limit how much time I can put into it.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on July 01, 2010, 02:48:27 pm
...Just out of curiosity, can I see what the 7-11 building looks like?

I'm not sure which one you are talking about...The first model is that green one you took a screenshot of, then there's that suburb map I was looking at, then some kind of in house map, and the last 3 models were pieces of the character including his suit. Not really sure where the 7-11 thing comes in?

The blue lines are joints, each joint is connected to a parent etc, think of it like a really simple skeleton. Yes it is kind of weird how maps can have joints too, but that's what they are, and you can easily hide/show joint display if you don't like it.
It should be on the suburban map. For some reason, there are textures for the Honda and 7-11 building. I can see the Honda building is on some sort of container floating on the top left of the screenshot. the 7-11 must be on that map somewhere for some reason.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 01, 2010, 03:40:09 pm
7-11? It's a supermarket? I live in Australia so I've never heard of it :p Geez I was looking for some kind of twin towers or something XD

(http://img87.imageshack.us/img87/815/711d.png)

[edit]

Wells I just found out they do have them in Australia...just not where I live :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on July 01, 2010, 04:25:39 pm
Very interesting.
Thanks Low Lines! ;)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 02, 2010, 08:07:30 pm
Wells here's the first bit of actual model editing! Albeit VERY basic and limited to manual input, at least its something. The only problem is that I don't know how to pull stuff from a 4x4 matrix (like a rotate X value), and several bits of data in NDS model joints are stored as 4x4 matrices :(

(http://img87.imageshack.us/img87/51/modeledit.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Friedslick6 on July 04, 2010, 05:31:29 pm
Oh my :cookie:, this looks so good. Like, Tile Molester only molests tiles. But this... molests everything. That doesn't sound good, but it certainly is.
I've already managed to rip DS sprites from Mario Kart DS using the heavy version (mind you, using print-screen. Will there be an export/convert feature?)
Anyways, you're a genius. Keep on geniusing, please.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 04, 2010, 08:15:06 pm
I've already managed to rip DS sprites from Mario Kart DS using the heavy version (mind you, using print-screen. Will there be an export/convert feature?)

XD you poor soul.

I'm still trying to decide how I want to display/edit certain file types.

Take Model Animations...first off there are at least three kinds, you have geometry animations that affect how a model is drawn on screen, material animations that I think manipulate how textures are skinned onto models, and texture animations which swap between multiple textures such as blinking eyes etc. Now each of these have fairly different data structures, and they all can be animating at once, and by themselves there is little to no other way of displaying that data other than in a table/list which isn't very fun to edit!! What I think I could do instead, is have a resource manager in the workspace that holds all assisting file types such as animations, palettes, mapping etc, and when you load such files into the workspace, it adds them to the resource list and can be accessed from all windows currently opened. So if you want to edit a palette, you'd be editing it in an attribute panel as opposed to its own window. It's going to take a little reworking to do this I think, but it makes more sense to do it this way rather than have X number of windows open to achieve one outcome such as a map or a sprite sheet.

I've been studying the NSBCA format heavily over the last few days trying to figure it out. I've determined that the first offset in a joint animation contains Pivoting data, and the second section contains Rotation data. Rotation keyframes are called when the second byte equals 0 and Pivot keyframes are called when it equals 128, however the data itself is definitely stored a little differently to how it is stored in NSBMD files. Scaling keyframes hold two scaling values for whatever reason, with Translation keyframes being just straight values that can be a signed word/dword.
Each animation has a frame length, however each object in an animation seems to have a start and end position that typically don't match up with the total number of frames. Now I haven't had much experience with model animations, but I'm guessing this might have like a decay effect on the animation if anyone has ever played around with animations in Maya? The trickiest part is calculating the right number of keyframes stored in each object as there isn't an actual value written down anywhere. From what I can tell it's calculated based on a bunch of things including the difference in object frame length over animation frame length, the rate in which the object (and possibly the animation) plays back as well as rounding to the upper whole number. This looks a bit funky and overcomplicated but so far all the animations I've been working with calculate the correct number of keyframes, so I'm assuming I'm on the right track. Am yet to get to the point of loading animations into a model, but hopefully I won't get some spastic result :S

If anyone's interested, here's my notes which I've been keeping while studying the format..So far I've only worked on small files as it takes a while to highlight all the data of complex animations in a hex editor. These are all animations found in NSMB. Sorry if my notes aren't very easy to understand :(

Code: [Select]
Object Flag: --zyx-Sr-RZYX-T-
> found in the header of  each object of an animation
===========================
T - has Translation keyframes (0 Yes| 1 No)
XYZ - flags for Translation attributes
R - has Rotation/Pivot keyframes (0 Yes| 1 No)
r - flag for Rotation/Pivot attribute
S - has Scale keyframes (0 Yes| 1 No)
xyz - flags for Scale attributes
===========================
if T-XYZ = 1
> Fixed Translation value (signed dword)
if R-r = 1
> Fixed Rotation/Pivot value (dword/2*word?)
if S-xyz = 1
> Fixed Scale value (2*dword)

Note: The below is only done when the bit flag equals 0 for that attribute (TX, TY, TZ, R/P, SX, SY, SZ)

a|b = datasize = playback speed
> a & b are flags stored in the object header for each attribute

Translate
-----------------------------------------------------------
2|0 = word = 1/1
2|1 = word = 1/2
2|2 = word = 1/3
0|1 = dword = 1/2

Rotate
-----------------------------------------------------------
0|0 = 2*byte = 1/1
0|1 = 2*byte = 1/2
0|2 = 2*byte = 1/3
> byte0 = index
> byte1 = 0 Rotation | 128 Pivot
> not completely sure on rotation yet

Scale
-----------------------------------------------------------
2|1 = 2*word = 1/1
2|1 = 2*word = 1/2
-----------------------------------------------------------

Attribute    [animation flag|start|end|a|b] bytes/keyframes - animation length
> bytes - the actual size of the data stored
> keyframes - bytes/datasize(see above)

Basabasa - 0 Pivot - 33 Rotation
===========================================================
Translate [3|0|34|2|1] 38/19 - 36 Frames
Rotate [3|0|34|0|1] 38/19 - 36 Frames
Scale [3|0|34|2|1] 76/19 - 36 Frames
===========================================================

Basabasa2 - 1 Pivot - 20 Rotation
===========================================================
Rotate [3|0|78|0|1] 82/41 - 80 Frames
===========================================================

Bilikyu - 31 Pivot
===========================================================
Translate [3|0|58|2|1] 62/31 - 60 Frames
Rotate [3|0|58|0|1] 62/31 - 60 Frames
===========================================================

Donketu - 12 Pivot / 20 Rotation
===========================================================
Translate [3|0|10|0|1] 24/06 - 11 Frames (dword)
Translate [3|0|10|2|1] 12/06 - 11 Frames
Translate [1|0|02|2|0] 04/02 - 02 Frames
Rotate [3|0|10|0|1] 14/07 - 11 Frames
===========================================================

Gesso - 39 Pivot
===========================================================
Translate [0|0|16|2|2] 14/07 - 19 Frames
Translate [0|0|16|2|2] 12/06 - 18 Frames
Translate [0|0|12|2|2] 10/05 - 14 Frames
Scale [0|0|16|2|2] 28/07 - 19 Frames
Scale [0|0|16|2|2] 24/06 - 18 Frames
Rotate [0|0|16|0|2] 16/08 - 19 Frames
Rotate [0|0|16|0|2] 16/08 - 18 Frames
Rotate [0|0|12|0|2] 14/07 - 14 Frames
===========================================================

TTL Bird
===========================================================
Translate [1|0|05|2|0] 10/05 - 05 Frames
Rotate [1|0|05|0|0] 10/05 - 05 Frames
===========================================================

Gamaguchi - 11 Pivot - 52 Rotation
===========================================================
Translate [3|0|08|2|0] 16/08 - 08 Frames
Translate [3|0|07|2|0] 14/07 - 07 Frames
Translate [3|0|16|2|0] 32/16 - 16 Frames
Rotate [3|0|08|0|0] 16/08 - 08 Frames
Rotate [1|0|07|0|0] 14/07 - 07 Frames
Rotate [3|0|16|0|0] 32/16 - 16 Frames
Scale [3|0|08|2|0] 32/08 - 08 Frames
Scale [1|0|07|2|0] 28/07 - 07 Frames
===========================================================
Title: Re: The Console Tool (by Low Lines)
Post by: drowssap14 on July 06, 2010, 05:43:40 am
Model editor/viwer looks good.
Will it support .kcl collision data - I think mariokart ds and sm64ds use them?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 06, 2010, 02:52:03 pm
Model editor/viwer looks good.
Will it support .kcl collision data - I think mariokart ds and sm64ds use them?

Considering I've only gotten models displaying and haven't even looked at collision stuff, not to mention the fact that Super Mario 64 DS models use a much older version of NSBMD that's only partially supported...That'd be quite a feat anytime soon :(
Interestingly though you got me looking at Mario Kart DS which in turn led me to a few discoveries. For starters, there was a bug in my NARC reader that didn't take into account that subfolders CAN be empty (it sounds like a stupid error, but due to the way NARCs are read, its an easy mistake to make).
Another thing I discovered in another game I was checking (Goldeneye - Rogue Agent), it turns out NSBMDs can sometimes not contain joints and material data...whether that means that the bones don't rely on them or that what I was actually looking at was a partial file that loads on top of a common NSBMD with the necessary data, I'm not totally certain, but it definitely puts a spanner in the works so to speak -_- If it is the latter, then there's all the more reason to follow through with my Resource Manager idea so that you can actually view these "special" cases...

[edit]

It seems NSBMD files can ALSO not contain polygons, or rather polygon data seems to be under something else :/ There is also definitely no obvious common file for Goldeneye models, so I am guessing that these files must use some unknown data blocks that haven't been documented yet by anyone.

[edit 2]

The model header is 12 bytes shorter in Goldeneye Rogue Agent NSBMD files, therefore it clearly used an earlier version of the NSBMD format. All the necessary sections are in the file, its just that the Joint section was indexed incorrectly because it starts directly after the model header.

(http://img87.imageshack.us/img87/3819/goldeneye.png)

[edit 3]

Heh...while mucking around with the model header I realised I had totally overlooked a few bytes. It turns out the proper model scaling was stored in them, so now those Megaman/Rockman maps display properly now as well as a bunch of other models (particularly with maps). Still haven't got the bounding box right though...

(http://img72.imageshack.us/img72/4691/townfixed.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on July 07, 2010, 12:21:45 am
Wow! That's awesome stuff there. Great work! The map is resembling the 2D version...
There should be a boat but I can't see it. Also, it seems like something is wrong with the building behind the pink building. There should be a texture with words above the door.

One more thing, if you're looking into the NSBCA motions format, I believe some of the character files I sent you may have animations associated with them. It could be an older format though.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 08, 2010, 03:09:47 pm
Typically moving objects such as boats are loaded ontop the map in the same manner in which 2D sprites are loaded into 2D maps. Unless this "sign" on the pink building is in the textures, it's probably an object loaded onto the map as well. I haven't added the ability to access internal textures in models yet so I can't check that.

Have figured out NSBVA (VIS0) files now. They store a bit array of joint visibility for each frame of an animation, it allows you to toggle parts on and off though it is clearly under used in most games. There are only 3 models in NSMB that use them, being Koopa Jr, the Spiny Boss and the normal walking Koopa. Koopa Jr for example often has instances where his apron and bandit mask are on or off, whereas normal Koopas toggle their inner body and shell so both a shelled koopa and a shell-less koopa use the same model. There's still a few bugs with properly toggling said joints however as other joints often use the same stack causing them to disappear as well.

I've chosen to separate the File Browser from the main window making it a popup window as I found it quite annoying having to change tabs back and forth between it and the workspace constantly. I've also decided to put floating toolbars/menus into separate top-level windows so they aren't limited to the dimensions of the workspace, and by default the first opened file will be loaded maximized rather than little tiny windows. Am mostly doing an overall clean up of the GUI code before I start building more menus such as the resource manager....
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on August 23, 2010, 11:49:52 am
This is very cool. I am looking forward to this. But I have to report a bug from the version I downloaded yesterday. When trying to open the sdat from Apollo Justice, I get an error tree view instead. The error is:
Code: [Select]
Error reading contents:
java.lang.ArrayIndexOutOfBoundsException: 6

And I have to say that the pac files used by the same game are looking rather interesting. A quick glance at them tells me it likely begins with offsets to sub files. And the stored data looks compressed to me.

At least I know that the .vx files are video. Maybe something to put on the low low priority wish list?

Anyway, I will be keeping an eye on this, it's interesting for sure.
Title: Re: The Console Tool (by Low Lines)
Post by: DaMarsMan on August 23, 2010, 11:51:09 am
Just stopping in to say I'm very impressed with the ground breaking work you are committing to.   :cookie:
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on August 25, 2010, 10:49:51 am
I love this tool. It's the first thing I've found that displays models from Kingdom Hearts 358/2 with accuracy. I was wondering though on your project page of your site it mentions an export feature yet I can't seem to find it. not even for textures which it seems would be easiest. I would really appreciate it if you could tell me where to find it and if it exists.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 25, 2010, 06:24:17 pm
It falls under plans and future implements. My site hasn't been updated in yonks...in fact when that site gets updated (as in total overhaul and redesign to new web standards) that's kind of when I can start working on this project again. I've also been doing a Java unit this semester and there's already a bunch of new things I didn't know about that you can do with it too in including shortcuts and better use of inheritance. I have a feeling what I will have learned by the end of my unit is going to greatly improve my programming skills, so I'm like quite stuck into it right now. It's nearly mid-way through my semester with one more week next week then I get a week off, so hopefully I can spend a bit of time on hobbies again *yay*

Also I really appreciate the support I keep getting off and on as time goes on, even after x-number of weeks/months with no updates someone suddenly digs up this thread and says "hey we're still here!!" lol It's probably half the reason why I have stuck to this project for so long and intend to actually go somewhere with it eventually.

Also I'm pretty damn sure I promised I'd have a new demo up for people to play around with before I started my studies...It's a little late, but here you do :p

Console Tool Demo #2 (http://llref.emutalk.net/downloads/console_tool_demo2.zip)

This is the last working build I have on my computer so it shouldn't crash much :p It will look like it can open a bunch of files, but it's mostly only able to handle a handful of archive types and model files. The real point of this demo is to demonstrate the new interface. I've tested it on both a Vista and an XP and it runs fine on them, won't work on a Mac though and I doubt for Linux either as I haven't been exporting different OS builds yet.

Have fun everyone!

[edit]

I haven't actually touched the code in over a month due to uni commitments, so my memory is a little fuzzy on where its at, so keep that in mind :p
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on August 26, 2010, 07:31:58 am
"a handful of files" -> no bitmaps, pallets, sound or such things
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 26, 2010, 07:36:21 am
...it's mostly only able to handle a handful of archive types and model files. The real point of this demo is to demonstrate the new interface.
I think I was quite clear...

It can read other stuff just there's no interface to manage it at the moment so there's really no point and saying they work.
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on August 26, 2010, 09:39:06 am
I understood you fully, I just clarified it.

BTW, I took the liberty of submitting your fileformat doc. Talk to the staff to get your forum account linked up with the submission.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 26, 2010, 09:42:53 am
I understood you fully, I just clarified it.

BTW, I took the liberty of submitting your fileformat doc. Talk to the staff to get your forum account linked up with the submission.
Tis cool, I just get a few replies questions off and on so I misread.
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on August 26, 2010, 02:35:26 pm
Awesome a new demo. This version has a save feature easily visible. But it doesn't appear I can save a model still. Is there any chance in your next update that this will be possible? I'd love it even if all I could export to was .obj or something else that was simple.
Title: Re: The Console Tool (by Low Lines)
Post by: Friedslick6 on August 30, 2010, 04:04:21 am
Vast improvements to the original make me happy :) . But would you by any chance remember how to zoom in on models in the newest release?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 30, 2010, 04:12:12 am
Vast improvements to the original make me happy :) . But would you by any chance remember how to zoom in on models in the newest release?
It works on a 3 button mouse system. Left button controls rotation, right button controls translation (up and down), and the mouse wheel is scale. Its down by holding the button down and dragging your mouse left or right, similar to Maya. It was set up to use the mouse wheel scrolling however sometimes caused an overflow of some sort so I disabled it for now.
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 01, 2010, 01:35:22 am
Hey Low Lines I'm curious will DirectX support ever be something added to your program rather than openGL?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 01, 2010, 03:30:10 am
Hey Low Lines I'm curious will DirectX support ever be something added to your program rather than openGL?
With Java, as far as I know there isn't like core support for 3D rendering (I could be wrong, there's a heck of a lot of Java to know everything), so you have to achieve it through plugins, or your own implementations. OpenGL is provided through a plugin called JOGL. As far as I know there isn't something like that for DirectX also but logically there would be some way of doing so just that's not in my area of expertise. Since I already have a working method of 3D so I'm not really concerned about looking into others right now, but hey maybe one day...
Title: Re: The Console Tool (by Low Lines)
Post by: Tauwasser on September 01, 2010, 08:08:12 am
DirectX isn't platform-independent, so it probably won't happen.

cYa,

Tauwasser
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 01, 2010, 02:56:01 pm
Ah okay I understand that's fine. In that case I hope a new version with the capability to export the 3D DS models your tool is capable of opening to some other format is released soon.

I'm currently using your tool to produce screenshots from the models of Kingdom Hearts 358/2 Days for the Kingdom Hearts wiki. But I've found that I get better texture display outside the program in my own 3D programs. But to get them there is quite difficult at the moment. I have to use 3DVia Print Screen to capture the model and textures. This is good and all but 3D Via tends to either totally lose or make the UV map extremely large compared to what it should be. That as well as quite often losing which textures are assigned to which parts of the model. So I have to spend several hours fixing them just to get one screenshot.

I understand that this may not be your main focus considering your work on an editor for the models in their native format. But if you were to make this capability of exporting to another format, I would forever be grateful to you.

Thanks Low Lines for all you do.  :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Friedslick6 on September 02, 2010, 08:29:29 am
Wow Zerox, that's a smart alternative. I think I've found my temporary model export work-around  :D .
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 02, 2010, 09:22:39 am
Well then I wish you the best Friedslick6. It's not exactly easy. And for some reason it doesn't appear to work on the newest demo of Console Tool which displays more models correctly. So it is probably best for us to just wait for Low Lines.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 02, 2010, 09:23:55 am
Well then I wish you the best Friedslick6. It's not exactly easy. And for some reason it doesn't appear to work on the newest demo of Console Tool which displays more models correctly. So it is probably best for us to just wait for Low Lines.
That would probably be because I'm using a lightweight container instead of the heavyweight one in the old version :P
Title: Re: The Console Tool (by Low Lines)
Post by: Friedslick6 on September 03, 2010, 03:55:03 am
Oh snap, that means I can't rip the textures/models like I was hoping to :\ . Oh well, cool program anyway.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 07, 2010, 06:00:12 am
Hey guys,
I'm on my term (one week) break at uni right now, which I have been using to get my website back on track and up to date. In terms of this project, it will be used to keep track of updates, and hold all my documentation which I will eventually get round to uploading.

Here's the link... http://llref.emutalk.net/v2/ (http://llref.emutalk.net/v2/)

As you can see, it's completely different to the old website, with the main focus being making it more compatible and standards compliant than the old one, which although was nice and fancy, seriously lacked in. It makes heavy use of PHP to try and automate a lot of the work of updating for me, and also has some basic filtering of content already implemented.

If anyone has the time, can they look over it and give me feedback please!! I particularly would like to know how the overall look and feel of the site including navigation sits with people, especially with the XML document viewer which is used to display file structures in a tree format. You can post comments and stuff on there also, however I'm going to clear everything once I'm ready to replace the old site, so it's all fresh and clean.

I still have to add some Javascript form validation as well as some filtering options and probably rework my logo too, but other than that, I think the site is pretty much ready.

Any feedback is greatly appreciated!!
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on September 07, 2010, 06:08:25 am
It looks a bit screwy on safari, unless the right updates box is meant to push down the website description :P but the overall visual design is very nice, inset text always gets a big plus in my book, good work :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 07, 2010, 06:17:33 am
It looks a bit screwy on safari, unless the right updates box is meant to push down the website description :P but the overall visual design is very nice, inset text always gets a big plus in my book, good work :thumbsup:
What version of Safari are you using? Mine displays it fine...

It should look like this.
(http://img87.imageshack.us/img87/5894/siteb.png)

I haven't really spent a heck of a lot of time debugging for older browsers, since I'm going to assume that the only sort of people who will look at it are computer people who generally care about browser versions and stuff :p I also refuse to make it work for IE, even though it can be done, I'm just not going to bother -_-

[edit]

Clearly there are still some cross browser bugs though since the section arrow of mis-aligned and the email icon on the footer isn't displayed, but I'll get round to fixing those :p
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on September 07, 2010, 06:24:28 am
I get this when I load the webpage from your link, other sections of the website look fine.
(http://i54.tinypic.com/orp2lh.png)

I find if I then click home it fixes a bit, but it's still a bit dodgy alignment wise:
(http://i56.tinypic.com/knjuc.png)

I'm using Safari 5.0.1 on OS X 10.6.5
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 07, 2010, 06:37:36 am
Hmm wells I just tested it on all Windows web browsers and none of them are plagued with the alignment issue...I guess it must be a Mac thing :( The closest thing I have to a Mac is a Touch and it doesn't have any problems either. This is why I prefer programming over web development -_-

Thanks for your info though, much appreciated.

[edit]

My guess is that there is some kind of global default to do with the width of the content that's not getting overridden on the initial load, probably padding or something. That's the annoying thing with floats, if you are even one pixel over, it pushes everything onto the next line -_-

[edit]

Or the fact that the right column is generated with PHP, I'm not PHP expert though, I just like the fact that I can cut back redundant html so I only have to edit it in one place.
Title: Re: The Console Tool (by Low Lines)
Post by: Spikeman on September 07, 2010, 08:41:30 am
I'm getting the same thing as Rhys in the latest Google Chrome in Windows XP.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 07, 2010, 08:49:19 am
I'm getting the same thing as Rhys in the latest Google Chrome in Windows XP.
Try it now...(fingers crossed)
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 07, 2010, 10:47:41 am
I don't have any issues with your site displaying incorrectly. It seems to be nice and simple yet with the capability for easily giving new information at ease from anywhere on the site. In short it seems good to me.

I'm glad to see you have time for working on your projects for at least a little while. I'm keeping my fingers crossed for a new version of console tools with any kind of export function for textures and/or 3D models.  ;D
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on September 07, 2010, 12:39:04 pm
works fine now, good job :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on September 08, 2010, 06:21:32 pm
Nice. I'm using the model viewer, but how does it zoom in and out?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 08, 2010, 06:38:31 pm
This was answered a page back...
It works on a 3 button mouse system. Left button controls rotation, right button controls translation (up and down), and the mouse wheel is scale. Its down by holding the button down and dragging your mouse left or right, similar to Maya. It was set up to use the mouse wheel scrolling however sometimes caused an overflow of some sort so I disabled it for now.
Please do try looking through the thread a little first in case someone has already asked the same question. I know it currently lacks a decent How To guide, however it is a demo...
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on September 08, 2010, 06:56:30 pm
I tried everything except that...
What about these NSBCA files? When I open one, it always goes to hex.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 08, 2010, 08:29:04 pm
I tried everything except that...
What about these NSBCA files? When I open one, it always goes to hex.
They're supported, just there's no interface yet to display them so it opens them as hex.

Just to re-clarify everything, here's what the program can do right now.

- It can open 3 types of models, NDS, Gamcube and Playstation (TMD) models, though only the NDS ones display properly if at all
- It can open NDS textures which has a basic interface for zooming, and toggling transparency, though there might be some errors with certain formats...

- It can open NDS Roms, NDS Archives (NARC), Gamcube ISOs (I think...), Gamecube Archives (RARC), and a few other Wii/Gamecube Archive formats including those used in the SSBB.

- It can export files from Archives/Roms which includes basic decompression support however it cannot covert any one file type to another

It may be able to do some other stuff, but other than the above, don't expect anything else to work, not in a form that's useful to users at least. It can open and understand a bunch more files but because there's not interface for them it resorts to opening them as hex, which again isn't working yet either... If you want to open more files, particularly those in the NDS 2D graphics area, use the older version.
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 09, 2010, 09:12:30 pm
Well Low Lines I decided I stop by and show you what I plan on using your tool for. I'm working on a fan game of sorts for Kingdom Hearts. It's still heavily in development but my hopes are that you will impliment some format to export DS models to a useable format or perhaps you could give me a link to where you have all your current knowledge of the DS model format so I could share it with someone who may be able to write a simple converter for me using that information.

The reason I hope for such a feature is because Kingdom Hearts 358/2 Days (on DS) is the only Kingdom Hearts game I have found that has the actual play areas exportable as a model/models. And I would like to use these playing area models to create my own levels for my fan game.

Now I realize people don't like doing anything without proof that work on said project has actually been worked on some. So here are 3 screenshots from it so far.

(http://i930.photobucket.com/albums/ad146/iZerox/Sora1.png)

(http://i930.photobucket.com/albums/ad146/iZerox/Sora2.png)

(http://i930.photobucket.com/albums/ad146/iZerox/Sora3.png)

I'd really appreciate such a feature or could you at least tell me if you plan to add a feature like that at all? I apologize if this seemed like advertising, it's not my intent here.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 10, 2010, 07:39:21 am
Zerox the thing is, its not a matter of "if" but "when". I fully intend to do so, just I can't give you like a date or anything right now. You also should probably be a little more conservative on your requests, otherwise it comes across to everyone else as a "noobish" sort of nagging if you know what I mean. I appreciate your eagerness here, but things just don't happen overnight, in fact I can't even work on my project much at all right now since I am not in the right mindset because of my studies.

If you check out the documentation on my temp new site I posted up earlier this week, it has the most complete documentation on NDS 3D models (NSBMD) on the net, including the stuff I've figured out on my own. If you happen to come up with an export implementation that works, feel free to give me a look so I might be able to support it in the future, since there are a ton of different formats to choose from and I'm likely to only support a few on my own.
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 10, 2010, 09:43:59 am
Well I'm glad to hear it is indeed planned. I was beginning to wonder such. I apologize if I appeared to be nagging. I'm eager is all.

I'll take a look at your documentation and I'll let you know if you get any success.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 19, 2010, 11:16:14 pm
Nothing like a new game coming out to get you back in the right mode for a bit of game hacking...
If anyone is familiar with the new Pokemon DS file structure, you would know it has been deliberately packed in a way to make it harder to extract files from it. However since my app doesn't rely on filenames to anyalze files, it made the job a little easier. It took about 3-4 hours to go through the 'a' folder, and it gave me an interesting chance to see how well my app faired. I found I was jumping back and forth between the newer version and the older version since one can open images while the other can open models/textures and I also had to restart them both several times because they gradually choked up a lot of memory, and it kind of didn't help I was streaming video to my PS3 and had multiple download apps running at the same time. The LZ77 compression found on a lot of files seems to be another extended version similar to that in kingdom hearts, in that it tries to decompress bytes further back that what is decompressed already, so I am yet to make much progress extracting the pokemon sprites...The few map models I looked at though are pretty damn awesome, really cannot wait until this comes out in Australia.

Here's the 'a' folder structure in case anyone is interested. Most of the key files are located in the first handful of archives. Note if by posting this causes any problems I'll remove it right away :S

Pokemon BW 'a' Structure (http://llref.emutalk.net/pokemon_black-white_structure.txt)

[edit]

Two new file formats that were popping up quite a lot in this game were NMCR and NMAR. NMAR is basically a clone of NANR, but NMCR is totally new. I'm not certain but I think these might have something to do with masking, since when you catch a new pokemon on your pokedex and does this cool little swipe up effect from a black silhouette. Can't do much with them though until I can extract the NCGR files which are compressed and probably encrypted too >_<

[edit 2]

The Pokemon Sprites narc (a/0/0/4) is a hefty archive, with 14285 files, which will take a while on any narc extractor to extract. I so need to build a progressbar, because it just looks like my app hangs for 2+ minutes :/

The layout of these files is as follows, each pokemon owns 20 files in sequential order, this further splits in half to cover front and back sprites with the assisting palettes being the last two files of each 20 file batch. This continues for 712-713 occurrences with the left over files all being extra palettes.

A breakdown of one batch is as follows...

Code: [Select]
#1 NCGR
#2 index file (just points to the same offset as the previous file i think...)
#3 NCGR
#4 index file
#5 NCER
#6 NANR
#7 NMCR
#8 NMAR
#9 raw data file
------------------> I assume this is where the break is between front and back sprites
#10 NCGR
#11 index file
#12 NCGR
#13 index file
#14 NCER
#15 NANR
#16 NMCR
#17 NMAR
#18 raw data file
------------------> End of back sprite
#19 front NCLR
#20 back NCLR

All NCGR and NANR files are compressed using a variant of the standard LZ77 compression.
Title: Re: The Console Tool (by Low Lines)
Post by: Bond697 on September 20, 2010, 01:14:50 am
i've been going through the white rom manually finding stuff, but i don;t have a way of decompressing the nanr files.  is there anything that exists now that can do it?  crystal tile doesn't, and i can't seem to find anything that will. 

thank you for the breakdown of "a", btw.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 20, 2010, 02:01:03 am
i've been going through the white rom manually finding stuff, but i don;t have a way of decompressing the nanr files.  is there anything that exists now that can do it?  crystal tile doesn't, and i can't seem to find anything that will. 

thank you for the breakdown of "a", btw.  it helps a lot.
I don't think anyone has yet documented the new compression format, most likely because no one with the skills has yet had reason to bother, which is why I'm hoping one of the Pokemon hackers does since it will probably allow for other games such as Kingdom Hearts to be accessible too. As far as I know, someone has to do a bit of disassembly of the game arm routine that decompresses it, now I can handle a bit of disassembly but I don't know the first thing about finding it in RAM or however they do it.

On a side note, I'm currently putting together a compact version of my app so I can work more on file support instead of being held up by the complexity of the GUI. Big biggest issue I've been having is with the fact that since I've started doing Java at uni, the way I program has changed quite a lot (for the better), and so when I sat down and look at what I had, it was a combination of not being able to understand my own code, and just not liking what I saw :(
Since the start of this project its been a repetition of completely rewriting my code near on from scratch after every significant break I took from it. I really hate doing it but at the same time it also consolidates my understanding of everything much better each time. In general the file management classes were pretty good, albeit needing a clean up, but most the GUI and property management stuff was just a bad case of nasty, which is why I'm doing a compact version without them. What I mean by compact is just a basic side panel and the display panel with maybe a simple toolbar for basic manipulation. The rest of the stuff I was working on in terms of the GUI is gonna be left out until I iron out the class design.

I'm also updating my docs to the XML format as I work through adding each format. My focus is the Nitro Graphics set (NCLR, NCGR, NCER, NANR, NSCR, etc), which will then be followed by my model stuff.

I don't know how much I'll get done since I am in the last month of my studies, but definitely November onwards I should be able to put full focus onto this again, and hopefully get past a few milestones.

[edit]

Here's a sample of the Compact version...

(http://img823.imageshack.us/img823/2848/ctoolmini.png)

As there won't be a browser, what will happen is when you open the app it will be blank, then all you will have to do is drag and drop a file onto it and it will load it, creating a new window for each one. Each window will behave on its own, with the exception of accessing other opened files to get combined types working together (ie palette with graphic). Really really simple and a heck of a lot easier to implement, meaning less time hopefully :D
Title: Re: The Console Tool (by Low Lines)
Post by: Klarth on September 21, 2010, 03:04:11 am
It's pretty rocking that this project is still going.  It's quite a lot of work on the programming part...and just as much on the reverse engineering side as well.  :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on September 21, 2010, 06:08:28 am
I've hacked the 0x11 compression format quite a while ago already. I've uploaded a C# and Java source of the decompression code here; http://code.google.com/p/dsdecmp/ feel free to use it however you like =). (you can also find a compiled version of the Windows command line version there)



By the way, the pokemon list file order is actually like this:

Code: [Select]
#1 NCGR (front, single frame)
#2 NCGR (front, single frame, female. pointer to previous is no gender difference)
#3 NCGR (front, frame parts. (limbs, eyes, etc. used for animation))
#4 NCGR (front, frame parts, female. pointer to previous if no gender difference)
#5 NCER (I think this is only for NCGR #3 and #4, looking at the labels stored in the LBAL section)
#6 NANR
#7 NMCR
#8 NMAR
#9 raw data file
------------------> end of front sprite data
#10 NCGR (back, single frame)
#11 NCGR (back, single frame, female)
#12 NCGR (back, frame parts)
#13 NCGR (back, frame parts, female)
#14 NCER
#15 NANR
#16 NMCR
#17 NMAR
#18 raw data file
------------------> End of back sprite data
#19 NCLR (normal)
#20 NCLR (shiny)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 21, 2010, 06:16:47 am
I've hacked the 0x11 compression format quite a while ago already. I've uploaded a C# and Java source of the decompression code here; http://code.google.com/p/dsdecmp/ feel free to use it however you like =). (you can also find a compiled version of the Windows command line version there)
Dude you are a legend!!
You know I was actually wondering why they had zero size files in archives (have seen it in other roms too), seemed totally weird, but using in that way with genders makes total sense!!
I'm working on some assignments for uni right now but I'll have to have a look through your code hopefully on the weekend so I can fix my code up. Mine is based on GBATek which is probably a little outdated by now :(
Title: Re: The Console Tool (by Low Lines)
Post by: RetroHelix on September 21, 2010, 04:20:40 pm
Sorry for posting offtopic but I have a to ask, where guys like you, Low Lines and Garyong, are publishing their knowledge about new fileformats or anything else related to reverseengeneering binary files. (besides this site)

Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 21, 2010, 05:31:35 pm
Not actively no, but I plan on making it a habit to keep my docs updated from now on. They'll be held on my site (which is in dire need of more monthly user visits) once I push the new design forward. I could have probably found Garyong's googlecode with a quick search but the last time I did do that, there hadn't been much progress on that front. Plus, no offense to Garyong but knowing what keywords to write to actually find the site I think would be hard since typically if I were to search for LZ77 decompression, I'd write that or maybe "ds lz77" etc which doesn't put his google page within the first page of results (and probably not the second or third either). Whereas if you put in "console romhacking" you get this forum as the second result, which is a more likely kind of search a person would make. "low lines" actually gets my old site on the first page too XD. This is all to do with accessibility in websites, which doesn't just mean people with disabilities as things such as search engines need a bit of help to better find stuff, which is why having a meta keywords list or try using words more likely to bring up your page is handy. Stuff like flash decrease accessibility since you can't access text stored in them easily, and japanese websites are worst with their images of text >_<
I want it that if someone wants to find docs on formats they can hopefully get my site on the first page, or at least someplace on romhacking.net so it can then point to my site. It probably the only benefit of having a doc submitted to the database here, it shows up in a search really quickly due to the large amount of visits to this site. It would be better if they were just linked though so I can make changes to the docs without having to resubmit them to the database...Mind I haven't really looked much into it as of yet.
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on September 21, 2010, 07:04:03 pm
Plus, no offense to Garyong but knowing what keywords to write to actually find the site I think would be hard since typically if I were to search for LZ77 decompression, I'd write that or maybe "ds lz77" etc which doesn't put his google page within the first page of results (and probably not the second or third either). Whereas if you put in "console romhacking" you get this forum as the second result, which is a more likely kind of search a person would make.
None taken, as I have to admit I just filled in some terms that came to mind when filling the tags.

I must admit most file formats I've hacked only have a somewhat-documented format on my own computer, scattered over the various tools I made (most of which were for personal use only). That is, unless I hacked a format to fulfil a request, the format is still not known to the general public (although most are just simple formats, or slight derivatives of more well-known formats). The formats hacked for requests usually get their format posted at the place where the request was made, but I don't have an organized online storage of any of these.
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 22, 2010, 10:51:35 pm
Plus, no offense to Garyong but knowing what keywords to write to actually find the site I think would be hard since typically if I were to search for LZ77 decompression, I'd write that or maybe "ds lz77" etc which doesn't put his google page within the first page of results (and probably not the second or third either). Whereas if you put in "console romhacking" you get this forum as the second result, which is a more likely kind of search a person would make.
None taken, as I have to admit I just filled in some terms that came to mind when filling the tags.

I must admit most file formats I've hacked only have a somewhat-documented format on my own computer, scattered over the various tools I made (most of which were for personal use only). That is, unless I hacked a format to fulfil a request, the format is still not known to the general public (although most are just simple formats, or slight derivatives of more well-known formats). The formats hacked for requests usually get their format posted at the place where the request was made, but I don't have an organized online storage of any of these.

I apologize as this is a bit off-topic but would any of the formats you worked with involve Kingdom Hearts on the DS?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 24, 2010, 05:22:18 am
Finally had a chance to go through Garyong's code and after a bit of fiddling around, I've successfully got it working in mine. When ever I go through someone else's source code the first thing I do is look for comments, and he has some really good ones in there, however they don't match up with his code.

Here's an updated version on his notes on the NDS LZ77/LZSS Compression...

Code: [Select]
----------------------------------------------------------------------------------------
NDS LZ77/LZSS Compression -> Credit goes to Garyong
----------------------------------------------------------------------------------------
Data header (32bit)
  Bit 0-3   Reserved
  Bit 4-7   Compressed type (must be 1 for LZ77)
  Bit 8-31  Size of decompressed data. if 0, the next 4 bytes are decompressed length
----------------------------------------------------------------------------------------
Repeat below. Each Flag Byte followed by eight Blocks.
----------------------------------------------------------------------------------------
Flag data (8bit)
  Bit 0-7   Type Flags for next 8 Blocks, MSB first
----------------------------------------------------------------------------------------
Block Type 0 - Uncompressed - Copy 1 Byte from Source to Dest
  Bit 0-7   One data byte to be copied to dest
----------------------------------------------------------------------------------------
Block Type 1 - Compressed - Copy LEN Bytes from Dest-Disp-1 to Dest
----------------------------------------------------------------------------------------
  If Reserved is 0: - Default
    Bit 0-3   Disp MSBs
    Bit 4-7   LEN - 3
    Bit 8-15  Disp LSBs
----------------------------------------------------------------------------------------
  If Reserved is 1: - Higher compression rates for files with (lots of) long repetitions
    Bit 4-7   Indicator
----------------------------------------------------------------------------------------
      If Indicator > 1:
        Bit 0-3   Disp MSBs
        Bit 4-7   LEN - 1 (same bits as Indicator)
        Bit 8-15  Disp LSBs
----------------------------------------------------------------------------------------
      If Indicator is 1:
        Bit 0-3   LEN1
        Bit 4-7   Indicator
        Bit 8-15  LEN2
        Bit 16-19 Disp MSBs
        Bit 20-23 LEN3
        Bit 24-31 Disp LSBs
        LEN = (LEN1 << 12 | LEN2 << 4 | LEN3) - 0x111
----------------------------------------------------------------------------------------
      If Indicator is 0:
        Bit 0-3   LEN1
        Bit 4-7   Indicator
        Bit 8-11  Disp MSBs
        Bit 12-15 LEN2
        Bit 16-23 Disp LSBs
        LEN = (LEN1 << 4 | LEN2) - 0x11
----------------------------------------------------------------------------------------
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on September 24, 2010, 05:57:29 am
@Low Lines: The GBATEK-like specification of the compression was indeed more geared to how I interpreted it, instead of how it actually should be. I thought I got the format right, but apparently not (which is why I didn't use bit-numbers in the in-line comments).

Also, when LEN is composed of LENx's, it should be +0x11 and +0x111, instead of -0x11 and -0x111 ;]
(Oh, and would you mind crediting me as either Barubary or Aygox if you decide to upload the specs somewhere? I've no idea why I used another username when registering here =/)

@Zerox; I sent you a PM instead, since we do not need to derail this topic. The answer's yes by the way.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 24, 2010, 06:11:17 am
@Low Lines: The GBATEK-like specification of the compression was indeed more geared to how I interpreted it, instead of how it actually should be. I thought I got the format right, but apparently not (which is why I didn't use bit-numbers in the in-line comments).

Also, when LEN is composed of LENx's, it should be +0x11 and +0x111, instead of -0x11 and -0x111 ;]
(Oh, and would you mind crediting me as either Barubary or Aygox if you decide to upload the specs somewhere? I've no idea why I used another username when registering here =/)
Yeah I personally hate dealing with compression formats because they explain them so poorly. The GBATEK style is incredibly clear and easy to understand, aside from that minus value stuff which I was considering editing anyways, you or I might be able to understand it, but if you read it like any normal person would, you'd get the wrong impression, which I think I did do when I first read GBATek all those years ago...

When I manage binarys like the above, I prefer putting them into one variable as it's much simpler referencing only to one rather than 4 like in yours (and probably 95% of programmers out there who do it and love short meaningless variable names >_<). I actually had to write them down on paper just to work them out properly myself, but I'm not complaining, these compressions have been a thorn in my back since I started working with binary files so I'm more than happy to think a little if it means I'm going to get somewhere with it :)

Will do on the credits part too. That's kind of why I try to be as consistent with everything I do, if a lowlines is on this forum, he's the same lowlines as the one on x forum etc :p Though I've kind of made a migration from writing it as "Low Lines" to "lowlines" because it's shorter and often fits better as a username in games :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on September 26, 2010, 12:10:05 am
So this new LZ77 compression is the reason why your tool couldn't open some compressed overlay files in another game? I think they could be the same as the Pokemon and Kingdom Hearts LZ77 compressions. Should I send you some samples?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 26, 2010, 12:28:27 am
Correct, Kingdom Hearts and Pokemon BW both used this variant of the LZ77 compression. It's easy to pick it out since the first byte in the file is 0x11, whereas the standard LZ77 will have 0x10.  If you want to test files, you can use Garyong's app (just drag the file over the exe and it should do it automatically, but basically any file his app decompresses, mine will do also (except for Huffman which I haven't added yet). The one other exception is where the compressed file has a "LZ77" stamp at the start of the file, which his app will hang, and the public releases of mine might bring up an error, but those are fixed too (Super Mario 64 DS is one game that uses them).

If people can send me Huffman compressed files though that would be great, because whenever I add another person's code, I like to be able to understand it first.

[edit]

And just to clarify a little, I probably wouldn't call it a new compression format. It's used to better compress files with a lot of long repeated sequences such as zeroed bytes in an image. The normal compression I think can only compress up to 15 bytes per flag, whereas this variant allows for several hundred. So instead of five bits required to compress a long sequence of zeros, it could be done using one bit, etc. That's the basic gist of it anyways.
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on September 26, 2010, 07:21:34 am
Since files with a LZ77 stamp do not get passed on directly to the decompression routine, but first through the code of the game itself, I haven't included such format(s) in DSDecmp. (and, to be fair, I didn't encounter such files while creating the tool)

This archive (http://www.propl.nl/random/huff_exs.zip) contains several files with huffman compression. I think they're from Summon Night: Twin Age, but they can also be from Summon Night X.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 26, 2010, 07:25:10 am
Since files with a LZ77 stamp do not get passed on directly to the decompression routine, but first through the code of the game itself, I haven't included such format(s) in DSDecmp. (and, to be fair, I didn't encounter such files while creating the tool)
I know, I'm just pointing it out in case someone actually tests it and gets an error :p It produces a weird error though.
Thanks for the files btw, will have a look at them at the end of week hopefully :)

[edit]

The image below is a prime example of something I figured out while going over the NCGR file format.

(http://img823.imageshack.us/img823/2356/tsutarja.png)

As you can see, the sprites come in two flavours, your standard 96x96 still sprite, as well as a Yoshi's Island style body part one for animations. First off the animation one is not tiled, but forget about that for now. With the standard sprites, they seemed to lack a buddy file that you'd assume be used for arranging the cells as a 96x96 block (if you were to open this file in the old console tool you wouldn't get a nice 96x96 image like above. But interestingly there were values stored in the width and height spots, but if you apply a bit of logic in there its kind of obvious. The first chunk of tiles make up a 64x64 block, then the following makes up the remainder on right hand side which is then followed by two blocks on next row, it turns out the width and height are used to arrange the tiles, I think this was how it was done in the GBA.

Code: [Select]
< - - - -  96 - - - - >
1 1 1 1 1 1 1 1 2 2 2 2
1 1 1 1 1 1 1 1 2 2 2 2
1 1 1 1 1 1 1 1 2 2 2 2
1 1 1 1 1 1 1 1 2 2 2 2
1 1 1 1 1 1 1 1 2 2 2 2
1 1 1 1 1 1 1 1 2 2 2 2
1 1 1 1 1 1 1 1 2 2 2 2
1 1 1 1 1 1 1 1 2 2 2 2
3 3 3 3 3 3 3 3 4 4 4 4
3 3 3 3 3 3 3 3 4 4 4 4
3 3 3 3 3 3 3 3 4 4 4 4
3 3 3 3 3 3 3 3 4 4 4 4

This only applies to one way the ncgr file seems to store an image, the other two being like the animation graphics above where you have a straight graphic that is drawn line by line, and the last type being one without any type of internal alignment and relies on a NCER file to position everything (these are normally very complex sprites that makes use of layering cells on top of each other).

[edit2]

Wrote a quick batch routine and exported all single frame sprites (normal, shiny, gender, forms) for BW to PNG :) Some of the new ones look awesome! I'm not going to put them up since this image ban is on, and I also doubt I'm the first to extract the images, but its an odd feeling, going back several years I was one of those noobs following fileb and all those other hackers as they hacked the games :)

Anyways just a few things I noted on the archive...The first 712 batches of 20 follow the format already mentioned, then you have the substitute doll sprite, followed by about a dozen palettes which I think are used for battle effects (ie lightning makes your poke go funny color). Also with the nmcr and nmar, this is still an theory but seeing how the ncer and nanr handle animations of a pokemons limbs separately, I think these last two files are like layer on top of that. So the nmcr would position each body part (which are a group of cells), and the nmar would manage the animation for them. I'm still only up to adding NCER cells so I can't test my theory yet, but due to the similarities in the file structures and the way pokemon sprites are animated, I think it's highly likely :)
Title: Re: The Console Tool (by Low Lines)
Post by: Mauron on September 27, 2010, 03:27:44 am
If you don't mind my asking, how are you handling the variations between compression formats of the same type?

Are you making fifty million variations of an LZSS routine, or do you have one routine that can take variations into account?
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 27, 2010, 03:32:28 am
Low Lines I hate to be a bother but is there any chance you could compile your program into a .exe file as opposed to .jar?

I realize that your intent is full multiplatformability but I would love to try something with a .exe version of the program and I think it might fix some issues my computer seemed to have. I'm not asking for new features just the same as your newest demo only as a .exe file if that's possible.
Title: Re: The Console Tool (by Low Lines)
Post by: BRPXQZME on September 27, 2010, 03:40:52 am
It wouldn’t fix anything. Java don’t swing that way.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 27, 2010, 03:49:03 am
@Mauron
At the moment, it only decompresses the one NDS uses, I haven't actually dwelled too much into other stuff that use LZSS mostly because I don't understand them well enough. But when I get to that point I'd definitely try and write it to allow for variants, the iffy part is detecting compression in a file properly.

@ZeroX
I don't think Java exports to exe, I know it can be done, but its a lot of mucking around and as you said, it will then only work on Windows. So that'd have to be a no there. The only way you could do it would to port the code to another programming language, which I'm not going to do since I'm quite comfortable with Java and I think I've already given myself enough work to do.
If you are having issues running the program, by all means post them up so I can see what's going wrong.

@BRPXQZME
Was that a joke? Because I found that funny, "java don't swing that way" haha.
Title: Re: The Console Tool (by Low Lines)
Post by: BRPXQZME on September 27, 2010, 04:08:26 am
Well, I wrote a post in an attempt to explain how compiling Java to a Windows executable, while possible (namely, through GCJ (http://gcc.gnu.org/java/)), will cause more problems than it solves. The “why” behind it is essentially that the JVM is the only implementation of Java with the entire API done and working, and the only implementation of Java with the language compiling and running relatively error-free. But translating the entirety of the details in lay terms was taking more than two paragraphs, which IMHO causes more confusion than it clears up.

In the end, I had a pun in there somewhere and couldn’t resist throwing it away, so I deleted the rest of the post. I’m a sucker for groaners; it runs in the family ;)
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on September 27, 2010, 08:40:03 am
Wrote a quick batch routine and exported all single frame sprites (normal, shiny, gender, forms) for BW to PNG :) Some of the new ones look awesome! I'm not going to put them up since this image ban is on, and I also doubt I'm the first to extract the images, but its an odd feeling, going back several years I was one of those noobs following fileb and all those other hackers as they hacked the games :)
You are correct. (http://spriters-resource.com/ds/pokemonblackwhite/) The palettes at the end are alternate palettes for Acreus (#493 = files 9860-9879) by the way.

Quote
Also with the nmcr and nmar, this is still an theory but seeing how the ncer and nanr handle animations of a pokemons limbs separately, I think these last two files are like layer on top of that. So the nmcr would position each body part (which are a group of cells), and the nmar would manage the animation for them. I'm still only up to adding NCER cells so I can't test my theory yet, but due to the similarities in the file structures and the way pokemon sprites are animated, I think it's highly likely :)
I suppose I'll put my info so far on the NCER format here, if you need it or want to compare notes;
Code: [Select]
/*
 * NCER:
 * <NitroFile("RECN",version,fileSize,headerSize,numSections)>
 * CEBK cells;
 * LABL labels; // not always present.
 * UEXT uext; // ?? only has one value, 0 or 1. not always present.
 */
/*
 * CEBK; Cell Bank. header size: 0x20
 * <NitroSection("KBEC", sectionSize)>
 * u16 numCells;
 * u16 extendedCells; // 0 or 1. if 1, uses extended cell data
 * u32 unknownConst; // always 0x18 ?
 * u16 padding; // is it?
 * u16 flags;
 * u32 partitionOffset; // if 0, no partitions. rel to section start.
 * u32 padding; // is it?
 * u32 padding; // is it?
 * CellData[numCells] cells; // CellDataEx if extendedCells == 1
 * OAMData* oams;
 * u8* padding; // may not be present
 * PartitionData[numCells] partitions; // partitions the graphics for each cell
 *
 *
 * flags: ABCDEFGH IJKLMNOP
 * A->M: only seen 0 here
 *    N: if 1, uses sub-images
 *   OP: all tile indices in OAM data should be left-shifted this much  (and then right-shift 1 if 8bpp image)
 */
/*
 * CellData:
 * u16 numOAMs;
 * u16 something; // ????
 * u32 firstOAMOffset; // rel to first OAMData of section
 */
/*
 * CellDataEX:
 * <CellData>
 * s16 xMax;
 * s16 yMax;
 * s16 xMin;
 * s16 yMin;
 */
/*
 * OAMData:
 * s8 xpos;
 * u8 xflags;
 * s8 ypos;
 * u8 yflags;
 * u16 data; // &0x07FF = tile index. >> 11 = palette index(/offset?)
 *
 * xflags: ABCDEFGH
 *  AB: 1st index of dimension matrix
 *   C: ??
 *   D: ?? only seen 0
 *   E: ?? only seen 0
 * FGH: ?? PKMN BW has 111 or 011 here; which is the only game I've seen non-000
 *
 * yflags: ABCDEFGH
 *  AB: 2nd index of dimension matrix
 *   C: 1 -> mirror Y
 *   D: 1 -> mirror X
 * EFG: ?? only seen 000
 *   H: ??
 */
/*
 * PartitionData:
 * u32 start; // in pixels
 * u32 length; // in pixels
 */
The dimension matrix indices are the 'width' and 'height' on your old(?) format page.
'sub-images' means the graphics data is non-tiled, and the referenced OAMs are sub-images of the complete graphics. The start offset remains the same however.

This used to work for most games, however for this latest pokemon game, the OAMs in a cell do not seem to be put in the correct place any more. (unless they do not use the NCER, which I find unlikely, or the NMCR/NMAR can correct that) There's about 4 pixels between the OAMs =/

I'm available if you want to collaborate on this by the way. I know I'll be continuing my own investigations in any case. =)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 27, 2010, 09:57:14 am
Oh goody some fresh docs to go over :)

Am not used to calling them OAMs :p I did notice though that when you load them into my old app you do see that 4px gap you spoke of, but I'm thinking it is just a mis-assumption of the format cos it sounds awfully stupid to use a second file to fine tune positioning.

I've uploaded the handful of docs (http://llref.emutalk.net/v2/docs.php) I've been working on as of late including the NCER (http://llref.emutalk.net/v2/docs.php?order=&sort=&page=&id=ncer.xml) one, note that I'm still using my WIP site so links may change in the future...It's been yonks since I've really looked over the NCER format, and I was more taking stabs at it back then. I don't think I've made any real changes to it as of it, but will do as I work more on the format.
Title: Re: The Console Tool (by Low Lines)
Post by: Mauron on September 27, 2010, 11:35:06 am
Low Lines, thanks for the feedback. Much appreciated.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 27, 2010, 08:51:14 pm
Here are my preliminary notes on the NMCR format...
Code: [Select]
====================================================================
Nitro Mapped Cell Resource (NMCR) ??
====================================================================
0x0000    0x4    Magic (RCMN|0x52434D4E)
0x0004    0x4    Constant (0x0100FEFF)
0x0008    0x4    File Size
0x000C    0x2    Header Size (0x16)
0x000E    0x2    # Sections (Always 1)
--------------------------------------------------------------------
0x0000    0x4    Magic (KBCM|0x4B42434D)
0x0004    0x4    Section Size
0x0008    0x2    # Items
0x000A    0x2    Contstant (0xBEEF)
0x000C    0x4    Offset 1 (Index Table) - Relative to Section + 0x8
0x0010    0x4    Offset 2 (Data Block) - Relative to Section + 0x8
0x0014    0x8    Padding?
--------------------------------------------------------------------
Repeats * # Items (Index Table)
--------------------------------------------------------------------
0x0000    0x2    Count 1
0x0002    0x2    Count 2 (Count 1&2 always match?)
0x0004    0x4    Data Offset - Relative to Offset 2
====================================================================
Repeats * Count 1 or 2 ?? (Data Block)
====================================================================
0x0000    0x2    NANR Bank #?
0x0002    0x2    Signed?
0x0004    0x2    Signed?
0x0006    0x1    Always 32?
0x0007    0x1    NCER Bank #?
Am using tsutarja and zekrom as my basis for testing. Overall the file structure appears pretty straight forward except for the Count 1 and Count 2 values. Because there are two, it makes me think these represent X&Y or something similar, but the fact that these variables are separate and so far have been identical in every file I've looked at makes me wonder. The highest value of the NANR Bank # seems to match up with the number of NANR banks in the associated file and the NCER Bank # is does not seem to be higher than the total number of NCER banks in its associated file as well.
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 28, 2010, 02:45:10 am
Okay I realize this may have been a bit rude to even try but anyway Low Lines you mentioned that the first version of Console Tool used a heavy version of openGL and the newer one used a more lightweight version or something along those lines. So I downloaded the JDK and tried swapping .class files under formats from the new version to the old one. I wasn't really successful there. And then I tried swapping all 4 jogl related files from the old version to the new version in hopes to recreate the heavy openGL environment so I could use 3DVia Print Screen to capture the model from Console Tool (despite the fixing I had to do after doing such).

Your program is the only thing that's even been able to display KH models properly with any accuracy:

(http://img14.imageshack.us/img14/8381/xioninviewerripmeplz.jpg)

But this is only in the newer version of Console Tool. I was able to use 3D Via Printscreen on the previous version to capture models and textures but the mesh is terribly disfigured in the last version as well as the uv maps being totally destroyed. Though being able to capture the model even without UVmaps in a normal pose would help me a lot. Granted a real export feature would be loads better but I've already asked and you said I had to wait on that.

The real reason I asked about converting it to .exe was because I figured that maybe then I could capture the model easier. My only problem has been with your first version of Console Tool and it seems that it sucks all the memory out of my computer and drags it to a halt. Perhaps this is why you changed the way openGL was rendered.

Anyway I guess my question in all this is; If it's not a hard task could you possibly impliment proper format information reguarding DS models into the old Console Tool or perhaps make the new one use the same or similar openGL as the last one?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 28, 2010, 06:01:07 am
Let's see if I can explain it a little...Now some of this may be incorrect/poorly explained so if there's a smarter Java person, your welcome to clarify.

In Java there are two ways objects are rendered, heavyweight and lightweight. Heavyweight which is the older of the two basically draws things directly to the screen. This means each heavy weight object has its own "window" and I think the biggest problem you tend to face is when you want to layer heavyweight objects, because it doesn't work too well.

Lightweight was sort of like the solution to the layering issue, and in simple terms lightweight a drawn using a heavyweight as its base. The core GUI components are I think subclasses of the Window class which is a heavyweight object. The Window class is what is used to draw the GUI onto the screen. A lightweight object would be like a button or a textfield. They don't need their own window, and logically they sit inside one anyways (poor explanation I know, I'm sorry). I think another important point to make is that the updating (re-rendering) of lightweight objects is all handled by the heavy weight container.

But basically the old ctool drew the 3D directly to the screen, and if you noticed, nothing ever sat on top of the viewport, because it would get hidden behind it on refresh. And because it was a heavyweight object, and not to mention very poorly managed in terms of refreshing, it choked up memory since it was consistently refreshing the viewport by timed manual calls.

The new ctool uses a lightweight version which, basically removes the need to manually refresh it, cuts back on CPU dependencies and allows for other components to overlap it without needing any special coding needed to make it work. In general when you program in Java, the better use you make of the built in APIs, the better your program, aka you don't try to tame a lion by shoving a stick up its arse, because that will just make it angry :p

The reason why replacing the dlls didn't work is because the type of component the viewport is, is determined in the source code, ie new HeavyweightPanel, or new LightWeightPanel (not actual names but you should get what I mean), so it cannot be changed outside of the original java classes, since the package I give out is an already compiled version, same thing applies with C/++/# I think. Unless you actually implement support for render engine switching, you are basically stuck for options.

Essentially changing it between heavyweight and lightweight is just a matter of changing what the viewport is created as, as well as any references to it since Java is object orientated, but it will mean all the stuff that floats over the top may get overlapped, and a more complex refreshing system will be needed, and in general a bit more work involved to get it to display properly. Right now I want to keep away from mucking with the GUI code, since there are quite a lot of problems with how I've implemented and structured my classes which I'm now aware of since doing some Java at uni. I have to go through the whole thing and sort out the GUI better, which will take a lot of time, which I don't really have while I'm still studying. It's why I'm working on the Compact version so I can work on it in smaller modules and keep the more GUI intensive version for a major release later on once I sort out the problems I have with the code.

As for fixing up the older version with the fixes I know now, I actually don't have that complete version anymore, as back then I didn't have any sort of version control system for major changes.

Going back to the formats stuff, I've been going over the NANR file today and I've figured out a bit more of the file structure as well as having a few more assumptions. For one, I've been wondering where the rotation information for pokemon sprites are stored, because if you have opened up a set (NCLR, NCGR, NCER & NANR) you would notice that the animations have a bunch of frames but next to no movement. My assumption is that the NANR not only store NCER bank ids but also some form of transformations in the second chunk of data found in the file. From what I can tell, the data in the second block comes in three sizes determined by the value of the Data Type (4bytes, 8bytes & 16bytes). This is where I am assuming where the presumed transformation data is stored.

Added my updated notes on the NANR (http://llref.emutalk.net/v2/docs.php?order=&sort=&page=&id=nanr.xml) to my site, and below is a printout of Tsutarja's frontal NANR (hopefully its readable...). Just looking over the Frame Data, it looks like there are some signed values in there, though I'm kind of curious why this 0xBEEF constant keeps popping up, perhaps a stopper tag? I've seen it in various Nitro formats so I'm sure its not an actual value or anything :S

Tsutarja's NANR (http://llref.emutalk.net/tsutarja_nanr.txt)

I've got assignments to work on right now, but I'll have a go at analyzing the frame data later in the week...
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on September 28, 2010, 04:09:29 pm
Ah okay I understand a lot better now so thank you for taking some time to explain stuff to a newbie like myself. I won't bother trying to replace/swap anything anymore. I'll just wait patiently for a new version.

That's interesting about the Pokémon sprites and it sounds reasonable. Though I don't know anything that I could help with there so I'll just go back to waiting quietly.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 28, 2010, 05:53:19 pm
Zerox you are consistent at the very least, I'm quite a fan of consistency (though I'm not encouraging you to actively nag lol).

Its not the model viewer but its a quick snapshot of where I'm up to...

(http://img823.imageshack.us/img823/3848/ctoolpalette.png)

I've added support for reading Adobe Color Tables (.act) and Windows Palettes (.pal), which will be the two main options for exporting palettes to native formats. If anyone has another format they particularly want that's either got some docs on it or is easy to implement let me know.

The GUI is currently missing Toolbar buttons, which will be mostly file management options such as exporting and creating a new file from scratch, but what I'm basically aiming for here is to have both read/write support done for each file type before moving on to the next. I already have a basic display going for NCGR and NCERs but once I'm done with palettes, I'll move onto them. Then its NANRs and then models okay? I'm not making any promises or setting any dates, but hopefully that sounds a little better than "waiting patiently" without a clear indication of when :p I'll also start releasing it in modular versions so people can actually make use of it (not that anyone cares much for palettes anyways XD) instead of waiting forever :s
As for exporting graphics, I want to take another crack at reading PNG files, but I'll just use the internal Java image exporting functionality for now since it's basically a one liner to implement. You lose the control I want over the structure of the file (particularly the way the palette is ordered) but it'll be way better than screen capturing Mario Kart DS sprites one at a time which I think one poor dude was doing :p

Also with compression, right now I've got it set up that when you open a compressed file, it will save a decompressed version with "_extract" added to the name into the same folder of the original file. If it proves to be an annoyance, I may just move that functionality to the toolbar, however it is handy if you have a handful of compressed files and you want to decompress them all at once...perhaps maybe some kind of batch option so then you don't have to close x number of windows afterwards :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on September 28, 2010, 08:23:17 pm
Will it be able to export palettes to raw?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 28, 2010, 11:11:35 pm
Adobe Color Tables (act) are the same thing aren't they :p
Title: Re: The Console Tool (by Low Lines)
Post by: Mauron on September 29, 2010, 12:21:30 am
Adobe Color Tables (act) are the same thing aren't they :p

If that's the case (I don't know one way or another), I think it would be worth having something in the program that mentions that. It would be a simple way to avoid confusion.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on September 29, 2010, 01:03:00 am
The reason why I want to avoid using raw is because its like bin, it can hold just about anything and without a unique header/identifier it means that's one more thing the user has to figure out. Obviously there would have to be the option to open generic files like that, but since ACT's are a little more specific (and a less common extension), I thought it would be a better choice.

Will have to work on the help file too, thanks for reminding me :)
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on October 08, 2010, 06:43:26 pm
I was just using Console Tool to view BTX0 textures and apparently it's a container for many textures. But each has their own palette so I have to switch the texture and palette each time. My suggestion is that you have a check box option to be able to change both at once. And export function for at least textures would be helpful to me at least for getting to the KH stuff.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on October 08, 2010, 11:46:14 pm
I was just using Console Tool to view BTX0 textures and apparently it's a container for many textures. But each has their own palette so I have to switch the texture and palette each time. My suggestion is that you have a check box option to be able to change both at once. And export function for at least textures would be helpful to me at least for getting to the KH stuff.
The only iffy bit with linking palettes to textures is that like texture at index 1 will not automatically be partnered to palette at index 1, as far as I know the pairing is done by a section in the nsbmd file it is used for. A toggle option sounds like a good idea though.

Been busy busy busy with uni assignments, have to complete a website redesign, prepare a power point presentation and finish the report by the end of next week, so my buddy and I are flat out getting the job done :p After that I only have 3 exam days and then I'll be able to get stuck into this project again :)
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on October 13, 2010, 02:06:35 pm
Ah okay. Well thanks to some help from Barubary I was able to export textures. So I guess there isn't quite as urgent a need for an export feature in your program for textures yet I guess.
Title: Re: The Console Tool (by Low Lines)
Post by: beissemj on October 13, 2010, 11:52:27 pm
Any chance of getting the source code for this? I'd like to take a look
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on October 14, 2010, 04:20:25 am
@Zerox
Good for you :p

@beissemj
That would be a nope. If you can't make sense of my docs, nor locate all the other programs (with source code), that most of the stuff mine does and is generally ported/based from, then you would honestly have no hope of making sense of my code. That's not trying to be offensive, that's just a fact.

It's like when they say looking at an emulator's source code doesn't automatically mean you'll be able to write an emulator, if you don't understand the docs, it's pointless, I've tried and it's totally true.
Title: Re: The Console Tool (by Low Lines)
Post by: beissemj on October 14, 2010, 05:29:18 pm
@low lines
That's pretty unfortunate. You yourself have said on multiple occasions that university has kept you pretty busy, leading to long periods of inactivity on the project. Considering that so many people are interested in what you are working on, I don't see the harm in releasing it under a GPL/BSD license so that if you are busy other people can contribute to the project. Since it's not a commercial product and you aren't selling it, I see no compelling reason not to put it under CM on google code/github or similar. A very unfortunate part about the rom hacking community (imo) is that people start really cool projects like yours, but then for various reasons end up not finishing them (which is fine as I'm certainly guilty of that myself). The problem with this is that the community is then left with a prototype which shows lots of promise, but because the author didn't release the source code is now worthless. Why is it that no better tile editor has come out since Tile Molester which was released almost 7 years ago?

A community of people developing working in conjunction with one another always leads to a better end product. The more eyes you have looking at code the better that code will be. You claimed awhile back that you had to refactor various parts of your code to use a MVC pattern and that you were using classes incorrectly. Learning by trial and error is the most difficult way to learn, if you had a community of developers backing you they might have been able to provide guidance on better design paradigms. The end result? more progress on your tool.

Would I personally be able to understand your code? Maybe, maybe not. I've never dealt directly with openGL so I'm not familiar with the JOGL bindings that you are using, but that certainly doesn't mean someone else wouldn't be able to contribute.

*EDIT*
The portion of your code I'm interested in is the color table and the tile viewer / hex editor.
Title: Re: The Console Tool (by Low Lines)
Post by: Made in China on October 15, 2010, 01:23:38 am
I think he worries less about the design part and more about the format decoding\encoding part, which is the core of the project. I don't think many people can contribute to that part, as they lack understanding of various formats, and can mostly be one-shot contributers.
But even if you could help there, he is already doing a pretty good job deciphering the format documentations by himself and converting them to code. So if you want to help him you could just bring him the documents - there's no point in making the project open source.

There's also the point of ownership. I don't know if it applies here, but it's generally more rewarding to build something grand by yourself, so you can see Lowlines doesn't share.

Anyway, I think he's doing well enough, and don't see him abandoning the project, as he does update the project regularly. So I don't think there's any reason to worry about it.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on October 15, 2010, 03:10:56 am
@beissemj
You know I've actually kind of been expecting a response such as yours for a while. I've watched it happen and many other projects before, so I knew it was just a matter of time :p
I totally understand all the points you make, in fact they say in uni that you won't make it anywhere if you can't work collaboratively with others. But the other thing is I've just come out of my first group project on IT development and it was the most horrible experience I've had to date. Not only was my partner incredibly stubborn and believed he was right all the time, but his overall standard was way below what I was aiming for so I was doing a lot more work than what I had to just to get the project up to a level I was proud enough to submit as my work. Up until I spoke to my lecturers, who were super awesome and handled it magnificently, I was really dreading working on my assignment despite it being something I genuinely enjoyed doing because of the way this other fellow was. I'm not saying every group project is going to be like that, but the fact that you've brought it up right now is really really bad timing.

I've got several gigabytes of past projects (40+ projects, not just java) that I've started and not finished either due to getting bored or just not being able to go further with them. This project has been by far the one thing I've stuck to longer than anything else, and I can't see myself stopping, because I like hacking formats and it's something I can actually do, unlike making games or writing emulators which either bore me easily or just get too complex after a while. If I ever did suddenly decide to abandon it, then I would put up the source code, but I plan on sticking to my guns for a long while yet so :p

As for the Tile Molester thing, its obvious, people (who are smart) just aren't that into making little public apps like they did 10 years ago. Back then iirc it was sort of like a boom in the whole hacking front, but since then there's been lots of things going on, such as how companies are making a real effort to prevent illegal use of their stuff. And also alot of those "people" probably have jobs now, or lives, unlike me :p I'm not saying hacking in general is any less than it was, there just aren't any uber tools popping out anymore, because they do take a bit of time to create.

I still use Tile Molester myself to check my assumptions on things when I'm hacking files, because it does its job, just because its old doesn't mean it needs to be replaced...But hey a tile editor is easy to make so why don't you make one :p

My hex editor never made it past displaying the data onscreen, though I intend to change that...

@Everyone
I've still got an assignment to finish off today, and I need to put in some time to study over the next week for exams, but other than that I'm essentially on my break now so :)
Title: Re: The Console Tool (by Low Lines)
Post by: beissemj on October 15, 2010, 11:57:37 pm
@Made in China
I do understand the ownership concept, and I certainly agree that there is a sense of pride in accomplishing big things. On the other hand it usually takes many people to accomplish great things. Think of an author for example. His name is still on the cover, but there is an editor, publisher, illustrator, etc.

Quote
I think he worries less about the design part and more about the format decoding\encoding part, which is the core of the project
And just some food for thought: If the goal of the project is to take design documents--information which is freely available for the most part--then why can't the end product also be freely available?

@Low Lines
Well, as long as when you get tired of the project you are willing to open it up to others, then that's resonable. Sounds like you had a rather unfortunate experience at uni which is weighing into your decision, but I can understand that. While I don't agree with your decision, it is certainly your right to do whatever you want with the project. For what's its worth you are doing a great job, and I applaud your work.  :thumbsup:

Also, the maturity of your response is appreciated. I was expecting more of flame than an answer, but glad to see people can be civil.

Quote
I've got several gigabytes of past projects (40+ projects, not just java) that I've started and not finished either due to getting bored or just not being able to go further with them.
You and me both =(
Title: Re: The Console Tool (by Low Lines)
Post by: Jandazekon on October 16, 2010, 08:45:05 am
Low Lines.
Are you capable to build mid2bgm and dls2wd by using source code from both vgmtoolbox/mkpsf2 and vgmtrans.
I like to hear my own music in a ff x-2 psf2 file.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on October 16, 2010, 09:29:56 am
@Jandazekon
I haven't dabbled too much with audio as of yet and the last time I did was over a year ago while trying to view NDS audio formats, which worked somewhat though ADPCM (if that's the right acronym) files were a bit chocky. Have not done much on midi type audio though, I understand how they generally work but I haven't sat down and tried as of yet. Just looking at your suggestion, you seem to be talking about at least 4 different file formats, and I think to build a decent converter you would first need to be able to read and freely modify both formats first. Source code would help but documentation would be better. I'm not in any rush to work on audio formats at the moment though. The only stuff I'm really wary about are compression formats, mostly because they are generally quite poorly explained.

@beissemj
I think that if someone is going to put something out for the public to see, they should be accepting the fact that some people won't totally agree with them. If all they do is curl up into a ball and snarl at others when they give out opinions, then they shouldn't be putting it out in the first place :p Flaming is not something I see much future in, plus this is my own thread so I'd rather not sully it up with conversation that isn't worth reading, hence why I try to encourage discussion, and why I write a lot when I try explain something :p
Title: Re: The Console Tool (by Low Lines)
Post by: SCVgeo on October 19, 2010, 12:03:37 am
@Zerox
Good for you :p

@beissemj
That would be a nope. If you can't make sense of my docs, nor locate all the other programs (with source code), that most of the stuff mine does and is generally ported/based from, then you would honestly have no hope of making sense of my code. That's not trying to be offensive, that's just a fact.

It's like when they say looking at an emulator's source code doesn't automatically mean you'll be able to write an emulator, if you don't understand the docs, it's pointless, I've tried and it's totally true.
Just stopping by to add a note that one can understand the docs yet not want to undertake such a large project as yours.

Also, programming your project in Java is practically making it open source. You should just go all the way and make contributions easier.

@beissemj JD GUI is your friend.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on October 19, 2010, 06:42:28 am
@SCVgeo
Just because something can be done, doesn't mean you should just give in to people. I've made it clear what I am willing and unwilling to do, I'm not going to change my mind if I get a hundred people to tell me to do otherwise, whether it be right or wrong. Forcing/pushing someone into something can have a negative effect, and I don't recommend it here or anywhere else. People are welcome to voice their opinions, however anything after that is just not nice.

If people are so insistent on seeing my source code, then by all means, find a decompiler and use it, just be aware I don't condone this. I have my reasons, it's up to you whether you choose to respect them or not.

Also I totally agree with the docs comment, I'm pretty sure I've already pointed that out, however I just wasn't as blunt about it.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on October 19, 2010, 03:49:20 pm
Say, if Console Tool will be able to export sounds, would it be able to keep the names? I know sounds are named but none of the programs I've tried keep them.
Title: Re: The Console Tool (by Low Lines)
Post by: Vague Rant on November 06, 2010, 06:19:08 am
Unsure if this might already be planned, but I might just literally kill for it to gain TPL (texture format used in a number of Wii games) support. I'm working on a translation hack at the moment and looking for the exact TPLs I need to modify is needle-in-haystack stuff. I'm extracting u8s which themselves include ~20 LZ77'd u8s, which in turn have anywhere from 1-15 TPLs I have to convert just to get a look at them and go "Nope, not what I'm looking for." If I could just straight out open up these files and see a preview of the TPL right there it'd make the entire process so much less horrible.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 06, 2010, 11:01:57 am
@Koetsu
I'm a big fan of keeping stuff true to the source, so keeping internal names and the like during exporting would be expected. I don't really see the point in changing something like that other than just being lazy writing the exporter.

@Vague Rant
YACD has plenty decent docs on gamecube/wii formats including TPL so they'd definitely be on my list after the NDS stuff. I also know what you mean with finding the right files in games, and to me the companies are definitely making a clear attempt at making that job even harder by scrambling the files etc etc. Cutting a 4 step process (find, decompress,export,open) using multiple apps into a simple task has been one of major goals of this whole project, and it really does make life easier, when it works I mean :)

@Current Status
I'll be honest an say I got a bit peeved after the earlier conversation going on in this thread, and so just played GTA IV for a week while finishing off my exams which finally ended last Wednesday (hooray!!). But after that I sat back down and it was great to work on this project again, but at my own pace. Encouragement and all that is great, but too much pressure and it just becomes a task and not something fun :p

I got the palette editor/exporter fully working, which allows you to open/export between 4 different palette formats including NCLR, though there are still a few unknown bytes in the header which aren't being properly filled in yet. I immediately started working on the tile/image editor, but I kind of dropped it and went on to working on the drag and drop interface of the full version, which is coming along quite nicely :) Property/Actions are handled (I think) much more efficiently by use of property change listeners and you can freely move rearrange the gui almost as well as with Adobe apps. I still have to add resize functionality as well as rearranging the tabs in a group (the grey boxes) through dragging, but moving components from one place to another is all completely working. I've also started making use of the Properties class so you can rearrange the gui and it will still be the same the next time you open it :)

(http://img201.imageshack.us/img201/7170/ctoolmini2.png)

(http://img149.imageshack.us/img149/6629/ctool3.png)

For those who want to have a play with the palette editor, I've loaded a build of the Mini version along with a preliminary how to doc I've started writing. You can open the other NDS graphic formats (NCGR, NCER, NANR, NSCR), but at best you'll get a rendered image (if anything) as I've only got as far as the first format. Also you should be able to use it as a quick NDS LZ77 decompressor by opening any compressed files. It will automatically export the decompressed version with an "_extract" added into the filename.

Download: Console Tool Mini (Demo) (http://llref.emutalk.net/downloads/consoleToolMiniDemo.zip) 199 KB
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on November 06, 2010, 01:52:19 pm
Awesome.

For me this program is less used for practical purposes and more enjoyment.  I like just sifting through games and seeing their stuff in a different way, and always finding things unused.  Programs like these make that pretty fun to do :)
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on November 06, 2010, 03:17:07 pm
I really like the photoshop-inspired UI you've got going there, how long did it take to code that? I shudder to think how long that'd take to re-implement in Objective-C, probably months -.-;
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 06, 2010, 09:25:58 pm
@Celice
That's great. I also like doing that because I like to see how developers make their games, it's also really fun getting a new file and sitting there trying to figure out what it does :)

@Rhys
Wells obviously I had a partially working version from the older app to begin with but the only bits I really took from it were the component painters and a few good bits of code and concepts (though not very much). The top level component which I call the Desktop holds three layered panes, one for toolbars, one for the adobe uis and one for the actual open files. When it refreshes, it lays out each pane and then tells the next one how big it is. The adobe ui has three parts, the dark grey dock, the light grey group, and each individual panel with an associated tab to access it. In the old version I was treating each component manually but in the more recent one you only have to meddle with the panels themselves and the layout manager does all the rest. The groups and docks are all created and maintained automatically as well as removing themselves when they are empty. Writing the Drag n Drop was a bit of a pain, since you have to like implement 6 classes just to make it work, but that's all kept in one class and all you have to do when you want to activate it is add a GestureListener to the component you want to move and a DropTarget to the component you want to drop something on and then create a new condition in the Drag n Drop class. In general it's just a matter of removing a component from one parent and pasting it in another, but say for moving panels which is done indirectly through the tabs, you have to do a bit hierarchy walking to get the panel and then move it as the tabs are also created dynamically. I did try using a JTabbedPane but I found it was way more work, overriding everything than just writing a fresh one from scratch.

But basically all you have to do is set the id numbers in a panel (Dock Id, Dock Bar Id, Group Id, Panel Id) and the layout manager should the rest, also enabling/disabling a panel adds/removes it from the layout. Makes it very easy to save settings too :)
I don't quite remember when I started, but I'd say it took the better part of a week to do since I've only really been working on the project in general for just over a week now. I'd have to say a lot of the Java APIs made this job way easier than what it was, particularly the ArrayList class which is like my favorite class at the moment. I haven't done very much C, but I'd really hate having to do stuff without the ready made APIs, it WOULD take months O_O

I'm still considering using my own graphics for UI since the current ones are just crappy screenshots of originals without the nice lovely soft transparent edges etc, but I'm not the best at creating that professional look in photoshop, so it might still be a while before I do make any changes :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 07, 2010, 11:28:13 am
The LZ77 decompressor didn't work for me. Is the output file supposed to be in a certain format? The program just closes.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 07, 2010, 11:39:30 am
Its specifically for the NDS version. The first byte in the file should either be 0x10 or 0x11 iirc, and I also think I implemented it to open files with the LZ77 magic stamp as well (found in some NDS games ie Super Mario 64 DS).

The program will just close if it doesn't know how to open a file, but I was pretty sure the auto decompressor occurs before that point though so there should still be a newly created file in the folder for any valid LZ77 compressed files.

Actually now that I'm actually going back over the code, RLU compression should also be available too btw @_@ And also it definitely should decompress the file before closing...
You can always PM me the file and I'll have an look and see what's wrong :)
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on November 07, 2010, 12:04:39 pm
Can I double-check something with you real quick?

0x10 means the old GBA-style decompression, where every compression code is two bytes long, the first nibble is the length (minus three, i.e. add 3 to get the actual length) and the rest is the distance (minus 1, IIRC), right? And 0x11 means that the compression codes can be up to four bytes long and the length part can represent a sixteen-bit digit?

'Cause I thought I was treading uncharted ground with my current project until you mentioned those two numbers. (And lemme guess: 0x30 denotes RLE compression?)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 07, 2010, 12:21:24 pm
Wells I'm not totally certain how they relate to LZ77 compression on GBA's but that sounds basically along the right lines. I'm not a real expert on compressions but I'd assume they ought to be the same or at least similar, you should look at Garyong's source code (http://code.google.com/p/dsdecmp/) :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 07, 2010, 08:56:00 pm
Its specifically for the NDS version. The first byte in the file should either be 0x10 or 0x11 iirc, and I also think I implemented it to open files with the LZ77 magic stamp as well (found in some NDS games ie Super Mario 64 DS).

The program will just close if it doesn't know how to open a file, but I was pretty sure the auto decompressor occurs before that point though so there should still be a newly created file in the folder for any valid LZ77 compressed files.

Actually now that I'm actually going back over the code, RLU compression should also be available too btw @_@ And also it definitely should decompress the file before closing...
You can always PM me the file and I'll have an look and see what's wrong :)
I sent you the files with this problem before. They were overlay files. Apparently not all of those files are compressed, but the ones that are compressed always have problems. I just tested them with DSDecmp and some decompress larger than they should be or not at all. I know for sure compressed overlay files begin after overlay 0671 and there are dozens of compressed files after that. ConsoleTool Mini couldn't decompress any more than DSDecmp either. And those files begin with 00 00. I don't know if they even are LZ77 compression.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 07, 2010, 09:15:06 pm
Oh right, I forgot >_<

I'd say none of them are compressed as just scanning through the files there was also a RLU compressed file in there as well, you generally don't see a mixture of compression formats in batched files. It just happens to be that some of these overlays fit the header criteria of compressed files. The dead giveaway that they aren't is the massive arse gap between the compressed file and the decompressed file size. If it's anything over like a 10x increase or something, it's doubtful they were meant to be decompressed, 1.4KB > 4 MB is kind of pushing it I think :p

Without knowing the actual file structure of an overlay file though it's kind of hard to say anything for sure :p I have tried filtering the compression criteria a little but in doing so you might actually miss the odd file that actually IS compressed heavily so its a bit iffy :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 07, 2010, 09:53:44 pm
Well I stopped getting decompressed files at overlay 0769. There should be more since these files are ports of files from the GBA game. Overlays 0971 past 0769 are the game's main sprite files which use a game-specific file format. I've already got that part covered and I know there could be more files. I'm sure the overlays could just contain any graphic or text for the game which I know is inconvenient. Maybe there is a pattern of sorts?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 07, 2010, 10:08:33 pm
Were these overlays pulled from an archive file? Easiest way to look for a pattern is to have them all in a single file and look at it using a combination of tile molester and a hex editor. Going through 1114 files that have no obvious file structure would be a nightmare. There also ought to be some kind of index file somewhere which might shed some light on your problem. What game are these from anyways?
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 07, 2010, 10:12:09 pm
Yes, these did come from an archive. I got them from Rockman EXE Operate Shooting Star. When I looked through the game with a tile viewer, I figured out that all sprites were arbitrarily compressed since there are 1 or 2 types of sprites uncompressed for no reason.
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on November 07, 2010, 10:15:22 pm
What exactly is in the "overlay" folder, anyway? I use NDSTool to extract DS ROMs into a directory structure and I always get two subfolders: "data" and "overlay". The "data" folder usually has the scripts and graphics, but "overlay" has a collection of BIN files of uncertain provenance. Are the files in the "overlay" folder usually graphics of some kind?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 07, 2010, 10:56:49 pm
@Koetsu
Sprites and text are typically the two file types most likely to get compression since they are typically the bigger files when compared to their counter parts.

@Ryusui
God it's been a while since I've used NDSTool...Okay now I'm on track with what Koetsu is trying to do. I'd assume "overlays" have a specific use much like how there's always a utility.bin file in the root folder of every game. Though I thought much of this stuff was all generic (built by the SDK) and all the game data was kept in the files located via the fat. If this isn't the case and there are custom files in other areas of the rom then that's annoying -_- Will have a look more into this when I go over the NDS rom format :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 07, 2010, 11:16:03 pm
Yeah. In the previous game port, all the files were put into their own archive. This time, it seems the files in cmn and jp are probably screens and text while overlay is everything else like maps and sprites.
Title: Re: The Console Tool (by Low Lines)
Post by: MegamanX on November 08, 2010, 02:13:14 am
I don't want to take this that far off-topic, (if I am) but I unpacked a DS rom recently and the overlay folder is empty. 

I have a data folder with nothing but sub-folders with all the graphics from the game and then the empty overlay folder.  Aside from that, there's a few BIN files not in either folder (arm7, arm9, y7, y9, banner, and header).  That normal?  Haven't messed with other DS roms before so I don't have much clue yet.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 08, 2010, 02:26:09 am
No I think that's normal, I would think that when a developer creates a new project, it automatically builds all the default settings/folders etc, but as each developer would have their own way of doing things, not all parts of it might get used, it'd be like having 4 USB slots on your computer but you only ever use 2 of them. As for the bin files, those are straight dumps of blocks of data from the header of an NDS rom, these hold stuff like the actual game code and information used for identifying your game ie when you load in a game into your DS it shows up with an icon and the name of the game.
Title: Re: The Console Tool (by Low Lines)
Post by: MegamanX on November 08, 2010, 02:35:28 am
Thanks for the quick reply.  Appreciate it.

I'm trying to learn a bit about DS roms here because there's only one game I give a hoot about.  As of right now, I can change all the graphics and repack it and play it.  Unfortunetly, it's slowly looking as if it won't be an easy task.

Thanks again.
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on November 08, 2010, 02:57:14 am
You won't have to fiddle with the arm* files unless you plan on actually hacking the game's code. As for the directory structure, it can work with you or it can work against you. Right now, I'm working on a project where all the script files were neatly grouped in a single folder called...drumroll, please..."script". Mind you, almost all the other resources are in folders with bizarre abbreviations, so it'll take a bit more effort to find the pics that need modification. Adding complexity to matters is the fact that everything is wrapped up in proprietary (and compressed!) archive files. Fortunately, I've already written up a tool for cracking open those archive files. Recompressing them is another matter, but the neat thing is, I can reinsert files without compression if I so choose. Ryusui: 1, tricky programmers: 0.
Title: Re: The Console Tool (by Low Lines)
Post by: MegamanX on November 08, 2010, 03:20:14 am
Good to know I can eliminate something from anything I want to do.

I WISH I had a script folder.  Just about everything has it's own sub-folder.  Armor, battle, bg, enemy, saiga, weapon, world, etc.  I found the font because there's a font folder.  That was the first thing I edited just to fool around and the edit appeared when I repacked and played.  Just wish there was a script folder.  Haha.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 12, 2010, 02:59:51 pm
As it turns out, adding support for drag and drop rendered a transferhandler class I got off the net for handling drop and open (files off your desktop) useless so a bit of time was spent firstly trying to figure out how the whole DND system worked and as to how and where the file drop was located. A bit hectic but it's all working and I've found and fixed a few more bugs in there drag and drop interface :)

The next step was transferring the palette editor from the mini version, which is mostly just a copy and paste (hooray) but I've still got to build a few editor panels etc.
The main focus for most of this week has been the hex editor, which is kind of just a "viewer" at the moment. Displaying the file data is pretty straight forward but for the rest of the time I was playing around with the APIs to see what was achievable in terms of text highlighting etc. There's a built api for adding custom highlighting in which you specify a start and end position within the text as well as a painter, however as far as I know this only works if you want to apply different background colors for blocks of text. What I was also wanting to do is set different foreground (text colors) as well, but the best thing I've found so far is something called the AttributedString class, which allows you to set attributes to blocks of text (ie font, size, color, background, etc) however it can only be used when or if you are painting text onto the screen. Basically it means that you have to write your own custom paint class that firstly simulates all the default stuff of your normal textcomponent, plus applying the extra attributes. The rendered text also does not match the same smoothness of the text fields which ruled out a the idea of simply drawing over the text. I'm also having some issues filtering the text selection also, hence why atm it is just a hex viewer and not a hex editor. But other than that, the UI works pretty well, which is based on Hex Workshop if anyone has used it before. I'm going to put it aside and come back to it later once I've studied up on JTextComponents a little more :p

(http://img201.imageshack.us/img201/6918/ctool3hex.png)

Oh and I also got swapping the order of tabs in the panels working...You'd be surprised at how satisfying it is to be able to drag the tabs around (like in photoshop) and it just reorders them for you :)

[edit]

Cleaned up the UI a bit as well as fixed a bunch more bugs I've found and have made a start on the palette/tile image editors. The major milestone for this point is that I've got the assets working, which manages linking files together (ignore the empty Assets panel, I forgot to remove it). It allows you to change say the selected palette for an image to any other palette that is currently opened, and will load the last opened palette if the selected palette has been closed. It should also allow for live updating too :)

(http://img201.imageshack.us/img201/258/ctool3tile.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 16, 2010, 12:05:46 am
Aren't those Pokemon sprites animated? Is it specified in the format?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 16, 2010, 11:28:06 am
@Koetsu
There are four different sprites for each Pokemon, two for the front and two for the back, one being the animated sprite while the other is solid single cell sprite like how they were done in older DS Pokemon games.

@Update
The palette editor has been added with improvements on the version from the Mini app I posted a few weeks ago, namely the ability to drag and drop colors effortlessly (which took MANY hours to get working right), as well as input values either using the RGB textfields or as a hexidecimal value. When you switch between tabs, you can instantly see the changes made in the graphics, which is great up until the point where you are switching tabs constantly as it gets annoying. I haven't implemented the windowed mode yet, so tab swapping will have to do for now :p

I've been a little concerned about how CPU intensive the app performs when messing with UIs such as the color sliders and general component resizing. At he very least it doesn't cause the window to display "not responding" as does photoshop or flash if you play with the sliders too much :p Once windowed mode is supported it will probably be even more CPU intensive with each window watching other windows for changes :[ Interestingly though I realized that the mini version I posted actually uses a constant 40% of my CPU when displaying a palette (reminiscent to the old 3D renderer from the first version), but at least now it only knocks up CPU with dragging events.

(http://img87.imageshack.us/img87/5923/ctool3pal.png)

Next up, the tile/graphic editor :)

[edit]

It turns out you can't always view NCGR's on their own as some have been packed in a way that if you don't have the associated NSCR or NCER it won't display properly, particularly with files that have a 257 tiled flag byte. So instead I've gotten the NCER working. My notes on decoding the cell data was definitely overcomplicated as changing the x/y byte to signed (see Garyong's notes (http://www.romhacking.net/forum/index.php/topic,8407.msg170466.html#msg170466)) positions cells about 4px more centered than what I had achieved, though it still doesn't fix the gap with the new Pokemon sprites yet. Am still in the process of going over it, but the nostalgia is hitting again so I thought I might put up a picy of Coronamon (which was like one of the first sprites I was working on back when this project started for those who don't know). I've ditched the grid UI from the old version in favor of a blend between the Adobe Rulers and the Flash scene scrollPane (where you start in the middle of a scrollable area).

(http://img87.imageshack.us/img87/3636/ctool3cell.png)
Title: Re: The Console Tool (by Low Lines)
Post by: MegamanX on November 17, 2010, 04:23:35 am
I downloaded the consoleToolMini in your sig, just to look at it because I have free time, and when I finally was able to get it running, everytime I drag a DS rom onto it, it just closes.  I have no idea why that is. 

Can anyone help me out?  This is the first java thing I've actually ran that wasn't on some website so I'm really clueless.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 17, 2010, 06:43:48 am
@MegamanX
It opens four different palette formats and also NCGR, plus should automatically extract compressed files.

The Mini version was incredibly limited as to what it could do, and has mostly been an experiment to get a few concepts figured out (such as exporting/converting formats) that I was having troubles with. The complexity of the UI would also be (I think) alot more work to write code for. As of yet, I have not been posting regular builds so any new stuff I've been doing is still private. Sorry for any confusion :s

The first version did the most stuff, and would also open files as hex (iirc) if it doesn't know the file structure.
The second version focused on constructing the Adobe-styled UI as well as 3D rendering as well as a more user friendly file browser.
The third version (which was the mini) is as I stated above, very basic, and although was intended as a quicker way to put on file support, ended up just been an experiment (which has honestly helped clear my head so to speak). First version to support the export feature as well as a more structured file assets management.

The current (next) version I'm working on aims to do all of what I have achieved so far in terms of functionality, I also intend to start releasing progressive builds once I have a certain amount of stuff implemented, namely 2D Graphic and Archive support.

Current Functionality
- Drag & Drop UI (missing dock layout, minimize support, display options)
- Drop file opener
- Basic file browser (Mini version browser)
- Hex Editor (viewer only)
- Support for NDS LZ77/LZSS and RLUncomp compression
- Full support for editing 4 different palette formats (NCLR, Adobe Color Table ACT/RAW, Microsoft Palette, Usenti Palette)
- Partial support for NCGR
- Partial support for NCER

Current Milestones
- Finish Drag & Drop UI
- Internal Frames / windowed mode (not a priority)
- Right-Click Popup Support
- Advanced file browser (not a priority)
- Hex Editor (not a priority)
- Support for Graphic Editing/Exporting
- Support for 2D Maps (NSCR)
- Support for Cell/Sprites (editor is not a priority since there aren't any formats to covert to)
- Support for Cell/Sprite Animations
- Support for Archive browsing/editing (editing not a priority though)
- Support for 3D formats (not a priority)

So hopefully that's all very clear now for anyone who is wondering :)
Title: Re: The Console Tool (by Low Lines)
Post by: MegamanX on November 17, 2010, 07:43:05 am
For me, nothing would happen.  The program literally closed.  Nothing new would happen and I would have to relaunch it.  So I wasn't able to do anything with it.  Haha.

I can't wait for the full thing like everyone else that has posted.
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on November 17, 2010, 08:08:33 am
I'm impressed you've managed to do so much already, once those features are added I don't think there'll be a need for any other ds hacking tools apart from this one
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 17, 2010, 12:05:42 pm
Were you able to open those compressed overlay files? It turns out CrystalTile can actually decompress those just fine.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 17, 2010, 08:03:51 pm
@Rhys
...What DS hacking tools?? I was under the impression there was a general lack of such things, and of what there is, it's generally only focusing on one thing...Plus most of the tools I know of were like one-offs that haven't been updated in a while.

@Koetsu
I'm focusing on the documented formats atm.

Cleaned up the Assets code a heck of a lot, using lots of convenience methods to cut back on repeated lines, and have also enabled the formats to be opened without requiring all their assets to display. Still trying to figure out the NCER cells. I have gone over the notes Garyong posted about 5 pages back and added mirroring and palette indexing. Am having troubles extracting sub image coordinates from the tileOffset on some cells. It seems that any offsets in the 128+, 128+ location of the image use a different format...otherwise I've been getting the x and y from the first 10 bits (5 bits each) and the multiplying that by 8.

(http://img809.imageshack.us/img809/6706/ctool3cellerror.png)

Cell 2 has a value of 0x1F1 and is located at 128,128
Cell 3 has a value of 0x219 and is located at 192,128

This is a sprite from Super Princess Peach in case anyone was wondering.

Here are my notes on specifically the cell data structure. I've written them in the same format as Garyong's so its easier to compare.

Code: [Select]
s8 ypos
u8 flags1
s8 xpos
u8 flags2
u16 offsets

*flags1 WW????FF
W = dimension index 1
F = vertical flag?

*flags2 HHYX??FF
H = dimension index 2
F = horizontal flag
**if xpos <=-64 and F=0 add 256 to xpos
**if xpos >=64 and F=1 substract 256 from xpos

*offsets PPPPPTTTTTTTTTTT
P = palette offset
T = tile offset / sub image coordinates
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on November 17, 2010, 08:29:55 pm
I remember there being a couple of tools here and there to extract sound archives and nitro archives, and one application that let you browse a DS rom and explore archives and filetypes and replace files, afaik this is what zero day hackers used for the DS Pokemon games. There's some Pokemon-specific tools too.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 19, 2010, 10:58:10 pm
Been a bit busy with other things so I've been off and on working on this, but never fear :)

(http://img221.imageshack.us/img221/3888/ctool3anim.png)

And yes the playback buttons work and don't crash like the old version. It's the first thing in the program to make use of a Thread class which although I understand, I really haven't been shown properly how to use yet. From what I understand it's a core aspect of one of the next units I'll be doing next semester at uni.

A lot of focus was actually on improving the assets management which still had a few bugs that needed ironing out. As far as I can tell, it works without issues now, at least for these four data types I've implemented into the program.

Also going back to the Animation Viewer, I've also included a copy function which allows you to copy the image the clipboard so you can paste it in Paint or whatever. The only downside is you lose transparency as the image is exported to RGB and as far as I know that's a java thing and not something I can change much. At the moment you can copy an image by right-clicking on the image and selecting copy, but I will eventually include a toolbar option as well as implement it on the other graphical data types too :p Until I add support for web images (PNG, GIF, etc), that's as far as image exporting will go...but at least that ought to be better than screenshots :p
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 19, 2010, 11:26:18 pm
Ooh. I like the Photoshop-style animation strip. How are the timers calculated? Does it involve the 60fps and the value specified? And it looks like the palette of the sheet can be directly edited. Is this also possible?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 19, 2010, 11:35:16 pm
Ooh. I like the Photoshop-style animation strip. How are the timers calculated? Does it involve the 60fps and the value specified? And it looks like the palette of the sheet can be directly edited. Is this also possible?
Each data type is treated as a separate thing or file in this case so you can easily make changes and see the difference since the program will reload themselves with any changes applied to assets. Palette editing is the only fully editable thing at the moment with everything else just being in viewer mode.

The timers at the moment go on the assumption that the timer value stored in NANRs are n/60 fps (which is the number frames the DS updates per second), however after looking through a few files, its probably likely there's a second variable that tells you the divider amount as well.

On the Adobe-styling, I actually got a bit peeved when I tried dragging a panel to the top or bottom of the screen in Photoshop and discovered I couldn't do it, it seems docking on all four sides is only utilized in Flash, but still...was a bit of a shock you know :p
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on November 20, 2010, 06:10:21 am
Wow that UI looks incredible, well done! :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on November 21, 2010, 07:16:58 am
Well done on the GUI indeed.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 21, 2010, 11:41:42 am
Had a go at adding support for NCERs with a Bank Type 1 (aka Extended/Partitioned Cells in Garyong's terms), with success it seems. There were some issues getting the right offset addresses with cells however I'm pretty certain they're fixed now. Also as of yet, I've only come across one sprite sheet (NCLR+NCGR+NCER+NANR) with an 8 bit pixel depth, so it's probably likely there will be bugs with some sprites as I've only taken it into account for graphics that are partitioned. Digimon Championship is the game I've been testing when adding support for extended cells.

(http://img225.imageshack.us/img225/5757/ctool3excell.png)

One thing I noticed however (not mentioned in Garyong's notes, unless I missed it) is that the partition section of the NCER seems to have an 8 byte header that makes up two 4 byte values. The first value looks like it is possibly the maximum size of a partition, whereas the second value always seems to be 8. There also seems to be some irregular use of 0xCCCC in both NCER and NANRs as some kind of break/padder, what its significance is I would have no clue.
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on November 21, 2010, 03:45:15 pm
It's quite interesting that the DS has standard file type for pretty much everything a DS Developer could want, it's something that competitor platforms haven't really done too much outside of basic naming conventions for executables and a specific directory structure (the PSP/PS3 comes to mind).

It makes emulating these platforms on future platforms for backwards compatibility quite interesting, as the hypervisor/emulator could analyse the game's code for library calls for accessing data from these formats and swap it out for code much more optimised for the new platform, or even for the case of the ARM7 binaries replace it altogether for games that ship with nintendo's binary (afaik all retail games still do this).

It also makes translating DS games much easier as if a game uses a known format that can be un/repacked then translating the game should be fairly straightforward with little to no ASM hacking required. It'll be interesting to see if other platforms adopt this approach in the future.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 21, 2010, 09:45:29 pm
Going a little off-track from my milestones, I've been looking at the NMCR and NMAR files found in newer games, such as those in the new Pokemon DS games.

(http://img153.imageshack.us/img153/8828/ctool3pokeanim.png)

Now this above picture may look like crap, but I think, that if I had the cell gap error fixed, it would be displaying properly. After getting this far, I'm almost certain NMCRs are like a second cell mapping file that positions each animated limb (bank in a NANR file) with some offsetting added to the positioning. In the case of the Pokemon sprites there are two cell banks, which make up the two frames in the NMAR animation. As far as I can tell, there's no rotation information stored in any of these files, which leaves me to assume that its kept in next file following the NMAR.

There's also not enough data in the NMCR to allow for fixing the cell gap, so the real question is, if the cell gap is dependent on the NCER, how is it meant to be fixed XD
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 22, 2010, 04:49:00 am
The rotation data is stored in the NANR file, in the Frame Data field. I'm assuming you've already figured out some of the Frame Data format, since you've shown an animation loaded into the tool, but these are my notes on NANR so far (ignoring the generic file header and LABL and UEXT sections);
Code: [Select]
/*
 * ABNK; Animation Bank
 * <NitroSection("KNBA", sectionSize)>
 * u16 numAnimations;
 * u16 numFrames; // total number of frames. not really necessary, since they have various sizes and can only be read once the format is known from the animation.
 * u32 animationOffset; // relative to section start + 0x8
 * u32 frameRefsOffset; // relative to section start + 0x8
 * u32 frameDataOffset; // relative to section start + 0x8
 * u32 padding; // is it?
 * u32 padding; // is it?
 * Animation* animations;
 * u8* padding;
 * FrameRef* frameRefs;
 * u8* padding;
 * FrameData* frameData;
 */
/*
 * Animation:
 * u32 numFrames;
 * u16 dataType; // Type of FrameData used for this animation. 0: 2 (4) bytes. 1: 16 bytes. 2: 8 bytes
 * u16 unknownConst; // always 1 it seems
 * u32 unknownConst; // always 2 it seems
 * u32 frameRefOffset; // offset of first frameRef, relative to start of frameRefs.
 */
/*
 * FrameRef: (frame reference/pointer)
 * u32 frameDataOffset; // relative to start of frameData.
 * u16 delay; // duration in #frames. ~= (1/60)s?
 * u16 beef; // always 0xBEEF?
 */
/*
 * FrameData:
 * - Type 0:
 * u16 cellIdx; // cell of the NCER to display
 * u16 garbage; // if this is the last frame data for an animation, this is 0xCCCC. Otherwise this is the first u16 of the next FrameData. Just ignore it.
 * -------
 * - Type 1:
 * u16 cellIdx;
 * v16[5] transform;
 * s16 xDisp; // displacement of cell in x-direction
 * s16 yDisp; // displacement of cell in y-direction
 *
 * The transform is usually of this format:
 * {B, A, C, D, E}, with:
 * E = 0
 * A,B,C,D as used in the formula in the bottom of this section in GBATEK: http://nocash.emubase.de/gbatek.htm#lcdiobgrotationscaling
 *
 * However C and E can be 0xFFFF sometimes (pokemon B/W file 505, pikachu), which seem to be some special marker. Not sure yet what they mean.
 * -------
 * - Type 2:
 * u16 cellIdx;
 * u16 beef; // always 0xBEEF
 * s16 xDisp;
 * s16 yDisp;
 */
(the terminology is most likely different again)
A v16 is a 1.3.12 fixed-point number, as defined here (http://dev-scene.com/NDS/DOCfixed_point).
Title: Re: The Console Tool (by Low Lines)
Post by: MegamanX on November 22, 2010, 05:05:16 am
Will this support .ob, .pb, .ab, .nbfc, .nbfs, .nbfp formats?

Curious because the game I'm looking at right now uses those formats.
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 22, 2010, 06:25:43 am
Will this support .ob, .pb, .ab, .nbfc, .nbfs, .nbfp formats?

Curious because the game I'm looking at right now uses those formats.
Extensions are only weak indicators of the content of files. I know the .nbf* formats should be just raw graphics, tile maps and palettes, which could be supported. I think it's unlikely the others will be supported however, unless you can provide example files (or mention the game you're looking at) ;].


By the way, I've unfortunately not yet had the opportunity to test the GBATEK formula for 2D tranformation. However the expected results of applying the formula to the values in the pokemon files match the actual animation too closely for it not to be a coincidence.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 22, 2010, 09:21:32 am
Yeah it's a colossal amount of data that needs to be processed to get the end result, I mean the complete animation of one sprite is made up of 6 files at least.

I actually had the NANR bank types in my implementation (excluding knowing what the extra data did), however I must have thought it was wrong and removed it. Silly me :p

At the moment I've been trying to figure out the cell gap problem, and my current assumption is that if the vflag == 3 an offset is added based on the x & y coordinates, much like how with the Digimon World sprites (which I think are vflag == 2) add/subtract 256 if the x coordinate is <=-64 or >=64. I can get sprites to display within about an 80% accuracy but I'm doing it the long way so it's sort of trial and error atm. The key thing that helps support my assumption is that when you load up everything up to the NMCR file, the body parts line up better with the feet generally just above to horizontal center point (red line in my gui).

@MegamanX
I'll add support for any formats with a clear indication of what they do (ie structured header, documented, etc), but my priority is getting the more generic ones supported first, specifically right now I want to have all the Nitro and Nitro3D formats supported. At the moment, it can open NCLR (as well as the other stated palettes), NCGR, NCER, NANR, NSCR, NMCR and NMAR with varying levels of functionality, with the most supported being the first 5. As far as my current milestones go, I've only got to add support for NARCs and probably DS Roms, plus finish up the drag and drop ui, before I'm going to be happy releasing a build so people can start testing it :p  Though as I've said I've kind of sidetracked at the moment trying to fix errors in the NCER and NANR, depending on how they go I'll get stuck into the rest later.

[edit]

Not having much luck with the cell gap bug. There is almost always two conflicting cells that if you fix one, it breaks the other, so perhaps there could be another variable involved, such as the u16 value in the bank header? The issue though with that is that other NCERs have values stored here also.
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 22, 2010, 07:14:16 pm
I'm quite positive the 'cell gap' (=the gap between OAMs in a Cell) is inherent in the NCER files, and should thus somehow be mended in the NCMR/NMAR files (I'm guessing with position scaling). Using more GBATEK specifications, I believe I've nearly got the complete NCER format (there's only two unknown values left);
Code: [Select]
/* CEBK section:
 *
 * <NitroSection("KBEC", sectionSize)>
 * u16 numCells;             // number of cells defined in this CEBK
 * u16 extendedCells;        // if 1, cell data is CellDataEX instead of CellData
 * u32 cellDataOffset;       // relative to section start + 8
 * u32 flags;                // 2 LSBs: left-shift all tile indices this much.
 *                           // 3rd LSB: if 1, use `sub-images' instead of normal
 *                           //           addressing mode
 * u32 partitionDataOffset;  // offset to partitioning data header.
 *                           // partitioning data is only present if this is non-0
 * u32 padding;
 * u32 padding;
 * CellData[numCells] cellData; // is CellDataEX[numCells] if extendedCells == 1
 * // no padding in-between; cell data is properly aligned
 * OAMData* oamData;
 * u8* padding;              // may be empty
 * // --- partition data header starts here. is not present if partitionDataOffset==0
 * u32 maxPartitionSize;     // not sure; could be something else.
 * u32 headerSize;           // partition header size, offset of first partition
 *                           // relative to this section, or just constant 8
 * PartitionData[numCells] partitions;
 */

/* CellData:
 *
 * u16 numOAMs;
 * u16 unknown;
 * u32 firstOAMOffset; // relative to first OAMData
 */
/* CellDataEX:
 *
 * <CellData>
 * s16 xMax;
 * s16 yMax;
 * s16 xMin;
 * s16 yMin;
 */

/* OAMData:
 *
 * u16 OBJAttribute0;
 * u16 OBJAttribute1;
 * u16 OBJAttribute2;
 * (see also GBATEK, http://nocash.emubase.de/gbatek.htm#lcdobjoamattributes)
 *
 * Attribute0: ABCDEFGH IJKLMNOP
 * IJKLMNOP : Y position (signed, range [-128,127])
 *        H : R/S flag (Rotation/Scaling flag)
 *   if R/S flag = 0:
 *        G : OBJ Disable. (if 1, do not display) ignore.
 *   else
 *        G : Double-Size flag. (if 1, double-sized display reactangle) ignore, unless some
 *                developer purposefully used a larger/smaller clipping rectangle for displaying
 *                the rotated/scaled sprites properly for some obscure reason.
 *   endif
 *       EF : OBJ mode. 00 -> normal.
 *                      01 -> semi-transparent.
 *                      10 -> 'OBJ Window' (some sort of masking; see GBATEK).
 *                      11 -> Invalid.
 *                      Should be 00 for NCER values.
 *        D : Mosaic flag. Should not be set in NCER.
 *        C : Color Depth. 0 -> 16 colors, 1 -> 256 colors. (can technically be ignored, since the depth is also known from NCGR/NCLR file)
 *       AB : OBJ Shape. 00 -> square
 *                       01 -> horizontal
 *                       10 -> vertical
 *                       11 -> Invalid.
 *                       
 * Attribute1: ABCDEFGH IJKLMNOP
 * HIJKLMNOP : X position (signed, range [-256, 255]. read as unsigned, subtract 0x0200 if X>=0x0100)
 *   if R/S flag = 0:
 *       EFG : unused.
 *         D : Horizontal flip.
 *         C : Vertical flip.
 *   else:
 *     CDEFG : R/S Parameter selection. Should not be set in NCER.
 *   endif
 *        AB : OBJ Size. (resulting OAM size depends on this and OBJ Shape from Attr0)
 *       
 * Attribute2: ABCDEFGH IJKLMNOP
 * GHIJKLMNOP : Tile index
 *         EF : Priority relative to BG (usually not set in NCER, but sometimes is)
 *       ABCD : Palette index
 *       
 *
 * OAM size table:
 *
 * Size  Square   Horizontal  Vertical
 * 0     8x8      16x8        8x16
 * 1     16x16    32x8        8x32
 * 2     32x32    32x16       16x32
 * 3     64x64    64x32       32x64
 *
 */

/* PartitionData:
 *
 * u32 start;   // (in pixels)
 * u32 length;  // (in pixels)
 */
I find it unlikely (although not impossible) the unknown value in the CellData can fix the gap, since values at 'broken' cells also appear for non-'broken' cells. (was this the value you mentioned? I'm interpreting 'bank header' as the CEBK section header however, since the CEBK is a singular Cell Bank / Bank of Cells ;])
The other unknown value has to do with the partitions, so that's certainly not the fix-it-all value.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 22, 2010, 08:28:43 pm
My terminology sucks, in fact I'm going to go and fix all my naming right now. In MY terms, a cell is a bank and an OAM is a cell..Whenever I first started my notes, that's how I put it down and that's how it's kind of stuck >_<

You are clearly much better at figuring this stuff out than me, and it didn't even occur to me to look at GBATek :p

Just out of curiosity have you built a viewer along with your specs?
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 23, 2010, 08:58:57 am
My terminology sucks, in fact I'm going to go and fix all my naming right now. In MY terms, a cell is a bank and an OAM is a cell..Whenever I first started my notes, that's how I put it down and that's how it's kind of stuck >_<
Heh, I know the feeling. I probably still have some mis-named comments somewhere in my code as well; rarely anybody get them right the first time. =)

Quote
You are clearly much better at figuring this stuff out than me, and it didn't even occur to me to look at GBATek :p
I probably only have more experience at these things ;]
I started looking at DSTEK/GBATEK for the capabilities the DS/GBA have for rotating sprites, when I also noticed these very familiar format. Also, since these Nitro formats are built-in, it's more likely the Nitro formats use data structures the DS/GBA need in their memory anyway.

Quote
Just out of curiosity have you built a viewer along with your specs?
Yes, although it only displays NCER+NCGR+NCLR or NSCR+NCGR+NCLR triplets. It's a Windows-only viewer (http://www.propl.nl/random/GlycerinMapViewer_dev.zip), partly because I dislike designing Java UIs, partly because I like the binary data handling of C# better.
I've not yet made it open-source, because I wanted it to support more formats before I did (and currently only the recently re-written NCER class is not mostly ugly code). The included Glycerin.dll should be accessible by other .NET apps though, which contains the file format classes.
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 23, 2010, 12:34:44 pm
Well, sir, Low Lines, I must say your app is the best game explorer I've seen. Especially thanks to the cross-platform-ness.

Now, my only problem is that whenever I open this and start saerching, even just in the /browser/, the task uses up 98% of my dual processors even though my RAM is only at 20% (like it is normally).
I'm not sure if my computer is acting all shitty, or your app is too big, but it'd be great to have this working without bugs/ huge lag.

Also, does the extract button actually work or was the app lagging too much for the dialog to even OPEN.

Lastly, .nsbca type animations don't seem to load. Again, I'm not sure if it's computer doing this or whatever.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 23, 2010, 12:36:39 pm
Which version are you referring to?
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 23, 2010, 12:42:44 pm
Which version are you referring to?

Well, the demo 1 and 3 .zip files from your sig apparently couldn't be opened by 7-zip, so I guess I'm using Demo #2.

BTW, kudos on the fast reply.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 23, 2010, 01:05:05 pm
Demo 1&2 were designed with the file browser at the core, so whenever you start the program, it first scans your system to get the drives and the contents of the top level folder in each. Also due to the way I programmed it, each file reference that is created, also is scanned for file type matchups (as in reading the actual header for magics, compression, etc), which can also create laggyness. The only reason for a very high cpu usage like that is some kind of routine looping constantly and as far as I remember, only the older (demo 1) model viewer does that.

On my system, I think my hard drive is badly fragmented or something, because it takes a colossal amount of time to load up the browser now (which it didn't originally), though I also have networked drives including ones connected to very slow computers. These are all issues I want to address when I eventually redo the file browser, which in the current version, just uses the default browser dialog created in java (looks quite nasty and doesn't blend well with system look and feels). Threading is one thing I have not used at all when file browsing, which would probably dramatically resolve alot of the lagging issues.

As for the NSBCA, I got up to the point where I could fully understand the file structure however I had no way to assign assets in the older versions. Its on my list of things to do though.

All demo zips are working, perhaps they didn't download properly for you?
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 23, 2010, 01:41:42 pm
Well, I redownloaded Demo 3, and apparently, 3rd time's a charm.

This one doesn't take up all my CPU, but it doesn't work. I've tried dragging and dropping, opening with the app... It always seems to just stop working afterwards.
I have the latest java platform, don't worry about that.

I'll try and redownload all of them and try each, see if the CPU usage was just a download problem.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 23, 2010, 01:45:46 pm
Demo 3 only handles a a few of the 2D Nitro formats (was an experiment to help improve my class design), otherwise it just closes. Redownloading also won't fix any CPU issues.
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 23, 2010, 01:55:01 pm
Surprisingly enough, it did !

The interface has a different look, and it's a whole lot better. Actually I think I was using Demo 1, hence all the problems.

I have two things to point out though, when viewing a model, I can't seem to zoom, is there a control for it ? Secondly, the "Save as..." option doesn't work. Is it because it hasn't been implemented yet ?
See, I could extract the models with a printscreen ripper, but I don't want to lose bone/animation info.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 23, 2010, 02:10:37 pm
Exporting hasn't been implemented, and to be honest, just getting the damn things to display properly is an achievement in itself :p

3D controls are done with a three button mouse...
left - pivot/rotate
right - move center point (up/down)
middle (hold and swipe left/right) - scale

From what I've been told, the ram ripping programs don't work on the demo 2 model viewer due to the way it renders stuff.

I swear though...the moment my app is able to export DS models, I'm going to be in for a world of hurt @_@
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 23, 2010, 03:34:55 pm
What do you mean ? From the moment it's able to export, then it should be smooth sailing, am I wrong ?

BTW, how do I make it render textures ? It's just a white mesh, in it's current state.

Also, maybe I didn't say this enough, but bear in mind that your game exploring app is the absolute best, and no other comes close to rivaling with it.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 23, 2010, 03:56:18 pm
Wells with more support, comes more users, and from what I can tell, hacking 3D models is a big want atm, so this thread will inevitably get alot of the same questions asked over and over, either because people don't bother to read (which is understandable with 16+ pages), or it simply just isn't clearly answered anywhere :p I'll have to write up an FAQ too for the next version :p

With the textures, if they aren't part of the NSBMD file, they won't load. This is because I tried implementing the assets manager but then gave up (and then had a several month break from the project before releasing demo 2 without changes), otherwise it should have reverted to the last opened texture by default.

I'm sorry if I came across a bit iffy, I'm all for answering questions. The way I see it, is if people have to ask questions, that means I'm not doing my job as the designer, but also, all I want to do right now is get everything I've achieved to this point all working in the same build (which is not riddled with the leftover garbage of all previous versions), so I can remove the demos and so there isn't all this confusion as to what is and isn't working anymore. I am getting there slowly though :)
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 23, 2010, 04:13:47 pm
Yeah, I'm sorry if I posted already answered questions, but as you said, the thread is pretty long, and each of your posts are (very interesting nonetheless but still) walls of text.


You don't come across iffy at all, actually. I am truly grateful that you answer all (quite a bunch, looking back) of my questions, and with such speed, too.

I completely respect what you do as a software developer, and, unlike most fans, I can actually be patient from time to time.

I'll stop assaulting you with questions, now, cause that's one of (if not the) best way to deconcentrate the person who's making the software.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 24, 2010, 12:35:06 am
Filb appears to have links to large number of the animated pokemon sprites (forgive me if this isn't news...).
(http://www.greenchu.de/sprites/bw/025.gif)(http://www.greenchu.de/sprites/bw/495.gif)(http://www.greenchu.de/sprites/bw/491.gif)(http://www.greenchu.de/sprites/bw/646.gif)

I wonder if they ripped them manually, or if someone out there actually has a complete spec :p
After doing some random pokemon ids and getting 404s I'm guessing that these were ripped manually otherwise it makes no sense why they aren't all uploaded to the site.

@Status
Currently working on the archive UI. NARC and DS Roms are supported and load super quick since file format checking can be skipped until being called by the render component. Have had to tweak the FileAccess class a bit as for some odd reason some files return an access denied error when opened using the RandomAccess method, possibly because said files had uncommon characters (such as &) in the filenames.

@Garyong
Your app doesn't even load for me :'( Interesting name for it though :p
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 24, 2010, 05:02:56 am
Filb appears to have links to large number of the animated pokemon sprites (forgive me if this isn't news...).
(http://www.greenchu.de/sprites/bw/025.gif)(http://www.greenchu.de/sprites/bw/495.gif)(http://www.greenchu.de/sprites/bw/491.gif)(http://www.greenchu.de/sprites/bw/646.gif)

I wonder if they ripped them manually, or if someone out there actually has a complete spec :p
After doing some random pokemon ids and getting 404s I'm guessing that these were ripped manually otherwise it makes no sense why they aren't all uploaded to the site.
I find it quite unlikely they figured out the format. They probably took them from this topic (http://spriters-resource.com/community/showthread.php?tid=15517) (or a page with all the GIFs from that topic (http://dialgafans.pytalhost.de/)), or some other source of manually ripped animations.

Quote
@Garyong
Your app doesn't even load for me :'( Interesting name for it though :p
Whoops, packed the wrong .exe  :banghead:. Use this instead (http://www.propl.nl/random/GlycerinMapViewer.zip), although it doesn't contain the .XML with the documentation for the .dll (which you can copy from the other link if you want it).
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 24, 2010, 06:02:54 am
I find it quite unlikely they figured out the format. They probably took them from this topic (http://spriters-resource.com/community/showthread.php?tid=15517)

Funny you should mention tSR, that's where I plan on submitting model found through Low lines' app ! I am currently ripping levels from the wind waker games, as you can see here (http://www.spriters-resource.com/community/showthread.php?tid=15817).

BTW, are you Barubary ?, it said "by Barubary" in the glycerin map viewer. If so, then hello, fellow tSRian.
Title: Re: The Console Tool (by Low Lines)
Post by: Tauwasser on November 24, 2010, 09:18:56 am
I find it quite unlikely they figured out the format. They probably took them from this topic (http://spriters-resource.com/community/showthread.php?tid=15517) (or a page with all the GIFs from that topic (http://dialgafans.pytalhost.de/)), or some other source of manually ripped animations.

Indeed loadingNOW figures the format out and had to crack the key they are encrypted with, or so I heard. http://pokeguide.filb.de/

cYa,

Tauwasser
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 24, 2010, 11:57:04 pm
I don't believe Black & White have very many files encrypted with encryption used in the other DS Pokemon Games :p Still I think people are crazy if they're trying to rip all those animated sprites manually...I mean yeah back in the days, 151 pokemon is little work, but 649...that would take forever!!

I had a brief heart-attack yesterday as I had been backing up my files before I went to sleep and so I immediately wiped all the stuff off my hard drive (had about 250,000+ files and 8000+ folders on it) when I woke up and then realized I had also deleted the folder with my program in it. So I copy it back across from my backup disk and reload it up in Netbeans and for a while I had thought it hadn't updated all the class files I had made changes to since the last backup I did just over a week ago. There's nothing worse than getting heaps done in a short amount of time and losing it in an instant. So I'm there scanning through a Deleted File Recovery app, trying to pick out all the changed files one by one, when I then realize that the files I had changed WERE updated, just some of them I had moved into different folders in the project manager and so the old ones were still there from the previous backup...Stupid me >_<

The core parts of the Archive UI are done, the code behind it completely rewritten (and much much less code) though I quite liked the old design so I've pretty much kept it looking the same. Still have to add file extracting functionality but other than that, it does the bare minimum job for now. I want to maybe add support for different viewing modes such as like thumbnails and file filtering etc, but it's not my priority right now as I would like to move on to finishing off the drag and drop (plus fill in the menu bar with options) and then *finally* get stuck into the 3D :crazy:

(http://img152.imageshack.us/img152/8861/ctool3archive.png)

Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 25, 2010, 06:00:55 am
Still I think people are crazy if they're trying to rip all those animated sprites manually...I mean yeah back in the days, 151 pokemon is little work, but 649...that would take forever!!
You're underestimating the power of fanboyism. ;P

Quote
I had a brief heart-attack yesterday as I had been backing up my files before I went to sleep and so I immediately wiped all the stuff off my hard drive (had about 250,000+ files and 8000+ folders on it) when I woke up and then realized I had also deleted the folder with my program in it. So I copy it back across from my backup disk and reload it up in Netbeans and for a while I had thought it hadn't updated all the class files I had made changes to since the last backup I did just over a week ago. There's nothing worse than getting heaps done in a short amount of time and losing it in an instant. So I'm there scanning through a Deleted File Recovery app, trying to pick out all the changed files one by one, when I then realize that the files I had changed WERE updated, just some of them I had moved into different folders in the project manager and so the old ones were still there from the previous backup...Stupid me >_<
May I suggest some versioning system like Subversion? It'll prevent such (near-)mistakes, helps you keep track of what you've done when (if you get into the habit of writing 'proper' commit messages), and allows you to easily go back to any previous version of the tool or single files.
Assembla (http://offers.assembla.com/free-subversion-hosting/) has a free private Subversion (or git, if you prefer that over SVN) repository package available. (I'm assuming 2GB is ample space for the tool's source)


@Lexou: I am indeed =)


[EDIT]

You've probably figured most of this out already, after you've posted your preliminary specs on page 10, but there are my notes on the NMCR anyway (or rather its MCBK section);
Code: [Select]
/* MCBK format:
 * <NitroSection("KBCM", sectionSize)>
 * u16 numMappings;
 * u16 beef;             // constant 0xBEEF
 * u32 mapHeadersOffset; // relative to section start + 8
 * u32 mapDataOffset;    // relative to section start + 8
 * u32 padding;
 * u32 padding;
 * MapHeader[numMappings] maps;
 * u8* padding;          // probably empty, since sizeof(MapHeader)=8
 * MapData* data;
 */

/* MapHeader format:
 *
 * u16 numParts;        // Number of MapData's used
 * u16 numAnim;         // the number of unique NANR entries used
 * u32 firstPartOffset; // relative to mapDataOffset
 */

/* MapData format:
 * u16 animIndex; // index in NANR
 * s16 xpos;
 * s16 ypos;
 * u8 unknown;    // 'const' 0x20 or 0x21. some flags?
 * u8 priority;   // low = draw in front, high = draw in back. (I think, although it seems to coincide with
 *                // the order of the data anyway. Certainly not NCER index though ;])
 */

Posting the NMAR format would be silly, since it only has an ABNK and LABL section. The ABNK section is the same as the NANR ABNK section, but the first unknown constant in an Animation is 2 instead of 1.

This means the 'cell gap' must be fixed by the .dat, or I was wrong and the NCER/NANR actually do contain enough data to fix it. =/
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 25, 2010, 11:04:19 pm
After you posted your findings on the nCER I had some of the same assumptions with the data (particularly the priority value), however I didn't see the point in posting another 'wall of text' with 2 changes which would honestly have been guessing on my part so :p It'd better having two perspectives on something anyways before coming to a conclusion.

Does anyone know of any other games that have the NMCR/NMAR files in them? That'd be the best way to rule out if the cell gap is stored in the extra file or not. They have nitro structures so I can't see them as being custom files specifically for the pokemon games, but in saying that, it would kind of be weird to have 6/7 files all nicely structured and one without as part of your sdk if you know what I mean. I can live an hope at least :p

I've had no previous experience with versioning, nor have I been taught how to properly do it, but I've created an account, uploaded a copy and got Tortoise SVN or something on my computer so, at the very least I should have a back up from here on :p

[edit]

Just looking through the a folder in black/white, the trainer back sprites (a\0\6\4) are also done using this 7 file thing, including the raw data file, so maybe I'm wrong :p They have the cell gap too btw.

[edit2]

After going through the whole a folder in the rom again, I think I might have found the odd one out :)
In a/1/7/0 the last 7 files in the archive make up the animated sprite of the female professor displayed in the game introduction. Yes it still has the unknown raw data file, but interestingly unlike all other animated sprites (basically every animated sprite used in battles) there doesn't seem to be any cell gap, or at least as far as my viewer can tell (it only displays the first frame when viewing NMCRs atm).

(http://img507.imageshack.us/img507/4852/ctool3nogap.png)

Also after looking at a few of the raw data files, this is of course just a guess, but perhaps they are a type of custom packed commands like how you can program in overworld events?
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 28, 2010, 11:06:02 am
 :banghead:

I do believe I have a solution to the 'cell gap' that is so obvious, it almost must be true.
It has to do with the 'Double-Size' flag in an OAM data, which indicates that the rendering rectangle should be double the normal size (which is the size of the OAM), since the OAM may be rotated/scaled. You may feel it coming, but the X and Y in each OAM are actually the X and Y position of its rendering rectangle. Since the rectangle's centre stays in the same location, the X and Y in the OAM are half the size of the OAM less than when the Double-Size flag would be 0.

In other words, for a quick fix, for each OAM add half its size to its location whenever the Double-Size flag is 1. (since you still need the centre of the OAM when rotating it, the rest should be the same code)
Some quick tests confirm the cell gap to be gone in the Cells; I'm currently working on my own NANR rendering, so I cannot be sure of anything else. (Of course, I could also have accidentally only tested files in which this particular fix worked, while others need something else to be fixed)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 28, 2010, 05:08:34 pm
I knew it!!
I've been glaring at that flag for ages, trying stuff like dividing the x&y by two and getting the remainder etc and all sorts of dumb ideas with little luck. I also thought it was weird how your specs said it shouldn't be set and yet pretty much every animated sprite had it set so :crazy: Another conflict I've seen is generally the eye oam has its object mode set to semi-transparent according to your specs. I'm also seeing the R/S Flag set in some places too (typically most pokemon sprites, but some human sprites don't have it set for all oams).

(http://img404.imageshack.us/img404/2987/ctool3gapfix.png)

I've been caught up in a the archive export gui, specifically an option dialog similar to how Windows Explorer works when you are copying/moving files, because I was getting sick of switching between versions to extract stuff :p At the moment you can't specify the save location (which is defaulted to the directory in which the source file is located), as I'd rather wait until I make my own "Save-as" dialog instead of using the built in one with lots of tweaks.

(http://img221.imageshack.us/img221/1199/ctool3extract.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on November 28, 2010, 06:55:20 pm
I knew it!!
I've been glaring at that flag for ages, trying stuff like dividing the x&y by two and getting the remainder etc and all sorts of dumb ideas with little luck. I also thought it was weird how your specs said it shouldn't be set and yet pretty much every animated sprite had it set so :crazy: Another conflict I've seen is generally the eye oam has its object mode set to semi-transparent according to your specs. I'm also seeing the R/S Flag set in some places too (typically most pokemon sprites, but some human sprites don't have it set for all oams).
I never said it should not be set, only that it could be ignored (since I believed it only affected the clipping rectangle) ;]
The R/S flag should coincide with OAMs that are rotated in the NANR animations (and should be set for the Double-Size bit to have that meaning), although they may just have set the flag for all OAMs

The 'Semi-Transparent' flag means that some sort of blending is applied with the background layer (not with other OAMs!). I suppose it could mean that the .dat contains a u16 for each semi-transparent OAM with the blending data. It may also be the unknown u16 in a CellData, but that would mean the blend factor would be equal for the complete cell (which wouldn't be very surprising).
For completion's sake, this would most likely be the format of the u16: (from GBATEK, obviously)
Code: [Select]
4000052h - BLDALPHA - Alpha Blending Coefficients (W)
Used for Color Special Effects Mode 1, and for Semi-Transparent OBJs.
  Bit   Expl.
  0-4   EVA Coefficient (1st Target) (0..16 = 0/16..16/16, 17..31=16/16)
  5-7   Not used
  8-12  EVB Coefficient (2nd Target) (0..16 = 0/16..16/16, 17..31=16/16)
  13-15 Not used

For this effect, the top-most non-transparent pixel must be selected as 1st Target, and the
next-lower non-transparent pixel must be selected as 2nd Target, if so - and only if so, then
color intensities of 1st and 2nd Target are mixed together by using the parameters in BLDALPHA register,
for each pixel each R, G, B intensities are calculated separately:

  I = MIN ( 31, I1st*EVA + I2nd*EVB )
'OBJs that are defined as 'Semi-Transparent' in OAM memory are always selected as 1st Target ..., and are always using Alpha Blending mode'


.... I suppose I should get used to calling the OAMs OBJs. (OAM = OBJ Attribute Memory, apparently  -.-)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 28, 2010, 07:55:58 pm
No OAMs are good, no need to over complicate things :p An object could mean just about anything (Object is the base class type for everything in Java for instance), but most people should know 'OAM' has at least something to do with sprites.

Why the eye OAM needs to do all that is beyond me, the ONLY thing I can think of it being possibly used for is when you have those battle effects where the body gets inverted colors or something along those lines. Like a pass-through on certain effects or something (or the reverse). Must be frustrating to have a nearly complete spec and not know the final little bits -_- Like those 3 missing jigsaw pieces that are missing from your puzzle preventing you from finishing it :p

[edit]

Time for a release 8)
I haven't got round to fixing all the bugs in the UI as I found there were actually quite a lot more issues than I thought. Some bits have been disabled to prevent any wacky effects, but by rights, there should not be an instance where the program will simply stop working *fingers crossed* There may be some lags when processing a large number of files as no threading is currently used, but compared to the older versions, it should pail in comparison.

Along with exporting stuff from archives, I've also added the multi-select functionality that was used in the older versions for loading assets. Every time you select a file of a specific type (Palette, Image, etc) it will remember it and anywhere where that file is displayed (atm only in the Archive viewer) it will have a little link icon next to the file to indicate this file is part of chained action. So for instance if you wanted to load an NCER, you'd select all its assets first (NCLR, NCGR) and then double click on the NCER and it will load all of them together. Once I make my own file browser, this functionality will be added but for now it's only available in the archive viewer.

I've also included an updated help file (*docx) in the zip folder with a start on explaining the new interface. I'm open for any suggestions to make the document more useful than what it probably currently is.

So again, a quick run down of what you can do with it:

Hopefully it will work as smoothly for everyone as it does for me.

Download: Console Tool 3.0 (http://llref.emutalk.net/downloads/consoleTool_v30.zip)

[edit2]

Ack just found a bug -_- You can't view files in a compressed archive, it will just start reading from the start of the archive file. This happens because the caching was being overridden. Will be fixed in the next release...
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on November 29, 2010, 01:17:46 am
Ah yay a release with some export functionality I'm quite excited. Though admittedly without the ability to view DS models at all in this version it's not as useful to me. Though progress is always welcomed in any area. Thanks Low Lines.
Title: Re: The Console Tool (by Low Lines)
Post by: MegamanX on November 29, 2010, 05:19:40 am
It doesn't close on me now when I open a game with it.  That's cool.  Wish there was more I could do with it.  The few things I have don't have any of the supported file formats.  Haha.  Keep up the good work.
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on November 29, 2010, 03:54:19 pm
So, I'm guessing this means you completely abandoned 3d support... :'(

Kudos, though, for the release.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 29, 2010, 04:11:09 pm
So, I'm guessing this means you completely abandoned 3d support... :'(

Kudos, though, for the release.
Not at all...A few posts back I stated my milestones and what I wanted to have done before I released the next version. The things I wanted to achieve are mostly done now, which means it was time to release it. The aim here is to start releasing more frequent builds with small updates as opposed to one or two major updates every 2-3 months like I have been. Now I'm not saying I'm going to be releasing weekly builds or anything like that, but as soon as I do anything significant, I'll release it so people can use it.

3D support is the next thing on my list, in fact I'm working on textures right now...Sorry if I confused you :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 29, 2010, 07:04:20 pm
When you get to the motions, the 3D files I sent you a couple months ago should still be useful. Really standard DS stuff.
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on November 29, 2010, 07:26:37 pm
You're like a programming god Low Lines XD how did you manage to figure out the formats for all of these files? I really struggle to work out what it all means when looking at it in a hex editor.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 29, 2010, 11:24:54 pm
@Rhys
I think in fact only about 5-10% of the figuring out was done by me, the rest is off of docs other people have written, with a BIG kudos to Garyong who's been helping me figure out the sprite formats. Understanding how to read the hex is something that takes a while, though it helps to have a decent hex editor too! My app's hex editor sucks, but hopefully that will get better in the future!

@Koetsu
NSMB, Ben 10, Harvest Moon, Pokemon, and Zelda are my prime resources for debugging 3D stuff, mostly because they've got a wide variety of all the 3D file types, which are easy to locate since I've memorized the rom layouts. There were also some models from Ben 10 which still weren't rendering properly in the old build (such as Hex). Harvest Moon also has the most warped bones list which require stuff to be rendered in a specific order.

I think it's going to be a lot of work just getting Textures and Models supported, so motions/animations are still a fair way off yet :p

@Status
Textures are taking a while, mostly because I have to improve the data structure for graphics to allow for non-paletted/compressed/etc formats, as well as accessing internal assets, I've also been going over and updating the spec now that I'm a bit better at understanding the file structures.

You can view the updated spec here (http://llref.emutalk.net/v2/docs.php?order=&sort=&page=&id=btx0.xml), and I haven't updated any of the 2D docs, just in case anyone checks :p
Title: Re: The Console Tool (by Low Lines)
Post by: Toa of Mirrors on November 30, 2010, 03:08:41 pm
How come 2.5 and 3.0 were steps down? I thought we'd have more ability to work with MDL0s, not less! With my Java decompiler I can read the class codes myself, but I still can't rig it to both be able to view NSBMDs with custom bone opcodes correctly (2.0) AND have the ability to export the files in some way (0.2, 3.0?). Is there any way to just make 0.2 able to read BMD0s like 2.0 and still be able to export the models with 3D Via?

I'm sorry I upset the topic starter. I really don't understand the program at all.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 30, 2010, 04:07:59 pm
How come 2.5 and 3.0 were steps down? I thought we'd have more ability to work with MDL0s, not less! With my Java decompiler I can read the class codes myself, but I still can't rig it to both be able to view NSBMDs with custom bone opcodes correctly (2.0) AND have the ability to export the files in some way (0.2, 3.0?). Is there any way to just make 0.2 able to read BMD0s like 2.0 and still be able to export the models with 3D Via?
Okay this isn't even written nicely. Not only have you shown that you have not read anything from this thread (which has constantly talked about this), but you have also quite bluntly pissed on all the work that has been done to get to 3.0. Yes it doesn't support 3D stuff at this time, but the interface and 2D format support cannot be a step down from previous versions since with Garyong's help, we've gotten a much more complete spec on most of the formats than what I had before. I know that some people don't give a rats arse about anything but exporting models, and I think that if its such a big deal, that they should write their own programs or just wait, instead of trying to hack my program to do something it can't. People are welcome to complain, but please make sure you READ THE THREAD before complaining like this, it isn't fair on me and I'm sure no one else likes to read stuff like this.
Title: Re: The Console Tool (by Low Lines)
Post by: Toa of Mirrors on November 30, 2010, 04:52:13 pm
I'm sorry. I wasn't trying to upset anyone, I just was looking for some way to convert BMD0s to other meshes. Why is it that everyone becomes angered whenever I inquire about programming methodology?
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on November 30, 2010, 05:03:42 pm
Legit complaint, incendiary tone. NSBMD support is coming; he already said it's the next thing he'll be working on. Be patient.
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on November 30, 2010, 05:12:48 pm
also he made it quite clear he doesn't want people decompiling his code, so don't be surprised if he refuses to help you
Title: Re: The Console Tool (by Low Lines)
Post by: Toa of Mirrors on November 30, 2010, 05:15:59 pm
It doesn't matter, since I can't make heads or tails of it anyway.  God, I feel so foolish now. :banghead:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 01, 2010, 07:21:52 pm
Support for viewing textures in both NSBTX & NSBMD files has been added. However it seems that palettes and textures can be clones of the same data, which is also probably likely for other formats as well. At the moment I've only implemented it when decoding palettes because the size/end offset isn't specified in the data structure, so it would normally crash if the next palette offset is before the current one (when using the next palette offset to calculate the palette size). The associated texture is also generally cloned whenever there is an instance of a cloned palette.

Does anyone know of an open source (or well documented) model format that supports bones? I've been looking around and as far as I can tell only the major proprietary formats actually support them, meaning the really complex files with no documentation :banghead:.
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on December 01, 2010, 07:55:46 pm
Does this help any?

http://www.anim8or.com/resources/an8_format.txt

I'm still looking.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 01, 2010, 08:07:43 pm
Wells what would be ideal is if there's one that can be opened by proper 3D editors, or else exporting them to another format would be pointless. The other alternative I can think of is writing a custom I/O plugin for those apps, with possibly a custom bones format or something, though I only know my basic way around 3D editors and have not touched the scripts either, so that would be beyond me. IIRC someone wrote an importer script for gamecube models?

[edit]

The Maya ASCII format doesn't look too hard to write an exporter for, however I think that it is limited to Maya.
FBX seems to be openable by both Maya and 3DS Max, and it does have an ASCII version also. Probably the best option out of what I can find so far. But if people know of better options, I'm all ears...
I suppose what the real question is, which apps do people use? Does anyone use Blender?
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on December 01, 2010, 09:03:31 pm
What about Obj? As far as I know pretty much every 3D editor supports it, so it'd be ideal for exporting to, no idea if it supports bones though

http://en.wikipedia.org/wiki/Obj
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 01, 2010, 09:43:17 pm
Quote from: Wikipedia
The OBJ file format is a simple data-format that represents 3D geometry alone — namely, the position of each vertex, the UV position of each texture coordinate vertex, normals, and the faces that make each polygon defined as a list of vertices, and texture vertices. Vertices are stored in a counter-clockwise order by default, making explicit declaration of normals unnecessary.

Unless I've read it wrong, that means it only stores the geometry. I'm definitely going to try and support 3DS and Obj, since they're like the most common formats on the net, people will just have to realize that these formats do not export bones/joints.

I've got the 3D UI up and running. It was mostly just a copy and paste job from my old code, however I really have to rethink my data structures. I also figured out how to apply on-screen information, which is done by simply painting over the top of the 3d Panel instead of relying on the JOGL libraries. It's actually much better since its not limited to the basic text drawing methods plus I can also paint graphics etc on top if I choose. Am also experimenting with displaying multiple camera views, though its not a focus at the moment so it will probably be disabled in the next build...But just something to maybe look forward to in the future :)

(http://img221.imageshack.us/img221/999/ctool3model.png)

It is going to take time to properly plan out the data structures, plus it looks like I'll have other (paid) work to do during most of the day starting from next monday up until I go back to uni, so I will probably have less time to spend working on it. I do plan to work on this as much as possible in my free time though, so never fear :)
Title: Re: The Console Tool (by Low Lines)
Post by: Tater Bear on December 01, 2010, 10:42:57 pm
I know next to nothing about 3D stuff, but I read an article about the industries leaders wanting to create a universal 3D file format.
http://www.ecma-international.org/publications/standards/Ecma-363.htm <--- That is the format.
http://www.web3d.org/x3d/specifications/x3d/ <--- Here is another popular file format.
Title: Re: The Console Tool (by Low Lines)
Post by: BRPXQZME on December 01, 2010, 11:32:08 pm
Remember VRML? No? Good.
Title: Re: The Console Tool (by Low Lines)
Post by: Next Gen Cowboy on December 02, 2010, 12:38:25 am
Floops ftw! Sorry, it's not as though it's my speciality, but I will always remember that...that...well whatever the Hell that thing was.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 02, 2010, 05:25:06 am
There are a fair few XML-based formats to choose from, X3D is the successor to VRML according to Wikipedia. 3DMLW is another (more recent) one. Also I was only a kid in the 90s so I had to google just to figure out what you guys were talking about XD
Title: Re: The Console Tool (by Low Lines)
Post by: Azkadellia on December 02, 2010, 05:41:47 am
If you're going to support the .3ds format, may I suggest this: http://code.google.com/p/lib3ds/ ? It's a library for dealing with the 3ds format.

Awesome tool BTW.
Title: Re: The Console Tool (by Low Lines)
Post by: Lexou on December 02, 2010, 02:20:54 pm
Wells what would be ideal is if there's one that can be opened by proper 3D editors, or else exporting them to another format would be pointless. The other alternative I can think of is writing a custom I/O plugin for those apps, with possibly a custom bones format or something, though I only know my basic way around 3D editors and have not touched the scripts either, so that would be beyond me. IIRC someone wrote an importer script for gamecube models?

[edit]

The Maya ASCII format doesn't look too hard to write an exporter for, however I think that it is limited to Maya.
FBX seems to be openable by both Maya and 3DS Max, and it does have an ASCII version also. Probably the best option out of what I can find so far. But if people know of better options, I'm all ears...
I suppose what the real question is, which apps do people use? Does anyone use Blender?

OMG I should have seen this earlier. As a part-time modeler, I would recommend The COLLADA .DAE format. A lot of 3d editing programs support it, and/or have other importers for it. It's opensource (I think), and supports rigging, bone hierarchy, jointing, etc... You can learn more about it on their wiki. (https://collada.org/mediawiki/index.php/COLLADA_-_Digital_Asset_and_FX_Exchange_Schema)

Also, all the formats the other members here have posted about (apart from .3ds and .obj, duh) are really uncommon, even though importable in today's main 3d editing apps.

Lastly, are you going to add (no rush at all) import functions ? If so, again, .dae is recommended.
Title: Re: The Console Tool (by Low Lines)
Post by: Zerox on December 02, 2010, 05:55:29 pm
I'm pretty sure there is lot's of information on Valves SMD model format out there and it's even got bone support. I know of lots of 3D modeling programs capable of using it as well. =D
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 13, 2010, 06:09:02 am
Turns out this job is a lot more work than I was expecting. Imagine this, IT geek assigned to a computer which only has IE6, AND a ball point mouse. Not a single person at my job knows the first thing about web design, and apparently they have to get some guys from America to do any fancy stuff like Flash because they don't have anyone with enough skill in Australia. So I've been going retro for the past week, reliving my childhood with good old Notepad and Paint XD
And if anyone has heard of SharePoint...It's going to be my buddy for the next 3 months!

Anyways come the weekend, I almost didn't even want to touch my laptop let alone do anything other than watch videos, so I guess this project is on hold for bit until I get a decent amount of this job completed :(

At least I'm getting paid though :)
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on December 17, 2010, 01:55:58 pm
Nice work yet again. :cookie:

Here are some suggestions:

Fileformats I would like to see:
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on December 20, 2010, 11:02:31 am
Yeah, the file links was very confusing. I didn't want to say anything because I thought it was explained somewhere. It seems unintuitive and not like Tahaxan at all.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 21, 2010, 03:37:26 am
Really?

Wells first off blue files are compressed files (or at least have been interpreted as compressed). It was something present in the older versions of the application, which I had previously explained so it had slipped my mind when I was working on the manual.

As for the file linkage, I honestly thought that was pretty straight forward.

There's an array of file objects which keeps track of all files that have been opened. Whenever you refresh the UI (typically done by selecting the appropriate file tab), it first checks to see if the reference of the file of each asset type (ie Palette, Image, ANimation), still is opened, and if it isn't, it then tries to open the last opened file of that type, or nothing if there aren't any files of that type left. The Assets property, which is located under the Info panel shows you which assets are currently loaded into the selected file. If you click on them, it'll display a drop down menu which you can then switch to another file of that type if there is another one loaded.

In terms of loading multiple files at once, at the moment it won't attach an asset that is loaded after it, (ie NCGR then NCLR will not have assigned a reference of the palette to the NCGR), however as soon as you switch tabs, it will run the above routine again and you should be fine unless you already have a palette opened beforehand, in which case it would already have a reference to that. But again its just a simple matter of swapping the selected asset.

Was this mechanic really that confusing?? If so I'll have another go at explaining it later.

Bear in mind though that the Open File Dialog is only able to load one file at a time as it is just a quick implementation therefore lacks any further functionality. Same goes for the archive viewer, it's only set up to load one file at a time. Context menus are planned, though you should already be able to right click on any image and copy it to your clipboard.

I mostly use the Drag & Drop myself, though I suppose most people are more accustomed to dialogs >_<

Anyways, due to the fact that most people at my workplace are going to be on leave next week, thus limiting the amount of stuff I can get done since I'm still in the process of getting approval/feedback from heaps of people. I've got all of next week off, so hopefully I'll get to sit down and work on this a little :)

@henke37
SDAT is on my list, but if you want the others you've got to provide at least some of the information yourself, or link me to a place that has information about them. I know this sounds kind of lazy, and I'm sure I will eventually get round to looking into them myself since I find it fun, but if your looking for a quick way to be able to accessing custom formats, it'll happen much sooner if I've got the know-how already.

I've had one person (who I think is still wanting to keep what they are doing secret) PM me with their notes on a custom format, and it only took like 10-20 mins to add support for it. That being said, archives are by far the quickest and easiest formats to add support for since there's not a lot of variation in the data structure, but other things will take longer since their data structures are still a work in progress.
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on December 21, 2010, 04:13:42 am
Don't worry, Low Lines, I came clean in the IRC.

It's Solatorobo, and Low Lines helped me immensely by implementing support for its CCB archives (I'd worked out the format, but Low Lines figured out a few extra things I got wrong).

Project is stalled for now; I've got script extraction/insertion going, but of course I'm trying to figure out if there's some way to make the game display its English-language resources. Yes, there are English opening credits tucked away in the ROM, and in multiple flavors to boot.

Back to you. :3
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on December 29, 2010, 12:54:56 pm
I have asked the guy who wrote the sprite extractor. He isn't very active and is a pain to contact, especially when the relevant forum currently being down.

The pac files look rather easy in a hex editor. My wild guess is that it is a few bytes of header followed by a pointer/pointer+size table of the contained files.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 30, 2010, 01:44:20 am
@henke37
You are correct. PACs have a simple structure with file offsets rounded up to values divisible by 4. First DWORD is the file count, followed by a FAT table with the file offset and size (DWORD for each). Support has been added for them (tentatively titled Capcom Packages), which will be in the next build.

VX (Magic: VXDS) files are some kind of structured video format (possibly from the Nitro SDK?), however I have tried to look into these in the past and haven't had much success since I'm not that familiar with video/audio formats.

The Layton game I looked at (Diabolical Box), had both ARC & ARJ as well as ARB, but none of these seem to have a very obvious structure to them at a glance.

@Status
I have not had as much focus on this project for my week off work as I had hoped, which I think is mostly due to the festive season and not wanting to get too stuck into anything and then have to drop it again come next Tuesday. That being said, I've been studying Garyong's Huffman code to try and finally understand how the compression works so I can add support for it as well as other compressions that use a combination of Huffman & LZ, which is the case with a lot of common web formats.
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on December 30, 2010, 05:30:56 pm
Talking about strange package formats, the hotel dusk package files seem to come in pairs. One is obviously the file table (it got file names, yay!) and the other seems to be the file data (why is the file names repeated in that file as well? beats me.).

But hey, thanks for the effort.

Anyway, is there an option in the GUI to force a file to be parsed according to a certain format? Just in case you end up wanting to support some unguessable "*.bin" format.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 30, 2010, 07:24:12 pm
Having a complimenting file isn't that strange at all, I was looking at this iTunes App the other day and every image had a dat file which stored the name and coords of subimages within the full image.

As for a force option, at the moment no, but the idea is that if a file matches several format conditions (such as with BIN being a common extension for multiple files) and there aren't any overriding formats (ie with a matching MAGIC stamp), it will ask you which format to try and load it as. However as it doesn't currently support many files, there hasn't been a need to implement it yet. Forced format loading could also be added as a manual option however I think you'd probably find it to be a lot less useful than you'd think. If a file isn't automatically detected as one of the generic formats, it is most likely a custom one, and since there can be a real variation in how that format is structured, forcing it to open a certain way would be like trying to put an American plug into an Australian power socket. It wouldn't work out too well :p
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on December 31, 2010, 08:02:15 am
I just remembered that the Wiibrew wiki (http://wiibrew.org/wiki/File_formats) has some file format specifications. Implement them as you feel like.

I also recalled that the custom file formats used by Westwood (the C&C guys) have been reverse engineered. I found two (http://vladan.bato.net/cnc/) links (http://xhp.xwis.net/utilities/) on google. Consider adding it if you don't think it is too out of scope.
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on January 25, 2011, 11:43:09 am
you mentioned you wanted to add ps2 support at some point. this guy could help when your ready

http://ps23dformat.wikispaces.com/


also did you ever get tmd working (i have to reinstall java on my pc before I can try)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 25, 2011, 01:47:02 pm
@henke37
Before I got into this project, I was totally into the whole Wii Homebrew scene, and I still regularly go there and know of all the file specs detailed on the wiki. I only wish they had more information on the signing process. I've tried to port Segher's code several times, but I'm not big on math anymore and elliptic curve cryptography is bit out of my comfort zone, which has prevented me from my own hacking of savefiles etc.

As for C&C...perhaps a tad out of the current target of this project? I think the main focus at the moment is console games (and probably even more specific, NDS ones) since the resources available for them vary and are often only partially implemented, whereas PC games might even have whole communities devoted to developing new content for individual games. But hey, maybe somewhere down the line :p

@viperzerof-2
Nice, lots of custom formats!! As for TMDs, I'm pretty sure I've got a working version somewhere, however I have not yet even considered adding it yet to the current version. From what I recall though, they don't include bones/joints (or whatever is used for positioning body parts etc), so some models (ie a digimon from digimon world 1), will have have all objects relative to zero, which makes them look like they have been crammed into a suitcase. Will have to refresh my memory to be certain what's the go though when I am able to look at it again.

@Update
Once again, really have not had much of a chance to work on this project since my job wipes out the whole week and I end up spending the weekends recovering (is not made for waking up at 5:45am to catch a 6:40am bus and then another 40 min bus ride out to the site, and then catching that same bus back at 4pm in the afternoon, thus essentially getting home near on tea time and barely having enough time to do anything else but prepare for the next day -_-). I'm 2/3 of the way through it now and after that its back to uni again come late February, early March, so with any luck I'll be able to get a bit of work done before the studies kick after that.

Sorry for that lack of activity guys, but I hadn't even realized it had been a whole month already, that's how full-on this job has been!!
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on January 25, 2011, 03:51:58 pm
oh you were on wtw right? anyway you project is awesome, though i'm surprised I only just found out about it
Title: Re: The Console Tool (by Low Lines)
Post by: AzuShin on February 01, 2011, 02:46:42 pm
Speaking of models, it would be awesome if Someone could crack Star Ocean 3's file system. Putting it in the disc drive only shows like 3 files whic don't equal up to the game's actual size. Anyway, good luck with the project, I'm looking forward to it.
Title: Re: The Console Tool (by Low Lines)
Post by: azoreseuropa on February 03, 2011, 11:31:09 am
Is this tool available for SNES roms too?
Title: Re: The Console Tool (by Low Lines)
Post by: KaioShin on February 03, 2011, 12:02:50 pm
No.

SNES games don't generally share common formats in the way the DS does due to the SDK.
Title: Re: The Console Tool (by Low Lines)
Post by: azoreseuropa on February 03, 2011, 01:48:49 pm
No.

SNES games don't generally share common formats in the way the DS does due to the SDK.

Oh, I understand. Thank you very much.
Title: Re: The Console Tool (by Low Lines)
Post by: Jandazekon on February 03, 2011, 05:16:01 pm
There is 2 unique ps2 games called soul reaver 2 and metal gear solid 2.
They can't be dumped and converted without knowing reverse engineering according to ps2 assembly.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on February 03, 2011, 05:37:08 pm
@Naruto
Older console games (those that don't have their own internal file system) would probably require custom plugins on a game by game basis, and considering just how many games there are, that is a project within itself.

@Jandazekon
There would be a lot of games (for both PS1 & PS2), that attempt to hide their content from the hackers, though my program (at least not the most current version) doesn't have ISO reading support yet so I'm not sure what you are implying.
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on February 12, 2011, 11:26:40 am
hey you seem to like digimon if you want I can send you the sprite files from digital monster ver.s and maybe you could add support for them?
Title: Re: The Console Tool (by Low Lines)
Post by: KarlYu on February 18, 2011, 09:05:41 pm
Hi, Low Lines

We are trying to translate a NDS game into our language, but the most pictures are stored with NCER,NANR,NCLR,NCGR format.
I am very exciting to find your great The Console Tool by Google, it can show the picture right,
but can the picture be exported and then be imported?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on February 18, 2011, 10:28:17 pm
Unfortunately not yet, and I can't give you a date as to when it will be since this project is something I do in my spare time, which is what I haven't had a lot of lately :(

You have of course managed to find RomHacking.net which is chock full of people who obviously spend their free time doing such things.

I have not really done any translations myself, but from what I gather you can use my tool to find and/or extract the raw files (in their native format) and then rely on other tools to modify them. Assuming you are able to get to this point you will then need another program to repack the game you are trying to translate. You should only need to modify NCLR and NCGR, assuming you want the graphics to behave in the same manner as they originally do.

You should start a new thread and ask people what other tools you will need, there are some really helpful people here!!
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on February 18, 2011, 10:30:42 pm
Even though it can't export/import graphics yet, you can use it to narrow down the search for the graphics you need to edit.

My usual method is to extract the whole ROM to a directory structure using ndstool (and the same to reassemble the ROM for testing) and edit individual files using Tile Molester.
Title: Re: The Console Tool (by Low Lines)
Post by: KarlYu on February 19, 2011, 03:10:25 am
@Low Lines
Thank you for your detail replay, I will create a new thread about my problem.

And I hope you will still continue developing this tools, 'cause it's really useful for game translation.


@Ryusui

You are right, it can help me chose which picture we need to modify.

In this game, thousands of pictures are NANR format, if I can find the appropriate tools, we can only do ti manually.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on February 19, 2011, 03:34:21 am
No problem.

I plan to keep going with the project, I just need the time and the right motivation, like the 3DS coming out next month!! I think especially once someone manages to extract the games, there ought to be a resurgence in the rom hacking community again. I'm especially keen to know how much of the NDS SDK stuff is being recycled and/or what changes they've made, not to mention how they store/render 3D 3D models :)

Microsoft & Sony have never really caught my interest with all their PC Gamer garnered content and the Wii casual gaming has been really really dull, so I essentially am only interested in first-party Nintendo games which are few and far between. Am going to be so hyped when Zelda 3D comes out though, that's one game that I want to open up and dissect (as well as thoroughly play too of course)!!

Good luck with your project!!
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on March 06, 2011, 02:54:59 pm
This project now has an official home (http://llref.emutalk.net/projects/ctool/ (http://llref.emutalk.net/projects/ctool/)), and a super awesome logo/banner  :crazy:

Still haven't had much of a chance to work on it unfortunately, since my main focus for last fortnight has been gearing up for Uni and getting my website back in order (which still needs some of the old projects migrated, and some new ones added), but I do plan to sit down and do some work on this soon.

Due to my vacation job, I hadn't touched Java for 3 straight months, so it was quite the relapse when my Java lecturer told us to write a linked-list class from scratch making use of recursion :banghead:
So it looks like I'm going to learn a fair few more new things this semester that are going to help improve my programming :p
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on March 06, 2011, 05:34:38 pm
Since I'm working on Digimon World 1 for PSX it would be awesome if one of the next released could implement a 3D Model Viewer
and (dreaming) a MAP Viewer (for reference http://www.sendspace.com/file/uc228f) the TFS in there is such a Map for example, the MAP itself is just collision data and some TIMs)
Title: Re: The Console Tool (by Low Lines)
Post by: Friedslick6 on March 08, 2011, 03:54:41 am
Wow Low Lines, nice new website!
I hope you do well with the inevitable pile of work coming your way.
Title: Re: The Console Tool (by Low Lines)
Post by: dicamarques on March 10, 2011, 09:46:08 am
Sorry for comming into here but does anyone know how to change that season screen that appears in pkmn black and white when the game starts?
Thanks
P.S. tried with cristal tyle and nothing happened :banghead:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 04, 2011, 11:24:53 am
Gosh time really does fly!!

Studies have obviously kept me busy over the past month however I have been working on ctool off and on, more so since last Friday, which was the start of my mid semester break (though I still have assignments so it's more of a "semi-break" >_<). The main focus of my Java unit up to now has been on data structures, from which I'm certain will benefit on this project once I get the chance to apply my new knowledge. Not only that, but I've also been getting some advice from my new lecturer (who actually works as a Java programmer) with specifics to aspects of this project as I've been slowly reworking the GUI in my spare time. It's made the code behind the workspace design alot simpler and much less buggy than it was in the last version. Docks are now fully resizable, and lock once they fill up the content area. Workspace components can also be removed from the layout without it breaking and can be reapplied through the menu. This will help with hiding/showing components relevant/non-relevant to the type of file being edited, though I still have to reapply drag-n-drop and add Group component (the grey ones that hold the tabs) resizing before it is finished.

I've also been going over my file access classes and simplifying them down more while also trying to crack down on the memory stockpiling, which some may have noticed can happen when opening a large number of files (regardless if they get closed or not) over time. I'm quite certain it is caused by data objects not being properly de-referenced, preventing JVM from freeing up the memory. I had a lot of listeners latched on this way and that, so I'm going to rework this from the ground up and keep it simple like I've been doing with the workspace design.

Once I've got this far I plan to continue work on the 3D viewer/editor as I had planned to back in December, though it was probably better that I hadn't started work on it then as it would have likely meant I'd have to do a lot more reworking to get the data structures right since I know more now than I did back then :p

On a side note, I am for the time being working from a Mac, which aside from the fact is backwards in MANY MANY ways (#_#) has given me the opportunity to develop ctool on a second platform.

Come the next public build, I think I'll definitely start releasing versions for multiple platforms as personally I would kill for a decent hacking app for Mac right now :banghead:
Title: Re: The Console Tool (by Low Lines)
Post by: chrrox on April 05, 2011, 09:56:44 am
Hi can you add support for these models it should not be to hard to add it to your program.
it can already convert the textures.
http://www.mediafire.com/?pra6qlwg55a4r89

just in case here is a quickbms script to extract the arcs.

Code: [Select]
get arc long
get files short
get comp1 short
get null long
comtype lz77wii
goto 0xC
for i = 0 < files
get offset long
get zsize long
get size long
if null == 0
getdstring name 0x20
else
get name basename
string name + _
string name + i
endif
if zsize == size
log name offset zsize
else
clog name offset zsize size
endif
next i
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 07, 2011, 04:56:49 pm
@chrrox
I'm not that familiar with quickbms (in fact I've never heard of it before), and it wasn't all that clear you were pointing me to an archive format until I opened it up in a hex viewer :p
That being said I will look into it once fix up my archive classes.

@status
pretty much everything I had set out to do with the workspace reworking has been implemented. you can rearrange the layout to however you want which includes collapsing groups and/or hiding components freely. I've also added a bunch of menu options related to the layout management including a reset to the default layout in the event something goes sour (although I've tested it enough that it shouldn't). The only things I haven't got working yet is resizing groups (vertically) and keeping track of layout sizing. By default, everything will revert back to its minimum size upon reloading.

I also spent a fair bit of time trying to get a grasp on the memory concerns, and after re-reading up on the Garbage Collector, I have a few points people should be aware of with Java apps. The amount of memory allocated to the program in the taskmanager does not necessarily represent how much memory is actually being used. Having a computer with its own set of hardware problems I'm constantly keeping an eye on it and so I've been making the same mistake myself. Long story short, i've added a menu option ("Garbage Collection") which will manually call the process, and I've also included some usage statistics in the statusbar to make it a bit more clearer for people exactly what the app is doing with all your memory :p Calling the GC manually won't guarantee you'll free up memory, but it might produce better results if you are opening and closing files all the time ;)

The next step is to go through the file format specific ui elements as well as the file formats themselves and fix up anything that needs attention, and then I can get cracking into the 3d stuff :)

[edit]

Here's a preview of the reworked layout. Probably still looks mostly the same, but its better in terms of functionality compared to 3.0
It's all bare bones at the moment, I've still got re-add all the ui elements from the last version (albeit cleaner and more stable).

(http://img830.imageshack.us/img830/5143/ctool31.png)
Title: Re: The Console Tool (by Low Lines)
Post by: chrrox on April 08, 2011, 04:35:23 pm
quickbms is a command line tool.
http://aluigi.altervista.org/papers.htm#quickbms
it works with scripts to extract archives.


once these files are extracted they are standard ds formats except they cant be opened by your viewer yet.
they almost load in some other viewers.
your program loads the texture files fine.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 08, 2011, 04:46:59 pm
Hmm I see...might be a variation of the format. Just at a glance the hex reads off as NSBMDs though. What game is this from?
Title: Re: The Console Tool (by Low Lines)
Post by: chrrox on April 08, 2011, 05:06:41 pm
Love plus
Title: Re: The Console Tool (by Low Lines)
Post by: Aurelazza on April 08, 2011, 08:01:31 pm
Wow, what an amazing-looking program!

I came close to posting this question on the Newbie board, but in the end it's directly related to Console Tool. Suppose that I wish to export a Nitro Graphic Resource image (in this case, a Pokemon White trainer sprite) from Console Tool to Photoshop or a similar program. I know of the Copy Image function, but once I've copied and edited the sprite in question, I'm ignorant as to how to make it viewable in Console Tool again. Would simply compressing it in an LZ77 format do the trick? If not, how might I re-import an edited image or directly export one instead of using the Copy function?

Many thanks!
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 08, 2011, 08:15:10 pm
It's still not quite there yet.

Palettes are the only thing that can really be converted between formats at the moment. The copy function is just so people have a way of ripping images until I get a proper implementation working.

That being said the type of task you want to do probably requires being able to edit multiple files that make up the complete image and that's even more functionality than just converting a flat image from one format to another. I am working on it but it's unlikely there will be a working solution any time soon.

Sorry to disappoint :p
Title: Re: The Console Tool (by Low Lines)
Post by: Aurelazza on April 08, 2011, 08:29:34 pm
Bummer =(

Forgive me for asking in the wrong place, but might you know of any tools capable of doing such a thing?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 10, 2011, 05:32:44 am
@Aurelazza
Here's one alternative that's been mentioned in this thread...
Even though it can't export/import graphics yet, you can use it to narrow down the search for the graphics you need to edit.

My usual method is to extract the whole ROM to a directory structure using ndstool (and the same to reassemble the ROM for testing) and edit individual files using Tile Molester.
In general you will have to get your hands dirty to pull off any significant level of hacking (ie hex editing, writing your own code to do stuff, etc).
Title: Re: The Console Tool (by Low Lines)
Post by: Aurelazza on April 11, 2011, 09:52:06 pm
Gotcha, thanks Low Lines!

Edit: I just realized that Console Tool might be able to help me with another issue that I've been pondering over for quite some time. Has the map-editing aspect of CT progressed to the point of being able to insert small things like chairs, trashcans, or shrubberies into an area in games such as Pokemon White?

If it has, then might you kindly point me in the right direction as to get started?If it hasn't, then what sort of shortfalls are currently in the way?

Thanks again.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on April 19, 2011, 12:27:31 am
Hi there Low Lines,
I admit that I am lazy to read all 21 sites so post first and check later and sorry if I noticed something well known or ask something answered many times.

3 thing in my mind:
- it seems that your hex viewer misses the first 5 bytes of a file. Here u see CT on top and Hew workshop below with the same file open:
(http://img17.imageshack.us/img17/6210/cterror.jpg) (http://img17.imageshack.us/i/cterror.jpg/)

- is there a "in-deep" manual for CT??
-what is the best way to view the NDS graphic formats?? What is the right sequence if everything in 1 file??

Thanks alot
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 19, 2011, 12:59:51 am
First off, the most likely case with missing bytes is that the file is being decompressed automatically (ie LZ77 first 4 bytes is the compression header and the fifth being the start of the compressed data). Normally this would be indicated with filenames in the browser being colored blue however the current browser won't indicate that. Secondly the hex viewer is just a viewer, you can't use it to edit files yet just in case you weren't aware.

The only manual at the moment is whatever documentation that is included in the zip folder. If you have suggestion on where and/or what should be more detailed, let me know and I'll try to improve it for the next version. When you design an application you tend to not think everything needs explaining since you know how it all works, so it always helps to have another person look at it and point out what they don't understand.

Best order of opening associative files like graphics is opening them in reverse based on their relationships.

NANR
 needs NCER
 needs NCGR
 needs NCLR

NCER
 needs NCGR
 needs NCLR

NCGR
 needs NCLR

therefore the best order in which to open them is
NCLR, NCGR, NCER and then NANR

though version 3.0 should automatically reorganize its assets upon selecting an opened file, which can also be changed via the "Info" tab under "Assets".
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on April 20, 2011, 01:14:36 am
Umm, can an overlay on the NDS be compressed?? There is not the typical x11 on that file either.

I saw that u can write in the hex viewer but i believe nothing happens to the file...right??
Nothing would happen is I start corrupting an NCGR in your viewer while opened in CT..right??

My suggestion (for now) is to add this little pieces of advices u gave me to the doc in the zip.
I never used before I opened this files.

CT will reorganize its assets upon selecting an opened file but only if it countains only one complete set of format and not if the file countain multiple NCGR (for example)..right??

Is there an import/export function of those formats planned in one of the next versions of CT??

Thanks


Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 20, 2011, 03:04:03 am
There a multiple flags in a file header that might trigger file decompression of multiple types (0x11 is just one example), and currently there is no way to manually turn this off. There is no way for ctool to recognize what rules apply to a file unless it provides some kind of header in the file it recognizes. As far as its concerned, an exported overlay file is just some unknown binary file.

The hex viewer is is just loading the binary data into the fields. After that it's just text on the screen, you mangle this in any way you like and it will have no effect on the file.

You can have as many files of the same data type loaded at the same time. Every time a file is reloaded in the viewer it will first check to see if it has an asset for each association (ie an NSCR has two assets, NCGR and NCLR), then it will check to see if that asset is still opened, if it isn't it will try to open the last opened file of that data type. If there aren't any files of that data type opened, it will try displaying the file with dummy information, for instance NCER should just display numbered squares representing oams etc

Import/Export is planned with palettes being the only CURRENT data type that can be converted between supported formats in the current version. As to when it depends on when I have added support for other formats that they can be converted to. The main thing that holding up support for formats such as PNG/APNG/GIF (which would be the most ideal formats for exporting NDS graphics/animations), is the compression/decompression routines, which aren't my strong point. Java does have its own APIs for handling images however it has a limited level of control over the exported file.

You can also copy whatever graphic is loaded in the viewer to the clipboard by right-clicking and selecting "Copy" from the popup menu. This is just a basic implementation to allow graphics to be exported until a proper one is done.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on April 20, 2011, 01:52:09 pm
Understood.

I know of only one game where there is need to mess with the overlays and that is "Ace Attorney Investigation: Miles Edgeworth" where the text is stored in the overlays.

I will play with those NDS graphics. I think the best is to split my files into pieces and then open them up one by one.

Saddly i can't help u with programming...not my strong point either :(

Thanks
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on April 22, 2011, 04:45:07 pm
There seems to be some sort of rendering bug with NSCR files, they show pure green in spots where it is not supposed to do that. It only happens to some files.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 22, 2011, 05:43:37 pm
That means it ran out of palette colors (as in indexoutofbounds) during rendering. It should only happen when you try to match up assets that weren't meant to be used together. You can always pm me the files if you want.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on April 22, 2011, 10:09:41 pm
Ï am not sure it's only that ... look:

(http://img28.imageshack.us/img28/4139/55491157.jpg)

but here ok:

(http://img194.imageshack.us/img194/7489/58260076.jpg)

the 2 button look like this:
(http://img854.imageshack.us/img854/2791/69876474.jpg)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 23, 2011, 12:54:17 am
Yeah I've been looking at that exact bin archive myself...The cause is most likely the palette offsetting in both NSCRs and NCERs, as when just viewing the NCGRs they render okay. Will look into it once I get up to reviewing my implementations of those formats :p
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on April 23, 2011, 06:38:23 am
Auryn, What exactly is the problem? The button loads fine. Did you temporarily forget the drop down menu in the second toolbar?
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on April 24, 2011, 03:12:05 am
Somebody from the french translations said it's really saved like that and not an error of CT.

My question to be sure it's like that, is simple:
On the NCER screen u have those 4 rectangles, the NCGR data doesn't have to match those rectangles...right??
It's not that in the NCGR there has to be a piece exactly big as rectangle 03 (for example) but the can be 2 pieces (example an upper and lower half) that compose that rectangle 03..right??

The more we dig on this game, the more it get's better :p (is this a double screen image??)
(http://img90.imageshack.us/img90/7767/unknow2.png) (http://img90.imageshack.us/i/unknow2.png/)

or even better (double screen and bigger in width to allow a side scrolling??)
(http://img546.imageshack.us/img546/5953/unknow.png) (http://img546.imageshack.us/i/unknow.png/)

No NCER inside that directory, just some NSCR but none seem to work with this 2 images.
 
For now we are lucky that those images don't need to be translated.

Title: Re: The Console Tool (by Low Lines)
Post by: KC on April 24, 2011, 03:26:25 am
Umm, can an overlay on the NDS be compressed?? There is not the typical x11 on that file either.

Overlays are completely defined in the Overlays Source area of the NDS rom. That's usually extracted as some small .bin file. Every overlay has 32 bytes there.

Code: [Select]
typedef struct sOverlay {
   int OverlayId;
   int RamAddress;
   int RamSize;
   int BssSize;
   int StaticInitStart;
   int StaticInitEnd;
   int FileId;
   int Flags;
} tOverlay;

Flags decides whether the overlay is compressed or not.

Code: [Select]
int GetOverlaySize(tOverlay& Overlay)
{
   if (Overlay.Flags & 0x1000000)
   {
      return Overlay.Flags & 0xFFFFFF;
   } else {
      return Overlay.RamSize;
   }
}

If 0x1000000 of .Flags is set, then the overlay is compressed with the typical backwards LZSS stuff (pretty much like the BIOS one with a different header, just in a different direction). The lower 3 bytes of .Flags then point to the end of the file where it can start decompressing.
If you clear 0x1000000 and provide an uncompressed overlay, it will work just fine. The compression is optional and completely defined with this structure. As far as I'm aware, all NDS games support it.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on April 24, 2011, 03:07:53 pm
Oh thanks KC, even if it's too late because we solved.

About those pics, Low Lines, it seem they are just with the wrong size ( i don't know if from header in the files or error in CT).
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 24, 2011, 09:23:45 pm
They're images which are meant to 512 wide but instead are being rendered as 256. Since NSCRs work with tiles, that's how you get that stripy look. 256x256 is like your normal drawing area, when the image is bigger than that, it has to get drawn a little differently. I've NSCRs that store each 256 block after each other, but that one is clearly doing the entire row sequentially :p

[edit]

Ok with regards to the coloring error (where it displayed green), that happened because I hadn't included a conditional check to see if the Map (or rather the NCGR) was 4bit or 8bit. By default it opens as 4bit, which means it only uses 16 colors instead of the full 256.

(http://img576.imageshack.us/img576/5145/logobin.png)
Gyakuten Kenji 2 (BOXJ) - contents/com/logo.bin/2.nscr

Am looking into the other issue now...

[edit]

Looking at the specs on GBATek (http://nocash.emubase.de/gbatek.htm#lcdiobgcontrol), it looks as though the internal screen size can be set to different sizes...

The unknown value in the NSCR map appears to hold different values for the mucked up images, though I've also noticed that NCGR with no obvious NSCR assigned to them are also affected by this internal sizing thing. What I've done for the time being is add a few if statements that will use a bigger internal size if the image is bigger than normal...

(http://img62.imageshack.us/img62/4996/upcut406bin.png)
Gyakuten Kenji 2 (BOXJ) - contents/com/upcut.bin/406.ncgr

[edit2]

It looks like some of the NSCRs use a weird mapping layout...

(http://img571.imageshack.us/img571/5043/upcut94bin.png)
com/upcut.bin/94.nscr

The upper image is what the NCGR looks like (which is drawn row by row), and the lower image is what the NSCR looks like. It only seems to affect graphics wider than 256px so it's likely an error in my rendering code.

[edit3]

Appears as though the problem has been fixed...

(http://img171.imageshack.us/img171/2554/upcut94bin2.png)
Gyakuten Kenji 2 (BOXJ) - contents/com/upcut.bin/94.nscr

Even though it's repeating the image twice, that can't be an error in ctool because the index is counted in the loop (not by using math to figure out the index). I went and tested a few other big NSCRs to make sure it was working...

(http://img683.imageshack.us/img683/179/10asnscr.png)
Digimon World (ADNE) - contents/dat/BTMAP/10as.nscr

Note: The above is a before & after.

I've also uploaded a modified build with these fixes applied to the project page (http://llref.emutalk.net/projects/ctool/). Keep in mind that this is using the old gui since I'm still working on the next version which has the more stable interface.

Let me know if you find any NSCRs that still display wrong...Also it would be helpful to include the location of the file like I've started doing in my posts...

ie Game Title (Game Id) - filePath

Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on April 25, 2011, 10:04:24 am
[snip]

I've also uploaded a modified build with these fixes applied to the project page (http://llref.emutalk.net/projects/ctool/). Keep in mind that this is using the old gui since I'm still working on the next version which has the more stable interface.

Let me know if you find any NSCRs that still display wrong...Also it would be helpful to include the location of the file like I've started doing in my posts...

ie Game Title (Game Id) - filePath
I hope I'm not being picky, but the latest version (3.1a) cannot load any file (trying to do so results in a null-pointer (and subsequent errors));
Code: [Select]
Exception in thread "AWT-EventQueue-0" java.lang.ExceptionInInitializerError
        at util.gui.CDragDrop.drop(CDragDrop.java:279)
        at java.awt.dnd.DropTarget.drop(Unknown Source)
        at sun.awt.dnd.SunDropTargetContextPeer.processDropMessage(Unknown Source)
        at sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher.dispatchDropEvent(Unknown Source)
        at sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher.dispatchEvent(Unknown Source)
        at sun.awt.dnd.SunDropTargetEvent.dispatch(Unknown Source)
        at java.awt.Component.dispatchEventImpl(Unknown Source)
        at java.awt.Container.dispatchEventImpl(Unknown Source)
        at java.awt.Component.dispatchEvent(Unknown Source)
        at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
        at java.awt.LightweightDispatcher.processDropTargetEvent(Unknown Source)
        at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
        at java.awt.Container.dispatchEventImpl(Unknown Source)
        at java.awt.Window.dispatchEventImpl(Unknown Source)
        at java.awt.Component.dispatchEvent(Unknown Source)
        at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
        at java.awt.EventQueue.access$000(Unknown Source)
        at java.awt.EventQueue$1.run(Unknown Source)
        at java.awt.EventQueue$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
        at java.awt.EventQueue$2.run(Unknown Source)
        at java.awt.EventQueue$2.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
        at java.awt.EventQueue.dispatchEvent(Unknown Source)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
        at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
        at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
        at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
        at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: java.lang.NullPointerException
        at util.FileAccess.read(FileAccess.java:53)
        at util.struct.DPath.read(DPath.java:190)
        at util.struct.DPath.read(DPath.java:149)
        at util.Plugin.scan(Plugin.java:80)
        at util.struct.DPath.check(DPath.java:62)
        at util.struct.DPath.<init>(DPath.java:31)
        at util.struct.DPath.<init>(DPath.java:22)
        at util.struct.DPalette.<init>(DPalette.java:9)
        at ctool.Loader.<clinit>(Loader.java:14)
        ... 33 more

Also, if you truly want large NSCRs to test with, try the ones found in Izuna - Legend of the Unemployed Ninja (AHSP) -> contents/data/gra/village/
(eg; Dungeon01BG1.NSCR (http://img.photobucket.com/albums/v175/barubary/Dungeon01BG1NSCR.png), which is 1280x1280)

Apparently the internal screen size functions as something to make large, composite tiles. When loading a large NSCR, a sequence of those internal screens are filled with those tiles. To display the full NSCR, those screens should be laid out in a standard tiling fashion (row by row). At least, that is how I got the example image from Izuna given above.
Note that only the first 2 bytes of the 4 labeled as 'Unknown' in your documents indicate the internal screen size; the other two are supposed to be the 'BG Type', which I think only indicates if the game should rotate/scale it at some point or not.


Also, regarding the compression of 'overlay' files; The format is indeed like the built-in LZ with 'type' 0x10 read backwards, but has a different header as well as a slight modification to the algorithm.
The 'header' format is:
Code: [Select]
{
    u8* padding; // 0xFF-s
   
    // length of compressed data. May be less than the file size (minus header size).
    // Same format as the decompressed size in the 'normal' compressed file formats.
    'u24' compressedLength;
   
    u8 headerSize;
   
    // The size of the decompressed file is the original file size plus this value.
    u32 extraSize;
}
The data is located in front of this 'header', so 'extraSize' is the last 4 bytes of the file. When 'extraSize' is 0, the rest of the header is absent.
Only the 'compressedLength' bytes directly before the header are compressed data, anything before that is uncompressed.

The algorithm is slightly modified in the sense that the 'Disp' given in the compressed blocks is 2 less than the 'Disp' that would have been given for the regular algorithm (so just add 2). It also appears to occur regularly that the 'Disp' is far too large for the amount of data that has been decompressed up to that point; in that case I've found that setting it to 1 gives results that appear to be correct.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 25, 2011, 11:08:26 am
Try deleting the settings.ini...If that doesn't work I've re-uploaded another version.

3.1 has an unstable gui plain and simple, it was the main reason why I didn't really want to release any new updates for it. The latest version I've been working on has a much more stable gui but it hasn't got all the functionality of 3.1 as I'm still progressively re-adding stuff to make sure it's stable and is more efficient where possible. I've tried to follow a much more strict MVC model as well as fixing a lot of class dependencies which is one of the major reasons why I wasn't able to just simply copy and paste stuff from 3.1. Though a large portion of the code is still just copy and paste I have to go through an clean up bits and pieces. It's simple stuff that takes a bit of time to do, which I was just not aware of until I started doing more IT units at University this semester :p

Unfortunately I've also got studies to tend to, so work on the next version has somewhat slowed...I only applied those updates because they were fairly simple and greatly enhanced the usability of the program. I've also been spending the last couple of days setting up Windows 7 through Parallels on my Mac so it's been kind of a bit hectic :p

I always look forward to your input though, it makes my assumptions look either overly complex or uninformed XD I shall have a good look through it (and all of your other posts) more in-depth when I get the chance :)
Title: Re: The Console Tool (by Low Lines)
Post by: Garyong on April 25, 2011, 11:49:37 am
The old-new version with deleted .ini produces the same error.
The new-new version produces a similar error (which also happens after I delete the .ini file again) =/
Code: [Select]
Exception in thread "AWT-EventQueue-0" java.lang.ExceptionInInitializerError
        at util.gui.CDragDrop.drop(CDragDrop.java:283)
        at java.awt.dnd.DropTarget.drop(Unknown Source)
[snip]

Caused by: java.lang.NullPointerException
        at util.FileAccess.endRead(FileAccess.java:62)
        at util.struct.DPath.closeSource(DPath.java:203)
        at util.Plugin.scan(Plugin.java:150)
        at util.struct.DPath.check(DPath.java:62)
        at util.struct.DPath.<init>(DPath.java:31)
        at util.struct.DPath.<init>(DPath.java:22)
        at util.struct.DPalette.<init>(DPalette.java:9)
        at ctool.Loader.<clinit>(Loader.java:14)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on April 25, 2011, 12:21:49 pm
Something is returning null at some point within the drag n drop and file loading process >_< I've put a few exception handlers in that allow it to continue working... It's just outright nasty -_-

The next version doesn't have this issue at all so there's not much point in debugging it further.

Also I loaded up those big NSCRs and they display awesomely :) Mind they have an internal screen size of type zero so until an NSCR for all 4 types has been tested there's no way to be certain :p
Title: Re: The Console Tool (by Low Lines)
Post by: chrrox on April 30, 2011, 09:10:13 am
With 3.1a i cant open any of the love plus files anymore.
Title: Re: The Console Tool (by Low Lines)
Post by: Teknik on May 05, 2011, 05:59:47 pm
awesome job !

you know if one day it's possible to view model from diddy kong racing ?
if i remember on n64 roms it's  .lev format ...
i don't know if it much easy on DS but i think actually it's not possible to view modèle from Diddy Kong racing or Mario Kart 64 .... it's only possible with Lemmy plugin but for diddy kong racing the result are very bad ^^
Title: Re: The Console Tool (by Low Lines)
Post by: chrrox on June 19, 2011, 05:53:49 pm
Can you release the source code i would like to continue on this file format no reason to duplicate the work you already did.
Title: Re: The Console Tool (by Low Lines)
Post by: Rhys on June 19, 2011, 06:00:58 pm
The availability of the source code has been discussed earlier in the thread, in summary Low Lines does not wish to release the source code at this time, and won't support you if you decompile it yourself
Title: Re: The Console Tool (by Low Lines)
Post by: Hamtaro126 on June 19, 2011, 06:56:54 pm
What Rhys and Low Lines said: It is currently a Closed-Source project,

Still interested in the project even then, Makes a future tool that is promising!
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on June 19, 2011, 09:53:52 pm
Source code doesn't always equal less work. For one it needs to be in a language you understand (9/10 it isn't in the language you want it as well) and it is also written without a lot of explanation or documentation by the author. Especially if it deals with more complex stuff, which is the main reason you usually want it. So in the most likely case, you'd end up with something that's more like translating another language rather than the highlighted path to your goal.

If you are just looking at improving our understanding of how a file format works, then documentation is all you should need. I've tried to document the collective understanding I have of most of the formats I've supported now and in the past, which you can find here (http://llref.emutalk.net/docs/ (http://llref.emutalk.net/docs/)). If something doesn't make sense, I'm more than happy to elaborate on it, and/or update the docs to reflect upon it.

@Project Status
I finished by exams last week and have about 3-4 weeks of holidays. At the moment I'm working on another programming project and jumping between a couple of games I've wanted to play after seeing this years E3 (the WiiU man! I'm gonna call it my WU (wooo)). Should get back to working on ctool soon though, as its been the main thing I've been looking forward to doing during this break.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on June 30, 2011, 09:22:45 pm
I eagerly await the next update.

But another thing, does the tool support keyboard? I don't know if it's my system, but it's kind of irritating not getting to use arrow keys to navigate.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on June 30, 2011, 09:36:54 pm
I eagerly await the next update.

But another thing, does the tool support keyboard? I don't know if it's my system, but it's kind of irritating not getting to use arrow keys to navigate.
There's has been next to no keyboard support to date, mostly due to the fact that I just haven't gotten round to it. I'm not quite sure where you'd be wanting to use arrow keys though, if you could please elaborate.
Title: Re: The Console Tool (by Low Lines)
Post by: Seb on July 01, 2011, 07:36:13 am
Like 5 months ago I was crazy enought to read all the pages :o I hope you never quit making this :thumbsup:
If you'd like to check it out, I've *managed* to extract some .szp Luigi's Mansion files :laugh: Maybe you already have but would you like to take a look into them? I only extracted model files so far :( They're in .mdl + some other 5 or so folders which I can't remember the name of right now...
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on July 01, 2011, 01:19:22 pm
There's has been next to no keyboard support to date, mostly due to the fact that I just haven't gotten round to it. I'm not quite sure where you'd be wanting to use arrow keys though, if you could please elaborate.
Well maybe it's just me, but I feel more comfortable navigating tree structures with the arrow keys. Arrows to open archives and browse through the archive up/down using the arrows. Maybe even pushing enter to open. Keystrokes are faster than mouse movements, and since the icons are small, this is actually easier to do.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on July 19, 2011, 12:06:02 am
Hi Lowlines,
I have a question and I am not sure if you can help me.
I am changing the number of cells in a NCER because I made some bigger and I want to delete the left over and soon I will need to add 1-2 cell to a NCER but I noticed that in both cases the game will display the graphic wrong and freeze.

Do you know what is the crucial value(s) that I need to change??
here a copy of the NANR:
524E414EFFFE0001
0F01000010000300
4B4E4241E0000000
0100110018000000
28000000B0000000
0000000000000000
1100000002000100
0100000000000000
000000000200EFBE
080000000200EFBE
000000000200EFBE
100000000200EFBE
000000000200EFBE
180000000200EFBE
000000000200EFBE
200000000200EFBE
000000000200EFBE
080000000200EFBE
000000000200EFBE
100000000200EFBE
000000000200EFBE
180000000400EFBE
000000000400EFBE
200000000400EFBE
000000000800EFBE
0000EFBE00000000
0000EFBE01000100
0000EFBEFFFFFFFF
0000EFBE0100FFFF
0000EFBEFFFF0100
4C42414C13000000000000006D61726B

It's the "shake" of the objection (take that, overruled, etc...) exclamation on screen in the Ace Attorney serie.

Thanks
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 19, 2011, 07:02:45 am
First off you got some conflicting information there...NANRs are for animation, so a data dump of them won't help with editing the cell layout. Secondly the manner in which you would go about editing a NCER would be dependent on the format/structure as there are several ways this is done. Lastly probably the best way to approach editing NCERs would be to decode them first into objects (thinking Java here), make the changes and then re-encode them back. Hex editing would be too unreliable because you are also dealing with offsets and such which might be where the crashing is happening.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on July 19, 2011, 02:25:40 pm
I got used to edit ncgr/ncer by hex paired with consoletool and cristaltile2 ... that's the only way to do it for now especially because i don't have programming skills.
All other works i have done is ok and working.

Now I have animation to take in consideration and what probably is the problem with the nanr, is the cell index or the bank ...i don't get that part :(
In a ncer a bank is a group of OAM (i call them cells :p or a layer) and in the nanr??
What is the cell index??
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 19, 2011, 05:03:04 pm
As far as I recall NANRs only require a cell index for each frame, ie Display cell 0 for 0.5 secs then load and display cell 1 for 0.5 secs, so as long as you haven't modified the number of cells in the NCER they should still work properly.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on July 20, 2011, 12:19:33 am
OMG...is my english so bad??

For what I have done till now, it was ok to not change the number of cell in the NCER but for my future work I will need to change the number of cells to expand the text so I will need to change the NANR too but I not understand how I will need to change it.
What I not see is the connection between NANR and NCER in the NANR or if you want, what byte in the NANR call something from the NCER.
Let's parse the NANR together:

Nitro Header
524E414EFFFE0001
0F01000010000300 0100F>file size   10 header size    03> sections

Animation BaNK Section (ABNK)
4B4E4241E0000000
0100110018000000
28000000B0000000 E0> section size   01> banks    x11=17 frames and the 3 offset to bank/header/data block

Bank Block
1100000002000100
0100000000000000 x11=17 frames   x02> 8bytes data type and the 2 unknow values (my is 0101 and not 0102 :o)

Header Block
000000000200EFBE
080000000200EFBE
000000000200EFBE
100000000200EFBE
000000000200EFBE
180000000200EFBE
000000000200EFBE
200000000200EFBE
000000000200EFBE
080000000200EFBE
000000000200EFBE
100000000200EFBE
000000000200EFBE
180000000400EFBE
000000000400EFBE
200000000400EFBE
000000000800EFBE each line is a frame (with duration) and each of this lines calls one of the lines in the next block

Data Block (type 2)
0000EFBE00000000
0000EFBE01000100
0000EFBEFFFFFFFF
0000EFBE0100FFFF
0000EFBEFFFF0100 basically move all the cells (found in offset 0000 in the NCER cell bank*) from x,y=0,0 to the various combinations of (+1,-1) passing by 0,0 for every second frame.

Next block is just label.

If everthing is correct in my parsing, it should not matter how many cells I have in the NCER because the first cell will always be at 0000 in the NCER cell bank*.
Or where i do a mistake??


PS: I forgot to post a suggestion....a zoom in the viewer. When trying to look small images especially to check if there are misplaced pixels after an edit :p
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 20, 2011, 06:19:16 am
As stated on my blog, I just discovered that all my online docs on these file formats have reverted to the outdated versions from earlier this year. This probably happened during the last server downtime that happened a little while back (blame the retards that get off hacking into websites >_>). I've re-uploaded the updated versions so if you were at all using them as a reference, it might be a bit more clearer now (since the changes incorporate Garyong's notes), however other than that I can't help much since my studies have started again.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on July 20, 2011, 05:12:53 pm
umm..understood.
I just gave a quick look at it and it doesn't change anything for me :(
Thanks anyway.


UPDATE: It was my mistake, I was messing up the file size, the section size and there was 2 unneeded bytes just before the LALB in the NCER and not a problem in the NANR.
CT was showing everything right but the sum of all 3 things make the animation go gaga :)



July 23, 2011, 03:26:03 am - (Auto Merged - Double Posts are not allowed before 7 days.)
Hi Lowlines,
I am here to give other suggestions to Console Tool apart from the zoom in the last post.
First of all a "reload" button for the file loaded.
If I open a NCLR+NCGR+NCER (like I am doing in my work now) and while i have CT open (as reference or to check my last changes), I modify the NCER with an hexeditor, nothing happens in CT. For now i have to close the NCER (after edit it with the hex editor) and re-navigate to open it again. A reload button would be handy.

Second, that we can view more banks (layer) at the same time (something like the eye in photoshop).


This "Begin Investigation" (a montage here made in paint):
(http://img171.imageshack.us/img171/9664/begini.png) (http://imageshack.us/photo/my-images/171/begini.png/)
is splitted in 8 OAMs:
(http://img560.imageshack.us/img560/2530/ibegin.png) (http://imageshack.us/photo/my-images/560/ibegin.png/)
and divided in 4 different banks O.o

It was not so easy to align the OAMs.

I am looking forward to the new version of CT but I know it will take a while because of school. :woot!:


Title: Re: The Console Tool (by Low Lines)
Post by: DeathbyGiantChicken on July 26, 2011, 08:35:52 am
I've been running into a snag when trying to use this tool with the Ghost Trick rom.

The compressed files are in .lz format and have [1LMG] after the names. I'm aware that LZ files, in hex view, begin with something like 0x100. With these files though, they start with the text '1LMG'. Is this another type of LZ77 compression? I don't seem to be able to extract these files.

I've tried Googling for '1LMG' but haven't found anything.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on July 26, 2011, 09:37:23 am
@DeathbyGiantChicken
Normally magic numbers mean you are dealing with a structured format, and when you get a magic number and no google search results, then that probably means that it's an undocumented (private) proprietary format. File extensions generally share little relation to what is actually contained in a file. If you post a few sample files and/or post the locations of the files in the rom (ie folder1/folder2/file.lz) people can have a look at them, but this will only really happen if there is some level of interest as there are tons of custom file formats out there. As I've mentioned before my goal is to eventually fully support SDK and common file formats and implement a plugin system so people can write their own plugins for custom formats. Its a long ways off from that but if I'm given a spec on a file format that I can decode into something that can be loaded into ctool, I'll add it.

@Auryn
Zoom, and  AUTO-reload (ie a popup that says "hey you changed this file without my knowing!! do you want to reload it?") are some good suggestions. Replacing the banking combobox with a layers styled ui is sort of what I'd like to do with it, however I was considering doing something more like this (http://www.lonerobot.com/images/FBHH.jpg (http://www.lonerobot.com/images/FBHH.jpg), note this is just an example I found with a quick google search), but I can also see what you mean with the bank layering.
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on July 29, 2011, 09:34:04 am
could you if possible add this format I need the models for an important project
http://ps23dformat.wikispaces.com/Ace+Combat+4
only if its not to much of an issue this is coming from someone with only a basic idea of how this works
Title: Re: The Console Tool (by Low Lines)
Post by: AzuShin on July 31, 2011, 09:40:18 pm
Would you mind adding a another support, if you can?
.DAT
->.CAB
-->.BIN
-->.BLD
http://dl.dropbox.com/u/32409323/Rips/TOPX_DAT_BIN_BLD/e001%281%29.7z

The .dat has a FPS4 header, which FPS4Dumper can dump.
Then there's a cab file, which any windows, mac, etc tool and uncompressed.
The .BIN, I'm think this the cell data, like the NCER (http://llref.emutalk.net/docs/?file=xml/ncer.xml) files in NDS games.
The .BLD is the graphic and palette. You can view it TitledGGD, but piece those together would be nearly impossible, or at least very time consuming.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on August 01, 2011, 12:05:28 am
I am back to make critics / precisations for my own suggestions :p
The auto reload should only check if the loaded file has changed when the CT window becomes active.
So that CT not starts bothering about file changed as soon that somebody change a value in an hex editor.

The multi layer thing is relativelly usefull especially if the layers are not loaded to the same coordinates (like I later noticed with my work :p).

@viperzerof-2/AzuShin: CT want to be a general tool for all console but I don't think Lowlines will start implement support for specific games very soon.
As for the formats from you listed, .dat/.bin can be anything and in the most games they are archives in special format created for that game so the same applies to the above sentence.
Anyway Lowlines is busy with school so there will probably not be a new version of CT before fall/Christmas.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 15, 2011, 09:51:24 pm
(http://img20.imageshack.us/img20/4332/bosspackunagain.png)
It's always good to have a bit of nostalgia :)

I have got NSBMD models nearly up to the same level as was in v2.0, however it's taking a little while since I'm introducing proper implementations of fixed point numbers. With that said there's already a handful of things I'll have to update in the format spec later once I get these little buggers fully working again and figure out the unknown parts. If anyone can point me to a good tutorial for matrix decomposition (ie where you separate translation, rotation, pivot from a combined 4x4 matrix into their native/raw values, ie xyz) that'd be a great help.

Putting that aside, I'm happy to say that I've gotten pretty much every archive and image/graphic format that has ever been supported in earlier versions into the current build, including partial support for a bunch of others such as PowerVRTextures and Valve VTFs and S3 Texture Compression. All up that works out to be about 30 formats that at the very least will open in ctool. I've now got some reasonably good color/image parsing routines that make adding support non-compressed images a snap :)

Autodesk 3DS models open, though as I'm still in the process of refining the generic modelling data structures, they currently only display wireframes.

Am still having a bit of trouble gettings ISOs working properly (last time I tried, I got folders that opened in on themselves to inifinity O.o), but they aren't a priority atm. On a side note, keyboard navigation in treeviews is now implemented (including hitting enter to open) ;)

The Hex Viewer has also had a bit of work done too, with proper interaction with the editor panes (highlighting, selection, wip editing support), as well as a data inspector for converting values. There's still a lot more I want to do with it however it is at the far end of my priority list right now.

Finally, after brushing up my knowledge on class reflection, ctool now automatically loads all format plugins dynamically (meaning I don't have to add each new plugin to a list when I create a new class), which although currently happens internally, opens up for the possibility of a plugin based system (where plugins are loaded in from a folder) as I had always hoped to be able to do down the track. Again this isn't a real priority right now,but at the very least its something to think about :p

NCER, NANRs and the other SDK formats haven't been added yet as there's still a few issues with assets management that I have to sort out before I can do them. The idea at least is that I'm making use of class inheritance so an NANR could be treated the same as an NCGR in the sense that when you get to more complicated formats like NMCR, all you have to worry about is calling your assets' .getImage(); method and it will "magically" return the correct image without over complicating the implementation. It is also meant to reduce all graphic editors into one, which was one of the major problems I ran into in past implementations. All a bit complicated atm, hence why I'm working on NSBMDs right now :p

With all that being said, I've got assignments to do and not a heck of a lot of time to do them @_@ but I quite like mention about an update in Fall. Living in a place where we don't have the four seasons, I'm used to hearing that term with movie and game releases. So..."CTOOL IS COMING THIS FALL!!" or at least as long as I get what needs to be done until then :p
Title: Re: The Console Tool (by Low Lines)
Post by: Friedslick6 on August 16, 2011, 03:12:16 am
Oh yay! Another update~
Thank you for your continuous work on this project. I hope you can find the time and resources neccessary to work on it further :) .
Title: Re: The Console Tool (by Low Lines)
Post by: RetroHelix on August 16, 2011, 03:57:27 am
I'm quite sure I mentioned this in the context with something else but have you heared of quickbms (http://aluigi.org/quickbms.htm)? Maybe you could implement support for mexscript/bms into your tool.

May console tool become better then Señor Casaroja's Noesis (in terms of supported formats). Keep it up. :thumbsup:
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 16, 2011, 12:00:54 pm
That modelling tool looks cool despite the dodgy look and feel of the website :p I also think that I'd be an old man before I will ever be able to support that many formats :p On a side note, I dislike the use of DLLs and how they behave, putting aside the fact that I would have no clue on how to implement them even if I wanted to :p Any sort of third-party plugin support would probably be in the form of scripts or Java JAR files. This would also then reduce the opportunity of bad stuff happening as would be the case with DLLs if I understand it all correctly.

The QuickBMS on the other hand looks really interesting!! The BMS language looks pretty easy to implement, however scripts would need some extra header information in order to work with ctool (hypothetically speaking of course).

Here's example of the current structure of a file format class...

Code: [Select]
package ctool.file.formats;

import util.file.*;
import util.file.defaults.*;
import static util.Common.*;
//Imports a bunch of convinence methods mostly for quick and easy data conversion
//ie
//int i = getInt(byte[] data, int offset, int length);
//float f = getFloat(byte[] data, int offset, int length);
//String s = getString(byte[] data, int offset, int length);

/**
 *
 * @author lowlines
 */
public class CustomFormat extends util.file.formats.DFormat /*or any of its subtypes, ie DArchive, DImage, DPalette*/ {
public CustomFormat(DPath path) {
//DPath is a super class of DFormat and is basically your handle on FileIO in ctool
//In its basic form, it points to a File, otherwise it will recursively call to its parent
super(path, int fileType, String mimeType, String formatName, String extensions/*(separated by commas)*/, int byteOrder, CIcon customIcon);
//The are various super constructors you can use, but this the most robust one so you can see what information ctool currently uses
}

@Override
public boolean isValid() {
//This by default checks if the file has a valid file extension but you can put your custom detection here (ie hasMagic("NCGR");,
//which is a built in method that can be called)
}

protected void openFormat() {
//Code goes here
//File handling is all done through inherited methods
//ie read(), read(long offset, long length), read(long offset, long length, boolean decompressed)
}

protected void exportFormat() {
//Haven't implemented exporting yet but this would most likely be how you would do so
}
}

Being file objects, the DFormat class uses a parent/child hierarchy making it very easy build complex data structures like archives (and should make editing them a breeze), or for adding internal assets (ie a palette inside an image). Its still a work in progress, but the whole structure has come a LONG way from where it was back in the beginning. Any plugin support would have to implement all functionality of this structure in some way in order to work. Just off the top of my head, BMS scripts could have a couple of word sensitive comments at the top of the file that would hold the header information necessary. I might have a go at writing a BMS script loader later, it would be super cool if I get it working because that would potentially mean I've added support for a bunch of formats without doing anything XD

[edit]

After a bit of researching I've gotten a good handle on BMS. However there's a fair bit of variation in the language including undocumented commands etc. It's definitely a viable plugin format that can (and probably will) be supported in the future however as most of the BMS files I've come across so far deal with files in games I don't have access to or make use of various compression and encryptions that aren't yet supported, it's a bit difficult to debug a script loader for it (since the goal here is to support scripts already written by people). So if anyone could pm me files of supported BMS formats that could really help that along. It would be good to have a mix of simple and complex scripts to test with.
Title: Re: The Console Tool (by Low Lines)
Post by: Blazer on August 17, 2011, 01:14:19 pm
Hello there,

This is an awesome looking tool so far and it has a lot of support for a lot of files. However, I'm saddened by the fact that it doesn't support .ca, .ma, and .md files. These work similarly to .nsbmd and .nsbca files (I don't know about the .ma files, unfortunately).

I used your doc (I do think it's yours) here--http://llref.emutalk.net/docs/?file=xml/bmd0.xml#xml-doc and dissected the header of a file ripped from Fire Emblem 12, a game I'm currently translating. I need to translate the logo and the opening which use the aforementioned file formats; however, no program supports them despite the format of the .md file being so similar to the .nsbmd file. If you could somehow give support for these files, I would appreciate it very much.

I actually made a post asking for help but did not get anything so I continued to research even harder and found myself here. Here is the link to the post (http://gbatemp.net/t305405-texture-and-model-data-hacking) which includes screenshots and downloads for files of the file types I mentioned, as they may prove useful.

If you could at least give any information on how I might edit them, it would be greatly appreciated. Regardless of whether you're able to help or not, your work is very well done, and I hope you continue to make progress in your endeavors. ^_^

~ Blazer
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 17, 2011, 01:21:47 pm
They look like they're compressed to me, the 0x11 first byte is a dead give away :p Doesn't the latest version (v3.0 or v3.1) decompress them properly?
Title: Re: The Console Tool (by Low Lines)
Post by: Blazer on August 17, 2011, 07:32:19 pm
I already decompressed them, of course. It's not about decompressing it as much as it is finding some way to edit it. The decompressed graphics are shown in that link I gave. And your program doesn't like any files that aren't the supported file formats IIRC from what I tried.

Perhaps this is out of the scope of your program, in which case I apologize
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 17, 2011, 09:11:19 pm
I am 99.99% positive they are SDK formats compressed with the LZ77 variation used in NDS games.

(http://img15.imageshack.us/img15/1095/fireemblem.png)

The model is some sort of image blown apart which I'm guessing comes back together in the animation. Assuming you don't have any problems with decompressing they should load up in v2.0 just fine, (note this is last public version where models and textures were supported). Obviously you can't edit them with ctool yet, but my point is these aren't custom formats.

An NSBMD is just a container which can have a MDL0 section and sometimes a TEX0 section. If you understand the Nitro container structure its actually quite easy to split them apart and put your texture in a NSBTX file in a hex editor, that way at least you aren't dealing with all the model info as well.

Probably what the real issue you are dealing with is that about half of the textures are compressed using the NDS texture compression format which is a lossy format like JPEGs. So not only would you have to export them to native images but you would then need a compression routine to compress them back into the appropriate format, otherwise it'd be like having a BMP in your texture files :p As far as I know, no ones made a compressor yet but it's on my to do list somewhere down the track.
Title: Re: The Console Tool (by Low Lines)
Post by: Blazer on August 18, 2011, 06:20:10 pm
Sorry, I only meant that these formats were something I was unfamiliar with and couldn't find any documentation on and couldn't figure out myself. I'm not a veteran or experienced hacker, I just kind of used logic to figure things out, and these things were out of the range of my logic >_>

Quote
Probably what the real issue you are dealing with is that about half of the textures are compressed using the NDS texture compression format which is a lossy format like JPEGs. So not only would you have to export them to native images but you would then need a compression routine to compress them back into the appropriate format, otherwise it'd be like having a BMP in your texture files :p As far as I know, no ones made a compressor yet but it's on my to do list somewhere down the track.

So uh, let me try and understand this: the whole file is compressed, and within that are compressed textures using some compression that doesn't have a compressor program made for it (and possibly not even a decompressor). Even if I could decompress it, inserting it like any other graphic wouldn't work, so I need the compressor too...

I think I got a few details wrong, I'm sorry, this is not an area I'm familiar with. If you could clarify this a little bit more, I'd really appreciate it so I could learn this stuff. Thank you for your patience and help. ._.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 18, 2011, 06:45:58 pm
File compression the way I see it is where the goal is to reduce the file size as much as possible without the loss of data. This is generally utilized when you have lots of repeated strings of data (ie lots of 00s etc), the same technique is also used in lossless image formats such as PNG. Lossy compression, which is something used in images and audio (obviously including video as well) is where the goal is to compress the data, losing some of the data at the same time in order to get a smaller image but is still visibly acceptable/appealing. If you recall seeing JPEGs with lots of like specky dirt on them, that's the result of lossy compression and the more it is compressed, especially when the process is repeated on the same image, the worser it gets. This can be done in a variety of ways making it rather difficult to just edit an image in a tile editor since you aren't dealing with raw information.

ctool can decompress some some texture formats which includes the texture compression format used in NDS games, however atm the only option you have with them is to save a raw copy of the image to the clipboard, thus losing palette information and so forth. As I've already pointed out, you'll probably have to jump between versions in order to get to this point as certain functionality was only supported in specific past versions. This will all change of course starting from the next version (at least I am hoping so), but for now that's the best I can offer to your problem.

I'm also not a real expert with compression algorithms so I'm quite happy if I'm just able to decompress something with 100s of errors. But to write a proper compressor (even a simple one) will be a lot of work and I most certainly don't plan on racking my brain anytime soon :/

But although re-compressing your edited graphics might not be feasible atm you can still technically modify them, it just that you'd have to either save them in raw 16-bit color format (that's RGB5 with no alpha) or as 256-color paletted image. Both instances you obviously lose your alpha channel (although I don't think this specific file uses alpha anyways), and generally will have a larger file size too.

[edit]

On closer inspect, that file actually has a LOT of alpha channel usage including translucent effects. Your best option would be to create the translated graphics from scratch and then convert them in (I'm assume you're doing a translation here?).

[edit 2]

Here's the texture data extracted as an NSBTX.
http://www.mediafire.com/?zywh7m1r6qje7na (http://www.mediafire.com/?zywh7m1r6qje7na)

[edit 3]

In an effort to better understand 3D Modelling in the NDS I've sort of branched out and had a look at some other games with a strong emphasis on 3D. For now this would include Star Fox Command and Kingdom Hearts 368/2 Days.

Star Fox Command has a lot of raw data files including a custom container format with the magic stamp "chnk". It's an odd format in that every node is preceeded with the chnk header, which both defines the objects type and where or not it is a sub chnk or a data file. All data types a preceeded with "MGL" which I'm guessing is the short name of the sdk used to build the game.

So far I have figured out MGL Palettes, Textures, and have partially figured out Mesh files which are actually 90% DS geometry commands making them fairly easy to add support for. The fact that even header chunks in Mesh files were actually command dumps got me looking at unknown data in the Official SDK formats with that in mind and has helped me figure out a few more things :)

Moving on to KH368/2 Days there is an absolute ton of Model resources to sift through in it, most of which are hidden away in countless container formats. To that end, I've started to put together a few specs on the container formats, since a general google search turned up next to nothing or very poorly written documentation.

Code: [Select]
KAPH/D2KP
$0000 4 Magic "KAPH" or "D2KP"
$0004 4 Padding[0]
$0008 4 0 FAT
$000C 4 1 FAT
$0010 4 2 FAT
$0014 4 3 FAT
$0018 4 4 FAT
$001C 4 5 FAT
$0020 4 6 FAT
$0024 4 7 FAT
//
$0000 4 Number of Files
// Repeats * Number of Files
// Offsets Block
$0004 4 Offset
// Lengths Block
$0008 4 Length

// Types for KAPH
0:NSBCA
1:NSBVA
2:NSBMA
3:NSBTP
4:NSBTA
5:
6:
7:NSBMD

// Types for D2KP
0:NCLR
1:NCGR
2:
3:NCER
4:
5:NANR
6:NSCR
7:

A few things to note is that KAPH files deal with 3D and D2KP deal with 2D, but the file structure is identical other than what files are stored in each FAT table.

Code: [Select]
P2
$0000 2 Magic "P2"
$0002 1 Number of Files
$0003 1 Has Name Table Flag (00h=No, 80h=Yes)
$0004 8 Padding[00h]
$000C 4 Header Size (0x200, 0x400, etc)

//Offsets
Offsets Table = 0x10
Lengths Table = offTable + Math.ceil(numFiles/2) * 2 * 2
Names Table = lenTable + numFiles * 4

// Repeat * Number of Files
// Offsets Table
$0000 2 File Offset * 0x200 + headerSize
// Lengths Table
$0000 4 File Length & 0xFFFFFF
Type (>> 24) & 0xFF (00h or 80h)
// Names Table (Not always present!!)
$0000 8 File Name

This was a bit of a pain to figure out and still might be somewhat buggy, but as far as I can tell this should work on most P2 files.

Code: [Select]
0C Archive
$0000 4 Magic 0x0C000000
$0004 4 Length 1
$0008 4 Length 2 (Seems to coincide with File Size)
$000C 4 Number of Files

// Each file begins with a header block which allows
// the parser to jump to the next file in the archive
// until Number of Files is reached

// File Header
$0000 4 File Size (includes 8 byte header)
$0004 4 File ID

Now looking  at the models about 80-90% have at least one spastic vertex with others that look terrible. This could be happening due to a number of reasons including the likely possibility that the rendering order which ctool loads them is wrong (ie a polygon is drawn before the matrix stack is loaded appropriately).

There is also the bone command 09h. Although some models that have it still appear to load properly, I'm sure this command has some significance due to the data length. It is a variable length command which from what I can tell must somehow load/copy/clone other stacks into empty stacks with some possible calculations done on them in the process.

Here's a sample 09h command:
Code: [Select]
09 05 03 010140 040540 030480

09h Command ID
05h Target Stack ID
03h # Sources

// Source Data 1
01h Source Stack ID
01h ?? ID
40h ??

// Source Data 2
04h Source Stack ID
05h ?? ID
40h ??

// Source Data 3
03h Source Stack ID
04h ?? ID
80h ??

The number of sources can be either 2 or 3 as far as I've so far seen. Just looking at the data, its clear it wants to do something with the stack, as the stack IDs all refer to previously assigned stacks, however its the last two variables in each source data that's got me stumped. Particularly the last variable, which can have a seriously varying value.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on August 29, 2011, 10:17:23 am
Games developed by Griptonite store most of their files in a Glob.bin, of which I have figured out the file structure for.

Code: [Select]
Griptonite Glob.bin

$0000 4 Header Size / First File Offset
//Number of Files = Header Size/4-1
//File Offset * Number of Files
$0000 4 File Offset
//File Length can be calculated by finding the difference
//of the current and previous file offsets

//The GlobFiles.txt contains names for all files in container,
//it can be found by checking for a size match (numFiles*0x40)
//Each name takes up 0x40 bytes with a 0xA as the last byte
//and whitespace (0x20) after the name

Interestingly though I was having a look at Green Lantern - Rise of the Manhunters I came across some interesting files.

These files, although labelled as common SDK formats don't read like they're supposed to and with a quick look at the hex it looks as though they use some kind of byte/block swapping to make it more difficult to read them.

Here's a sample file...
http://www.mediafire.com/?4t556x0vu42xda5 (http://www.mediafire.com/?4t556x0vu42xda5)

At a glance this file is NSBTX texture file that was LZ77 compressed and then byte/block swapped with a 0x60 added to the start of the file. You should also be able to spot in a hex viewer the BTX0 stamp (and if you're familiar with the format header, parts of that too), all jumbled or written backwards. Even LZ77 compressed generally most of the header is stored uncompressed making it easy to spot regardless of how the file is obfuscated.

If anyone is keen at all to have a look what I'd like to know is whether this a specifically some custom obfuscation as I assume it is or is it an actual/known compression format?

Also the reason why I was looking into this was because the Ben 10 games generally have a good variation of good/bad models to test with.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on August 29, 2011, 12:41:23 pm
Looks like a neat 3D viewer. I can't wait to try it out.

Fall, you say?
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on October 25, 2011, 11:18:49 am
Think you can throw in some support for the utility.bin format? It should be a 15 min job. It's another of those file formats that contains a nds style filesystem. The format is very easy: fntOffset, fntSize, fatOffset, fatSize.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 26, 2011, 09:53:41 am
Added support for Utility.bin, and several archive formats in Harvest Moon: Island of Happiness / Sunshine Islands, which I had to figure out in an attempt to rip all the sprites (and game data) for my own play through of the game since there's a serious lacking in a decent online guide (with images) on the web. Through that I discovered it was pretty easy to build a batch exporter for graphics. Although it's currently hard coded specifically for what I needed it for, I would like to eventually add support for customizable exporter functionality in ctool down the track, I'm just not sure on the best approach as of yet. Right now I can see two options, an action based system like photoshop or a script based one that would allow for use of regexp and filter options to make customized export routines.

Since I've had other stuff to do, I haven't really done much work on ctool over the last couple of months, and I probably won't work much on it until I've played all the games that have finally come out (like Skyward Sword ^_^), which will probably get me stuck back into ctool once I've played them all. I am on holidays (well sort of) so I plan on working on it over the next couple of months until uni starts up again in March.

Now to the thing I said about coming this Fall...As to the reasons mentioned above, ctool obviously isn't at the level I wanted it to be before I release the next version, HOWEVER as I have sort of promised a release before the end of the year, I thought of a compromise :)

I will *try to* add support for any file formats requested here on this thread from now until I choose to release the next version (which will be before 2012). This means any archives, palettes, images, textures or models that you would like supported. I'd prefer documentation, but I will look at undocumented formats too. Adding support for new formats takes far less time and effort than doing more work on ctool core functionality, so I think people would get more out of it than me just working on what I've prioritized up until the next release.

Right now NCER support is still being worked on, so NANRs may or may not be supported come the next release, but I will also get image exporting working too. Exporting as in ripping them as flat PNGs. I will also make sure to have archive exporting added too.

Any requested formats that do not get supported come the next release will go on a to-do list which I'll try and work my way through for future releases.

I still plan on working through my priority list in the meantime, but this way I can spend more time working it. Note that the list below is just to show what my plans are for ctool, so take it more as food for thought down the track.

1. Get all past format functionality working (NCER, NANRs, etc), this is purely getting all past formats viewing at the same level or above to what they were in past versions!!
2. Add viewer support for 3D file formats (models, animations, collisions, etc)
3. Add edit/export/convert/create functionality for all format types. Note: This includes UI work and various other things and is going to take a long time to do properly, especially for 3d models!
4. Work on Hex Editor
5. Add built-in tools/widgets. Ideas I have in mind atm include:
- Tile Viewer/Editor add-on for the Hex Editor
- Built in calculator (modelled after the XP calculator).
- Diagnostics Tools (ie archive/iso file allocation, usage statistics, etc)
- Any other small tools that might be useful for hacking. I'm up for suggestions, but keep in mind this is last on my priorities!
Title: Re: The Console Tool (by Low Lines)
Post by: samurai goroh on November 26, 2011, 02:54:08 pm
Added support for Utility.bin, and several archive formats in Harvest Moon: Island of Happiness / Sunshine Islands, which I had to figure out in an attempt to rip all the sprites (and game data) for my own play through of the game since there's a serious lacking in a decent online guide (with images) on the web.
Well, I know of a page that has good information regarding those games (and HM games in general), perhaps you already know about it, but any case, these are the links...:
Harvest Moon: Island of Happiness (http://www.fogu.com/hm7/)
Harvest Moon: Sunshine Islands (http://www.fogu.com/hm8/)
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on November 28, 2011, 08:23:09 pm
There is one thing that I hope that you are rewriting, the hexviewer. It's fucking slow for big files in 3.1a. Not to mention that the row highlighter seems to be off by one column.

I did some work on the professor Layton games a while back, here is the results.

Quote
File formats in the 3rd Professor Layton game

This document describes the file formats unique to the Proffessor Layton games.

Several of the files encountered in the games are compressed using the stock DS compression. This document describes the format of the uncompressed files.


Layton packages
Magic bytes: LPC2
Observed file extensions: *.cpck, *.cdat, *.cani
File type: Archive
Filenames: Yes
Endian: Little
Compression: Practically always compressed

The file begins with the 4 magic bytes. The fields in the file are all 4 bytes long.
The header structure can be summarized as the following c structure:
struct LPAC {
 u8 magic[4];
 u32 count;//Number of subfiles
 u32 dataStart;//The start of the file contents data
 u32 archiveFileSize;//The total size of the archive file, headers and padding included
 u32 fatOffset;//The position where the fat is located
 u32 fileNameBase;//The base value for file name pointers
 u32 fileContentsBase;//The base value for the file contents pointers
}

The fat is composed of as many records as the count field indicates and can be expressed as the following c structure:
struct FileRecord {
 u32 nameOffset;
 u32 contentsOffset;
 u32 fileSize;
}
In order to get the absolute file name and file contents offsets you have to add the base values from the archive file header. The file name is stored as a null terminated string and uses an 8 bit text encoding.


Layton images:
Magic bytes: LIMG
Observed file extensions: *.cimg
File type: Image/Animation?
Endian: Little
Compression: Practically always compressed

Layton script:
Magic bytes: LSCR
Observed file extensions: *.cbin, *.lbin
File type: Game script
Endian: Little
Compression: Practically always compressed

Credits:
* Henke37, author of this document and lead researcher
* HCS, generally nice guy and specifically a great help in this research

I also had a go at hotel dusk, but I never did figure out much about those tbl/wpf archives. It's a fixed size record format that should be obvious from a quick peek at it. But I never figured out how to unpack the actual file data.

As for generic requests, do you think that you could throw in a raw palette loader? Some games just dump the palette two bytes per color index instead of bothering with fancy stuff.

And of course, get those gk2 archives in please.

There is also that shared font format. Given how frequent it is it would be nice to have support for that. Pity that I couldn't find any documentation better than sourcecode.

Also, basic SDAT support. Just unpacking it is a good start. Playing the samples back is even better. But playing sequences back is just unreasonable.

There also seems to be a common text format that begins with the file header MESG (Brain Age among other titles).

Finally, add support for accessing those overlays and reading the banner.

Also, have you seen those dtb/hmm/mdc/phn/str/tree files, what is up with those?

Yes, that was a long list, but I know that most if not all of these are easy and are quite possible for you to do.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 29, 2011, 07:13:06 am
LPC2 archives are now supported.
I have a couple of suggestions on the format spec...

"dataStart" should be "headerSize" since it seems to match "fileContentsBase"
"fileNameBase" should be "fntOffset" just to keep the naming consistent
"fileContentsBase" should be "dataOffset" or "dataStart" going by your name conventions.

As for LIMGs
Code: [Select]
struct LIMG {
u8 magic[4];
u32 headerSize; //0x30
u16 unknown;
u16 unknown;
u16 unknown;
u16 unknown;
u16 unknown;
u16 constant?; //1024
u16 unknown;
u16 unknown;
u8 flags [24];
}
I'll have to check this one out in Tile Molester on my other laptop (working off a Mac so it makes things difficult), but the rest of the data looks like image data. Whether that's one image or several I'm not really sure yet.

For LSCRs...
Code: [Select]
struct LSCR {
u8 magic[4];
u16 count;
u16 headerSize;
u32 dataOffset;
u32 nameTableOffset;
}
This is followed by count * 8 byte blocks.
Code: [Select]
struct LSCRNode {
u16 attr1;
u16 attr2; //Possibly a count
u32 offset;
}
This is followed by data at the dataOffset.
And finally a name table at the end of the file. The names in the table look like file names, so maybe they specify which file to use with the script?

The hex viewer is a lot better now. I can scroll through a gamecube rom (1.4GB) no problem, and compressed files should be using a cache, so there shouldn't be any slowness their either.

Raw Palettes is a good idea, I'll have to look into that :p

As for NitroFonts I've been considering that myself, I just haven't got round to looking at them...yet. Same goes for MESG files.

SDATs are sort of openable as archives however it would appear there's still some bugs with it. As far playback and such goes, that's on my to-do list but I got to do up a GUI for that, which probably won't be done for the next release.

Overlays and the banner are accessible as raw files through the Archive viewer, though I haven't written a format spec for either yet (also wasn't aware overlays had a format spec).

Give me a quick refresh as to exactly which GK2 formats you are on about because I've forgotten :/
Same with "dtb/hmm/mdc/phn/str/tree" it's been a while... :/
I'll have a look at the Hotel Dusk stuff once I get a hold of the game.
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on November 29, 2011, 07:44:30 am
Overlays have one simple table for each cpu core and can be compressed with their own variant pf LZ77, it is a documented setup. But as for the data in them, that is not standardized and completely game specific.

As for the gk2 files, I wrote a document on those (http://www.romhacking.net/documents/547/).

The "dtb/hmm/mdc/phn/str/tree" stuff has been observed in Apollo justice as well as in Brain age. I have no idea what they do. My vague guess is that it's yet another archive file.

Interesting is that the MSEG files seem to be just one format of many with a similar setup. Kinda like the sectioned file format used by the graphics, but clearly a different structure. Check "Yoshi touch & go" for more files that seem to share it.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 29, 2011, 07:23:22 pm
My guess is that the "dtb/hmm/mdc/phn/str/tree" are to do with game translation, considering they're all kept in an "english" file in the American version of Apollo Justice. If that is so, then any games that have them must of a similar file system, from which these files would redirect to the appropriate files etc. Again this is all assumption here but it sort of makes sense that it would be something like that.

Hotel Dusk tbl/wpf look like they work together to create a virtual sub folder. The tbl file contains the relative file path and what I'm guessing are offsets. At first glance it just looks like a list of files, but there could be a hierarchy going on there. The wpf file contains the files each starting with a header with a matching relative file path plus what could possibly be duplicate data as is seen in the tbl file.

Have just added support for format tagging. Basically what this means is now you can target a format spec to a specific game (or series of games) by specifying the appropriate Game Code (ie YB3E for Harvest Moon: Sunshine Islands). It will scan a file's hierarchy to see if any of it's ancestors have the specified tag(s). This was needed to deal with custom formats in particular ones that don't have any kind of magic number and/or rely on filename conventions and the like. This can obviously be extended for other platforms in the future however for now I'm just focusing on NDS.

I've also re-implemented previewing in the Archive Viewer alongside file linking.
(http://img88.imageshack.us/img88/5682/ctoollinkage.png)

I am aware that the linking process apparently wasn't very obvious the last time I had implemented it, but until I have a better way to implement it, people will just have to deal with little "link" icons next to filenames :p All you have to do select your assets in the reverse order (Palette > Image > Cell, etc) and you'll know right away if you've done it right by looking at the preview. By double-clicking on the top asset (in the case of this example, that would be the Cell), it will open that file with the selected linked assets already included, so you don't have to open each file individually, just select them in the viewer :p I'm probably making it sound more complicated than it is, but oh well...

Textures also can be previewed, and any files that have internal assets (such as Textures which can have palettes stored in them), can be accessed straight from the Archive Viewer, as once you click on a file, this effectively is opening/parsing the file which is when internal assets get added as child nodes to their parent file. This isn't done automatically upon opening a folder/archive because that can create a bit of overhead.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 29, 2011, 11:12:28 pm
I still eagerly await a 3D model animation viewer.
On another note, would it be possible to force-open files as a certain data type? Like, I know some games use non-standard file extensions but their contents could just be raw graphics data. Another file could be raw palette. It would be a hassle to rename and the program alone has no way of knowing what type of data it is. A force-open as function would be useful here.
Title: Re: The Console Tool (by Low Lines)
Post by: Ryusui on November 29, 2011, 11:45:38 pm
At the very least, it might be useful to have the program check for magic IDs. Death Note: The Kira Game stores its graphic resources in standard formats, but packed inside archive-like files.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on November 30, 2011, 01:28:20 am
At the very least, it might be useful to have the program check for magic IDs. Death Note: The Kira Game stores its graphic resources in standard formats, but packed inside archive-like files.
Yeah, I was going to say something about this too. Some games have a generic archive format of 16-bit relative pointers. Unpacking these in-program to manipulate would be a very useful feature.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on November 30, 2011, 12:36:40 pm
The Apollo Justice and the other 3 first games of the serie use almost the same archive format as the one descrived by Henke. The problem is that there are more than one archive in one file meaning that you have a pointer table, the data, a new pointer table, new data etc... in one single file.
From what I remember from the first game, there is no pointer table that point to those subarchive (at least not inside that file).

Lowlines, did you get my PM??
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on November 30, 2011, 03:39:45 pm
@Auryn
That's great to know >_>
Can you guys provide me with a list of these Apollo games and a sample list of said files?

As far as your PM, I can't control how another program handles files. NCGRs are also not normally meant to be edited by themselves as they are normally used to for storing image data, not displaying/editing it directly. From your description (any a quick look at the files), I'd say they work in conjunction with an NCER file that doesn't use the most common method of storing image data. At this point in time ctool cannot edit files, so I can't help you much there.

As for your other questions, no idea, except for the last one which I've already stated the other day :p

@Koetsu & Ryusui
It's all good an well that having such features would be useful, the question is how you'd go about it. If I'm understanding it right, you're talking about an in file magic id scanner? What might be more useful is the ability to split files by manually specifying where the internal file(s) sit. A force-open feature sounds good too. I will see what I can do about implementing all three for the next version.
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on November 30, 2011, 05:36:59 pm
Yeah, just give up if the file is called romdata.bin and the game is a port from a platform without a filesystem. All the root pointers are going to be in the executable.
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on December 03, 2011, 07:29:32 pm
Well, a year ago or so, I worked a bit on the first game only but I was checking the other games as well and that kind of archive is more or less common across the whole serie of the Ace attorney (even Gyakuten Saiban Jiten aka anthology)/ Ace attorney investigation just with little changes.
I stay at work and I don't have roms/data in my hands but what i remember is this variations:
-one pointer table/one data block for jiten
-one pointer table/one data block with compression flag in the pointer table for AAI games (with extracted files that are sub archives with their own pointer table/data block) Henke can tell you more about this.
-many pointer tables/many data blocks in one file (for PW:AA, AATT, AJAA and AAJfA)

To find those files should be easy by exclusion analizing the files in the game aka in PWAA, a big data.bin.
AAI games have 4+4 bytes as pointer table (4 bytes offset + 4 bytes compression flag).
All other games should have just a simple 4bytes for offset.

Like I said, this is all out of my memory so this could be not exact.

For my problem, I know you can't control other programs :p
I will try to create a standard NCGR/NCER to see what happens or I am condamned to use the 1:1 export function of CT2 :(

For the Wii things, you dont' have any idea where i can find something on the net either??
Sorry for the last question, latelly there was many post about "please support this/that" and I missed that important post you made.


For the "force open" of Ryusui, maybe a simple "open as.." for the many games that store NCGR/NCER/NSCR without header is a good idea as well.
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on December 03, 2011, 08:03:25 pm
Didn't an old version of CT support some variants of the stock graphics formats that the latest released version doesn't? Check the stuff from Sim city.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 31, 2011, 02:17:04 pm
http://llref.emutalk.net/projects/ctool/downloads/ctool31b_multi.zip (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_multi.zip)

It is currently 4:30am 1/1/12 where I live, but technically its still 2011 in SOME parts of the world, so that still kinda counts. I was a bit more busy this holiday season including last night so I didn't end up spending as much time of ctool as I wanted. I was working towards adding COLLADA support as a surprise but I only got as far as parsing the file as I still need to sit down and break down all the 3D data so I can build my classes accordingly.

I will try to prepare another update within the next week or so, which will hopefully have anything that was missing not completely implemented, which I had intended to include in this release. This also marks the first multi-platform release, however bear in mind that MOST of the testing as of the last 6 or so months has been almost exclusively on the Mac for obvious reasons and the Linux build has had no testing what-so-ever. The only real difference between builds is the OpenGL support which is auto-generated by the compiler, and some minor native support functions exclusive to Mac.

Again report any bugs you find, but don't be surprised is x file opens but displays nothing...That generally means I is working on it :p
NCERs are also only half implemented right now, but as i said I will try to put out a fixed version later.

Happy New Year!
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on December 31, 2011, 04:17:30 pm
Code: [Select]
C:\Users\Henrik\Desktop\ds reverse engineering\tools\ctool31b_multi\dist-windows
-i586>java -jar consoleTool3.jar
Exception in thread "main" java.lang.NoClassDefFoundError: com/apple/eawt/ApplicationEvent
        at ctool.Loader.init(Loader.java:23)
        at ctool.Main.main(Main.java:30)
Caused by: java.lang.ClassNotFoundException: com.apple.eawt.ApplicationEvent
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 2 more
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on December 31, 2011, 10:19:58 pm
Code: [Select]
C:\Users\Henrik\Desktop\ds reverse engineering\tools\ctool31b_multi\dist-windows
-i586>java -jar consoleTool3.jar
Exception in thread "main"
*Blip* *Blip* *Blip* End of Cheese Error
*Blip* *Blip* *Blip* Can Not Find Drive Z:
*Blip* *Blip* *Blip* Unknown Application Error
*Blip* *Blip* *Blip* Please Reboot Universe
*Blip* *Blip* *Blip* Year Of The Sloth *Blip* *Blip* *Blip*

http://llref.emutalk.net/projects/ctool/downloads/ctool31b_windows-i586.zip (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_windows-i586.zip)
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on January 01, 2012, 10:50:18 am
That runs. But I have some complaints.

The main issue to me is the directory tree and how it interacts with archives. The plus symbol only shows when you have selected the archive once. This is confusing and inconsistent with how the normal windows tree control operates.

There is also some performance issues. Some files hang the UI while being parsed.

Also, I suspect that you are using the wrong parser for the cpac_?d.bin files in AJ, those have three marginally different file formats. If I could figure them out in an hour so can you. Feel free to look at my code to save yourself the trouble of figuring them out.

The SDAT parser doesn't list streams.

Finally, do you think that you could change the default table the hex editor uses? It just doesn't work for files with non text data. It is impossible to tell the difference between the byte values. Talking about the hex editor, do you think that you could add some search and bookmarking functions?

And another file format that you should look into is the SRL file format. I strongly suspect that it is a chunk of dynamically loadable code, with a filesystem to go with it.
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 02, 2012, 12:14:20 am
Last time I played with this tool was about two years ago. I tried the latest version but I don't know how to get it running? I click the java applet but nothing runs :/
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 02, 2012, 02:50:39 am
Last time I played with this tool was about two years ago. I tried the latest version but I don't know how to get it running? I click the java applet but nothing runs :/
iirc you were a Linux user. The issue which was causing the Windows version to not run (and likely the Linux as well) was some Mac specific APIs that provide native functions for the Mac version. Since these APIs aren't part of the standard Java APIs they obviously aren't found on other platforms. One of the ups/downs of Java is that it doesn't actually store copies of the APIs in the application, it only references to them and expects the ClassLoader to find them when it starts up. The easy fix is including a stub for the APIs in other platform versions, which is what I have just done now. I'll have this properly fixed in future versions as its extra work to re-build the same thing twice, but hopefully this should resolve any startup issues.

Downloads:
Console Tool 3.1b (Windows) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_windows-i586.zip)
Console Tool 3.1b (Linux) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_linux-i586.zip)
Console Tool 3.1b (Mac OS X) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_macosx-universal.zip)
Console Tool 3.1b (All-Platforms *Updated*) (http://llref.emutalk.net/projects/ctool/downloads/ctool31b_multi.zip)
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on January 02, 2012, 03:51:19 pm
Wow, just tried the latest release. I'm liking it so far. It's like the expectations of earlier releases, but better. I am particularly interested in the 3D model viewer and I'm glad to see it's working fine. It actually uses the correct texture if a model uses an external texture which I think the previous version didn't do. Overall it feels improved since more models look correct. Still a few errors though. {Odd distorted points} (http://i55.photobucket.com/albums/g144/mega_rock_exe/ctool_31b_0318.png) {Texturing error} (http://i55.photobucket.com/albums/g144/mega_rock_exe/ctool_31b_0851.png)

In an earlier release, model-viewing options were available. When would these features return? Also, it would be useful to have the tool display the 3D model's internal name somewhere since the texture already does this. And when would animation support be included? I eagerly await future release of this tool because it's looking great. Keep it up!
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 05, 2012, 12:37:22 am
Ah, I am actually Windows (then XP, now Win 7). I just had a nifty GUI wrap going on that made the system look... not like default Windows :P

I got it running up now. But I couldn't get it to open ANY tlp files which was weird (opened fine with other programs though). I was wanting to go spelunking in the GameCube Fire Emblem game, looking for some unused graphics. Worked fine with some DS games though... hum. Thanks for all the work you've put into this program, it really is spectacularly awesome :)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 05, 2012, 12:45:56 am
Ah, I am actually Windows (then XP, now Win 7). I just had a nifty GUI wrap going on that made the system look... not like default Windows :P

I got it running up now. But I couldn't get it to open ANY tlp files which was weird (opened fine with other programs though). I was wanting to go spelunking in the GameCube Fire Emblem game, looking for some unused graphics. Worked fine with some DS games though... hum. Thanks for all the work you've put into this program, it really is spectacularly awesome :)

That's probably because I haven't implemented TPL (assuming that's a spelling error on your part) at this time. If they were supported in the past then I must have missed it :/
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 05, 2012, 12:50:24 am
Ah, alright. So much seemed changed between the two years ago and now I wasn't sure if some of the more generic formats were completely covered yet :)
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 05, 2012, 02:27:00 am
Ah, alright. So much seemed changed between the two years ago and now I wasn't sure if some of the more generic formats were completely covered yet :)

It's always a challenge to live up to expectations :p This will be included as part of the next release :)
(http://img807.imageshack.us/img807/9089/fireemblemtpl.png)
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 05, 2012, 02:46:41 pm
Awesome :p Actually, Would you be willing to try and get this game supported for 3D models? We don't have much data documented on the format (mostly cause we don't dabble in 3D much :p) but we do know how to decompress the package files and what the headers of the files say, like this:

Quote
There's a small trick to converting CMP files. I think CMP is short for compressed package, which should give you a clue. After decompressing a CMP file using the LZ77 compression, it'll become a standard pack (.pak) file, which I think you may have encountered in FE9/10.

I'm not sure if there's any documentation about these pack files, but from memory the format is, using system.cmp as the example (I think American version?) :

first 4 bytes = "pack" in ASCII = 70 61 63 6B
first 2 bytes = number of packed files = 00 10 (i.e. 16)
padding until
4 bytes starting from 0x0C = pointer to label indicating the file's name = 00 00 01 08 (i.e. 108 -> 0x108 -> FE8Data)
4 bytes starting from 0x10 = pointer to the start of the file = 00 00 02 20 (i.e. 220 -> 0x220)
4 bytes starting from 0x14 = size of the file in bytes = 00 02 3D 2C (i.e 146732 bytes, check the unused file FE8Data.bin in the root and size matches)
repeat the last 3 until there are no more files
'Cause we know there's unused models just sitting on there, 'cause we can read the main file with its table naming every internal resource. We just haven't found a way to get at the files yet... :p

I've also seen Square reuse an fbac package format for their Chocobo Fables and Chocobo Dungeon game on the DS... I don't think any of the data inside is especially different, it just needs to be unpacked.

Title: Re: The Console Tool (by Low Lines)
Post by: RetroHelix on January 05, 2012, 04:47:16 pm
It's always a challenge to live up to expectations :p This will be included as part of the next release :)
(http://img807.imageshack.us/img807/9089/fireemblemtpl.png)
Are all TPL-formats (and Palette-Formats) supported?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 06, 2012, 01:35:11 am
Are all TPL-formats (and Palette-Formats) supported?
Good question! It only supports the formats that are currently supported in BTI files, which are formats I have test files to check against. If people can send me files (basically anything with a generic texture in it, bmd/bti/tpl etc) of any texture formats I don't have, that would be awesome :)

I'll have to check this but for one I know I definitely don't have format type 10 (CI14X2).

[edit]

Awesome :p Actually, Would you be willing to try and get this game supported for 3D models? We don't have much data documented on the format (mostly cause we don't dabble in 3D much :p) but we do know how to decompress the package files and what the headers of the files say, like this:
'Cause we know there's unused models just sitting on there, 'cause we can read the main file with its table naming every internal resource. We just haven't found a way to get at the files yet... :p

I've also seen Square reuse an fbac package format for their Chocobo Fables and Chocobo Dungeon game on the DS... I don't think any of the data inside is especially different, it just needs to be unpacked.

CMP files are simply PAK files LZ77 compressed, notably in Little Endian format. Just out of curiosity to anyone that would know, can compression formats be either endian format, or are they generally not interchangeable?
Also there was a few errors in your format spec, which I've fixed up as below...

Code: [Select]
Square-Enix Package
Byte Order: Big Endian

-- header --
u32 magic "pack"
u16 # files

-- fat * # files --
u32 file type? (0=file)
u32 fnt offset (relative to start of package)
u32 file offset (same as above)
u32 file size (in bytes)

-- fnt --
list of null terminated (00) strings, max length 0x12?
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on January 18, 2012, 08:50:29 pm
since your a digimon fan I wanted to ask if you would ad the digital monsters ver.s format (from the sega Saturn) I know a few Saturn games use it

http://www.gamefront.com/files/21195411/AGUM.rar
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on January 19, 2012, 05:39:59 pm
It would be great if you could take a look at the Digimon World 1 (PSX Game) MMD Format of the 3D Models
http://www.fileserve.com/file/DVqMzAQ/AGUM.MMD

It uses a Structure which seems to be similar to TMD but with some additions, which I guess define the positioning of the objects.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on January 20, 2012, 12:58:18 am
Oh, a couple other things I noticed. In Windows, if Console tool is maximized and closed, trying to unmaximize it when it's opened again won't work since it forgets the unmaximized size and does a fake maximized thing.

Also, it's nice how it always seems to know which files to link to in building images out of NANR, NCLR, NCER, etc, but why won't it let me pick any other file? In this particular game's files, the Nitro formats for tilemap, animations, and tiles are used, but the palette one is a raw palette and console tool has no way of associating it with the sprite.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 20, 2012, 02:51:04 am
Oh, a couple other things I noticed. In Windows, if Console tool is maximized and closed, trying to unmaximize it when it's opened again won't work since it forgets the unmaximized size and does a fake maximized thing.

Also, it's nice how it always seems to know which files to link to in building images out of NANR, NCLR, NCER, etc, but why won't it let me pick any other file? In this particular game's files, the Nitro formats for tilemap, animations, and tiles are used, but the palette one is a raw palette and console tool has no way of associating it with the sprite.

I did not know that with the maximizing thing. I know I remember having a few issues with saving the windows size state, but since I generally don't keep it open long enough nor really utilize max screen (Mac also has sucky maximize functionality), its an easy thing to miss. If you do start to have any issues with display when it opens, deleting the settings.ini will reset everything to default. You can also edit it with a text editor if you are game (though it isn't all that complicated to be honest).

At the moment I think when it loads the Assets panel, it pulls all assets from all open files as well as assets from the same folder, the latter possibly needing said folder open in ctool. Force opening files currently behaves separate from this, in that when you force open a file, it makes a clone of the file handler and applies the new functionality to it, rather than overwriting the original. The main reason for this is in the case a force open fails or you overwrite its original state, you don't have to reopen the browser folder to reset it again etc. Just thinking about it now, adding a refresh but to the file browser would probably be a good idea :) By rights though, if you do force open a file and it opens. It SHOULD become available as an asset in the assets panel for any open files that can use it.

@Romsstar
I have in fact toyed with MMD models myself, however the problem lies in the lack of a specification for the format. I have gotten the model meshes displaying but they are missing the bone data or whatever is used to animate them as this is not part of the TMD format, which I suspect is somewhere else in the MMD file.

@viperzerof-2
Are these sprites or models? At a glance the files appear to have some form of a header, but they also look like raw image data too, but without some idea what I'm looking at they could very well be anything :(

@Update
Lately I've been mostly working on the gui side of things and getting more of that data onscreen. I've started using HashMaps to hold all object variables (and I don't know what I wasn't already), which long story short, makes it much easier to put generic and custom variables specific to certain formats into the gui in an editable manner. I've also moved asset bank switching (ie where you pick which bank you want to display for palettes, images etc) into their own separate panels where you will also be able to access the object data directly. Probably doesn't make a lot of sense, but I think it's a good step in the right direction.

Also in one of my other projects I've been experimenting with LAN Video Streaming through the use of ffmpeg and executing native code. The good news is that means I know how to add support for ffmpeg encoding/decoding, at least as far as the ffmpeg's command line goes. However as I AM running native code to do this, I will have to have a slightly different implementation for each platform although mac and linux are similar so that's a good start :) It's not a pure java solution for opening multiple file formats, which is kind of what ctool was intended to be however it provides a workaround for some of the problems I've had to face with various compressions and what not. Just looking at the internal documentation though, ffmpeg already has support for various game based file formats such as Gamecube THP which is really neat :) Something to look forward to in the next release perhaps if all goes well :)
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on January 20, 2012, 08:52:19 am
But does it do the videos from ds games?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 20, 2012, 08:53:43 am
But does it do the videos from ds games?
No. And if you want to know why, ask Google.
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on January 20, 2012, 12:21:13 pm
Oh, another small issue. Don't know if it's because new features are being implemented but if files are opened in the tool, then closed via the tab's X to the point where no file is open, opening new files will open up the files just closed. Even if a file is closed and the program is open, the opened file can't be moved or deleted because apparently it is still being used.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 20, 2012, 03:52:37 pm
Oh, another small issue. Don't know if it's because new features are being implemented but if files are opened in the tool, then closed via the tab's X to the point where no file is open, opening new files will open up the files just closed. Even if a file is closed and the program is open, the opened file can't be moved or deleted because apparently it is still being used.
Is this done through using the File > Open or by dragging and dropping files into ctool? I don't think I've tested this specifically but I have made some changes to the opening/closing process, in that for one, until I implement a proper File > Open dialog that action will open up the home folder for the current user in ctool OR the parent folder of the currently selected file if there is a file open. Also there was a bug with the Close All option where it did not actually close all files open. That's been fixed, but maybe that also has fixed the issue you described, I'll have to check that :p
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on January 20, 2012, 04:15:00 pm


@viperzerof-2
Are these sprites or models? At a glance the files appear to have some form of a header, but they also look like raw image data too, but without some idea what I'm looking at they could very well be anything :(


ops they are sprites, in a very similar style to digimon world one
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on January 20, 2012, 07:46:39 pm
ops they are sprites, in a very similar style to digimon world one

Now this statement can't be right. Sega Saturn never used 3D Models and while technically the GPU of the PSX isn't capable of displaying real 3D it's quite different.

Digimon World used 3D Meshes with applied textures.

Sprites are images with no texture and no meshes.
They are flat.

The sprites you sent are 4bpp linear with a palette that is most likely embed into the SPR file. Although there is a chance that it might be in the .ATR as well.
Uncompressed and easy to figure out, you just need to locate the palette. this could be easily done by checking the savestate of a Genesis Emulator with Tile Molester since the right palette with a very high chance will be in there.

Then you just check whether you can find it in the SPR file or not.

@Low-Lines:
I don't know if the TOD data is contained in the MMD files but the information of the position of the meshes must be in there. In some different kind of format.
What kind of is the difficult task I guess.
The TMD objects bone data or "linking" if you will must be somewhere in the MMD File. I've searched the disc and can't think of any other place it could be.
I mean there is a slight chance that it is in the texture data, but as far as I've checked it contains only multi-clut tim files. There is already a good viewer for that so no need to bother.

But I really would like to view those 3D models (and actually to mess with but hey let's start small)
If I can help you I'd gladly do my part.
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on January 20, 2012, 08:01:37 pm
what did I say wrong? they are sprites that look like digimon world one's models the game even uses the same sounds :)
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on January 21, 2012, 12:26:49 am
Is this done through using the File > Open or by dragging and dropping files into ctool? I don't think I've tested this specifically but I have made some changes to the opening/closing process, in that for one, until I implement a proper File > Open dialog that action will open up the home folder for the current user in ctool OR the parent folder of the currently selected file if there is a file open. Also there was a bug with the Close All option where it did not actually close all files open. That's been fixed, but maybe that also has fixed the issue you described, I'll have to check that :p
I dragged them in. Maybe you've already fixed it. I'll have to wait and see.
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on January 21, 2012, 07:13:43 am
what did I say wrong? they are sprites that look like digimon world one's models the game even uses the same sounds :)

Ah it just feels like looking down on the digimon world models which were brilliant I think.
I'm just a bit sensitive in this point, they never got the Credit they deserved.

It's true the sprites are good work but I think the models in digimon world are much better and can't be compared to Sega Sprites.
But well we have just a different opinion in this point. In any case I meant to help.

As I said try following with Tile Molester: 4 bit linear mode, try a canvas size of 32x16 and 2d-mode, and now all left to do is look for the palette.
Try import the palette from the SPR file and try different offsets. Look through the file and see if you can spot something.

Eventually you will succeed. Once you figured out the settings of the format the rest is just implementing it into Console Tool and I think you can save Low lines some work if you do the Research part :)
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on January 21, 2012, 07:23:14 pm
I don't recall insulting the models, I just said they use the same art style or based on the same art work I never intended to elevate one over the other since i've played both for years
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 21, 2012, 11:42:13 pm
Visual appeal means nothing, when you are trying to parse binary data :p Though I do wish they had made more games like DW1 as the game mechanics are really what defines Digimon, not all this turn-based RPG crap which all the sequels do :/

Moving on, here's what I've got on MMDs so far...
Code: [Select]
Monster MoDel (*.mmd)

TMD Header {
u32 TMD Start Offset,
u32 TMD End Offset
}

TMD file follows straight after...

Unknown Block

The rest of the file then starts with what looks like a bunch of offsets relative to the end of the TMD file (in other words a separate file and MMDs are container formats), however there's some ambiguous values that prevent it from being just a straight table...
It's worth noting that all Digimon models seem to be rotated +180 degrees on the Y axis, unless its some kind of error on my part of course, but I'm pretty sure it isn't...
Given the large amount of data left over after taking the TMD file out, it's very likely that MMDs hold animations as well.

[edit]

There does seem to be a pattern when comparing multiple MMD files.
For one, off0 and off33 always seem to be the same, and also point to the end of the offsets header. This is sort of an assumption, but my guess is that each offset points to an animation block and when the offset is null(00) then there's no animation for it. Each offset is for a specific animation (ie walk, attack, sleep, poop, etc), with some animations pointing to the same data.
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on January 22, 2012, 08:05:10 am
After comparing Agumon and Shogungekomons data I can confirm that your assumptions are completely right Low Lines.
Shogungekomon has 8 defined animations for it and they all are in the MMD file.
Agumon has a lot more and the parts where shogungekomon has no animation for are blanks (00)
while Agumons is filled with data.

We can take for given that the animation indeed is in the MMD file.
Although it seems to be an usual format and not the standard TOD format animations are usually in.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 22, 2012, 03:30:08 pm
Could you provide a list of which model files are which Digimon?
ie /CHDAT/MMD1/AGUM.MMD Agumon

So far the only thing I can gather is that the first byte in these animation pointers appears to be the number of keyframes in the animation. Each keyframe can also be of varying length as I've seen several files with the same number of keyframes but with varying file lengths.
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on January 22, 2012, 04:06:38 pm
MMD0:

/CHDAT/MMD0/AGUM.MMD   AGUMON
/CHDAT/MMD0/AIRD.MMD    AIRDRAMON
/CHDAT/MMD0/ANGE.MMD   ANGEMON
/CHDAT/MMD0/BETA.MMD   BETAMON
/CHDAT/MMD0/BIRD.MMD    BIRDRAMON
/CHDAT/MMD0/BOTA.MMD   BOTAMON
/CHDAT/MMD0/BOYS.MMD   PLAYER (HIRO)
/CHDAT/MMD0/DEVI.MMD    DEVIMON
/CHDAT/MMD0/ELEC.MMD    ELECMON
/CHDAT/MMD0/GABU.MMD   GABUMON
/CHDAT/MMD0/GARU.MMD   GARURUMON
/CHDAT/MMD0/GREY.MMD   GREYMON
/CHDAT/MMD0/HOEE.MMD   WHAMON
/CHDAT/MMD0/KABU.MMD   KABUTERIMON
/CHDAT/MMD0/KORO.MMD   KOROMON
/CHDAT/MMD0/MAME.MMD   MAMEMON
/CHDAT/MMD0/MERA.MMD   MERAMON
/CHDAT/MMD0/MONZ.MMD   MONZAEMON
/CHDAT/MMD0/MTGR.MMD   METALGREYMON
/CHDAT/MMD0/MTMA.MMD  METALMAMEMON
/CHDAT/MMD0/NUME.MMD   NUMEMON
/CHDAT/MMD0/POYO.MMD   POYOMON
/CHDAT/MMD0/PUNI.MMD    PUNIMON
/CHDAT/MMD0/SEAD.MMD   SEADRAMON
/CHDAT/MMD0/SKUL.MMD   SKULLGREYMON
/CHDAT/MMD0/TUNO.MMD   TSUNOMON
/CHDAT/MMD0/TYRA.MMD   TYRANNOMON
/CHDAT/MMD0/VEDA.MMD   VADEMON
/CHDAT/MMD0/VEGI.MMD    VEGIEMON
/CHDAT/MMD0/YUKI.MMD    FRIGIMON

MMD1:

/CHDAT/MMD1/ANDR.MMD ANDROMON
/CHDAT/MMD1/BAKE.MMD BAKEMON
/CHDAT/MMD1/CENT.MMD CENTARUMON
/CHDAT/MMD1/COCA.MMD KOKATORIMON
/CHDAT/MMD1/DIGI.MMD DIGITAMAMON
/CHDAT/MMD1/DORI.MMD DRIMOGEMON
/CHDAT/MMD1/ETEM.MMD ETEMON
/CHDAT/MMD1/GIRO.MMD GIROMON
/CHDAT/MMD1/HOUO.MMD PHOENIXMON
/CHDAT/MMD1/IGAM.MMD  ???
/CHDAT/MMD1/KUNE.MMD KUNEMON
/CHDAT/MMD1/KUWA.MMD KUWAGAMON
/CHDAT/MMD1/LEOM.MMD LEOMON
/CHDAT/MMD1/MGDR.MMD MEGADRAMON
/CHDAT/MMD1/MOJA.MMD MOJAMON
/CHDAT/MMD1/MONO.MMD MONOCHROMON
/CHDAT/MMD1/NANI.MMD NANIMON
/CHDAT/MMD1/OGRE.MMD OGREMON
/CHDAT/MMD1/PALM.MMD PALMON
/CHDAT/MMD1/PATA.MMD PATAMON
/CHDAT/MMD1/PENM.MMD PENGUINMON
/CHDAT/MMD1/PICC.MMD  PIXIMON
/CHDAT/MMD1/PIYO.MMD BIYOMON
/CHDAT/MMD1/SCUM.MMD SUKAMON
/CHDAT/MMD1/SHEL.MMD SHELLMON
/CHDAT/MMD1/SIRA.MMD  ???
/CHDAT/MMD1/TANE.MMD TANEMON
/CHDAT/MMD1/TOKO.MMD TOKOMON
/CHDAT/MMD1/UNIM.MMD UNIMON
/CHDAT/MMD1/YURA.MMD YURAMON

I will try to provide a whole list later.
So far this is something I guess.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 22, 2012, 11:21:09 pm
Also you mentioned TOD animations? I can not seem to find any documentation on Google (and seriously I've tried...) that's of use. If I could look at how the standard format works, it might shed a bit of light on the MMD format.

If anyone could send me a link to the docs, and some sample files (or at least name a game that uses them), that would be great :p
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on January 23, 2012, 01:15:34 am
http://www.sendspace.com/file/bjvmi4
Taken from the PSX SDK.
The program TOD_View with source code and samples.

http://www.sendspace.com/file/q5lafr
Some more samples. Also from the PSX SDK.

http://consolecopyworld.com/cgi-bin/dl.cgi?id=psx&file=tod_info!zip
The program TOD_INFO.
You can try to use that one on the MMD but it found nothing.

http://consolecopyworld.com/cgi-bin/dl.cgi?id=psx&file=tod_rip!zip
The same as TOD_INFO but it rips TOD files.

You could use like um any 3D based PSX game for testing I guess.
Right out of the top of my head I can't think of a game that uses TOD right now. But I most certainly encountered some of them. It is a standardized format
after all which means: It's been used, but as you know, developers tend to "create" their own formats in order to protect their precious little games.

(Seriously for games that are older than 10 years the source code should be public...)

Hope I could help you, I will try to dig for more information, but I hope this is something you can already work with.

PS: Some more information I took from the documentation of the SDK:


[1.1.7.]: I would like to know about the formats of the
rotation parameters (Rx, Ry, Rz), the translation parameters
(Tx, Ty, Tz), and the scaling parameters (Sx, Sy, Sz) in the
TOD data packet.
See the following.
* Rotation
32-bit signed integer. (1,31,0) 4096 represents 1 degree.
* Transfer
32-bit signed integer. (1,31,0)
* Scale
16-bit fixed decimal point number. (1,3,12)

Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 24, 2012, 12:55:42 am
Good question! It only supports the formats that are currently supported in BTI files, which are formats I have test files to check against. If people can send me files (basically anything with a generic texture in it, bmd/bti/tpl etc) of any texture formats I don't have, that would be awesome :)

I'll have to check this but for one I know I definitely don't have format type 10 (CI14X2).

http://www.mediafire.com/?a2rm8upmn5v57m9

I don't know if this is what you meant, but my normal tpl viewer doesn't display tpl files from this game.  I've found one or two which are, but every other one just appears to be an unsupported format to my viewer (P3d0 Explorer).
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 24, 2012, 01:31:36 am
@Celice
All textures in that file are S3TC (or rather the Gamecube variant), which all display fine.
(http://img20.imageshack.us/img20/9982/s3tctpltest.png)

@TOD/MMD Stuff
TOD File Format, extract from SDK sources provided by Romsstar.
It's actually a rather nice little block format. It basically creates the bone/skeleton like struct on the first frame and then runs from there :p I haven't checked yet but I'm sort of hopeful that the MMD animations follow this struct just missing the headers :p
Spoiler:
Code: [Select]
Packet Types
0=TOD_ATTR
1=TOD_COORD
2=TOD_TMDID
3=TOD_PARENT
4=TOD_MATRIX
5=TOD_TMDDATA
6=TOD_LIGHT
7=TOD_CAMERA
8=TOD_OBJCTL
9=TOD_USER0
10=TOD_USER1
11=TOD_USER2
12=TOD_USER3
13=TOD_USER4
15=TOD_SYSFUNC

header {
u32 magic (0x50),
u32 numframes,
}

frame {
u16 ?,
u16 numpackets,
u32 frame/time
}

packet {
u16 id,
u4 type,
u4 flag,
u8 length
}

tod_attr(0) {
u32 attr,
u32 pad
}

tod_coord(1) {
// if flag.b0 differencial else immediate
//Rotation if flag.b1
s32 rx, // divide by 360
s32 ry, // divide by 360
s32 rz, // divide by 360
//Scale if flag.b2
s16 sx, // if flag.b0 divide by 4096
s16 sy, // if flag.b0 divide by 4096
s16 sz, // if flag.b0 divide by 4096
s16 pad,
//Transfer if flag.b3
s32 tx,
s32 ty,
s32 tz
}

tod_matrix(4) {
s32 matrix*8
}

tod_tmdid(2) {
u32 tmd_id
}

tod_parent(3) {
u32 parent_id
}

tod_objctl(8) {
//if flag==0 Create New Object
//if flag==1 Delete Object
}

tod_light(6) {
//if flag.b0 differencial else immediate
//if flag.b1
s32 lx,
s32 ly,
s32 lz
//if flag.b2
u8 r,
u8 g,
u8 b
u8 pad
}

[edit]

Nope, MMD animations are completely custom :/
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on January 24, 2012, 08:46:11 am
Shit, this was to be expected.
Is there anything I can help you with Low Lines that we can actually crack down that format?
the PSX SDK has some documentation on TMD as well.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 24, 2012, 04:21:42 pm
I think I've found something...
Spoiler:
Code: [Select]
PUNI.MMD | Animation #1
15
00
00000000
00000000
D3FF0000
00000000
00000000
00000000
FF10
0100013000341808040201000130201818042800010081F4
0200330300F8330300001700
030081F00100500031FF50000200
040081F40100E5FF3D00E5FFD4FFFFFF
050081F40100CBFF9200CBFF4B00FFFF
060081F40100B3FF0001B3FFB6FFFDFF
070081F40100B2FF0A01B2FF2B00FEFF
080081F40100DCFF7E00DCFF0000FFFF
090081F001009B005FFE9B000400
0A0081F00100240019FF24000200
0B0081F0010007FDD60407FDEBFF
0C0081F401002AFC73062AFCD1FFE5FF
0D0081F4010035FDB70435FDC000EDFF
0E0081F4010089FFB70089FFB4FEFDFF
0F0081F40100E2FF1D00E2FF9001FFFF
100081F40100FDFFF4FFFDFFBBFE0000
110081F40100980038FF9800AF000400
120081F4010047022DFD470297FF1100
130081F401009602CEFC96022A001300
140081F401008A0105FE8A0100000800
152001000000
As you can see, the latter part of the file can be broken up into blocks because each block starts with an index. The last block index also matches up with the first byte in the file. The data at the start of the file appears to be base values (ie D3FF0000 as s32/360 equals approx 180 degrees, the amount needed to turn models up the right way), however I'm still seeing a bit of variation with it. Also the indexed blocks are likely to hold information for each polygon, so far I've only been only able to see the pattern in smaller models (ie with 1 polygon :p) as it's possible the index is only u8 thus can be mixed up with data easily without knowing the proper block lengths/distribution.

[edit]

It also appears that the numbering doesn't have to be sequential...Logically making blocks keyframes, and the first byte in the file being total time (in frames)

Spoiler:
Code: [Select]
TUNO.MMD | Animation #1
15
00
00000000
00000000
B9FF0000
00000000
00000000
00000000FF10010001300036180808020100013020181806200001000080
010001A001001A00
020081A001003500FFFF
030081A00500D601F9FF
080081A001002E000000
090081A001001300FFFF
0A0081A0010000000000
0B0081A00100EDFF0100
0C0081A00100D2FF0000
0D0081A005002AFE0700
120081A00100CBFF0100
130081A00100E6FF0000
140001A001000000
152001000000

[edit 2]

Here's another breakdown...I think I've got block section figured out (at least visually), I now just have to figure out how its done programatically :/

Note the block structure is brokendown as so..
Code: [Select]
FRAME#, ENDFLAG {
ID, FLAG, TIME, DATA
}
// ENDFLAG is 0x20 when last frame
// ID is & 0x3F (max 64 objects which matches with SDK notes I think...)
// ID.b6-7 are part of FLAG
// FLAG is what data is included
// TIME the number of frames this lasts for or transitions to?
Spoiler:
Code: [Select]
AGUM.MMD | Animation #1
11
00
000000FC
00000000
9CFF0000
00000000
3BFB
050000000000000000001EFF
2200000000000000000042FD
320000000000000000040000
220000000000D4FF730297FF
180000000000B3FF69FF99FD
340000000000F8FFC5FF3DFF
2D0000000000000000FC0000
2200000000002C008DFD97FF
1800000000004D00970099FD
34000000000008003B003DFF
2D0000000000000000004A03
050000000000000000040000
24000000000068FD2EFD2200
190000000000E6FE48FFEFFA
21000000000002FFA5001B06
180000000000000000FC0000
2400000000009802D2022200
1900000000001A01B800EFFA
210000000000FE005BFF1B06
180000000000FF1001000130
20201808200001000080

01 00
C1 80 0900 0300FEFF
02 82 0900 97FF
03 82 0200 2900
04 82 0900 D0FF
06 8E 0900 FCFFF9FFFEFF
07 8E 0900 040004000300
11 00
08 8E 0900 EBFFC5FF6EFF
0A 8E 0900 04000700FEFF
0B 8E 0900 FCFFFDFF1100
0C 8E 0900 15003B006EFF
0F 8E 0900 2F0010001C00
10 8E 0900 E8FF0C00A7FF
11 8E 0900 F8FFF4FF6400
13 8E 0900 D1FFF0FF1C00
14 8E 0900 1800F4FFA7FF
15 8E 0900 08000C006400
03 00
03 82 0800 AEFF
0A 00
C1 80 0700 FDFF
02 00
02 82 0700 6900
04 82 0700 3000
06 8E 0700 040007000200
07 8E 0700 FCFFFDFFEFFF
08 8E 0700 15003B009200
0A 8E 0700 FCFFF9FF0200
0B 8E 0700 04000300EFFF
0C 8E 0700 EBFFC5FF9200
0F 8E 0700 D1FFF0FFE4FF
10 8E 0700 1800F4FF5900
11 8E 0700 08000C009CFF
13 8E 0700 2F001000E4FF
14 8E 0700 E8FF0C005900
15 8E 0700 F8FFF4FF9CFF
0B 00
03 82 0600 2900
11 20
01 00 0000

[edit 3]

From what I can tell, the number of "objects" doesn't match up with the number of polygons, so I'd assume that polygons are attached in-frame (sort of like how TODs work). However I haven't bothered to figure out the flags in the block section yet because I wanted to be able to get there programatically first. With that said, I think I have now figured out the header part of the file...

Code: [Select]
header {
u8 numframes,
u8 flags, /* 0x80 means scale included */
}

init_header {
/* if scale included */
s16 sx,
s16 sy,
s16 sz,
/* endif */
s16 tx?,
s16 ty?,
s16 tz?,
s16 rx,
s16 ry,
s16 rz
/* above could be six s16 values, or three s32 values, I'm not sure however most of the data seems to be in s16/u16 */
}
/* repeats until 0xFF10 is found, call the init-header your base values for objects, numobjects can also be calculated from this */

FF10_section {
/* still needs some figuring out, has varying lengths, normally 16 bytes */
}

frame * numobjects {
u8 frameid,
u8 flag, /* 0x20 means last frame, 0x00 otherwise */
framedata /* repeats until next frame */
}

framedata {
u16 flagid, /* b0-5, objectid, b6-15 flags
u16 time/length (in frames),
data /* depends on flags */
}
Title: Re: The Console Tool (by Low Lines)
Post by: Celice on January 28, 2012, 12:35:11 am
@Celice
All textures in that file are S3TC (or rather the Gamecube variant), which all display fine.

Ah, I copied the wrong file :p I could see this one, but some of the other tpl files throw up errors.

http://www.mediafire.com/?o8ki8pl6doxav84

Like this one, I was able to open it with a different viewer, which then showed numerous textures inside like it was a package.

What does your program do with this file?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 28, 2012, 05:38:32 am
It likes that one too, formats in it include I8, RGB5A3, IA8, and S3TC.
Title: Re: The Console Tool (by Low Lines)
Post by: huckpie on January 28, 2012, 06:56:43 am
Low, do you think you can add support for these kinds of archives?

http://www.mediafire.com/?71jx2vdx5q5rfzd

Most of Webfoot's titles for the NDS store their assets in a single .pak file, and I haven't seen anyone doing their stuff on this.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 29, 2012, 09:04:58 pm
Low, do you think you can add support for these kinds of archives?

http://www.mediafire.com/?71jx2vdx5q5rfzd

Most of Webfoot's titles for the NDS store their assets in a single .pak file, and I haven't seen anyone doing their stuff on this.

That might be because a lot of their NDS games are targeted at children. I would personally like to hack their DBZ games though...
The Webfoot Package format is rather straight-forward, however the files themselves all appear custom as well, so this might not help you much...
Code: [Select]
Webfoot Package (*.pak)
header {
u32 fileCount, /* shift it left by 1 and that gets you the size of the file table */
file_table
}

file_table_entry {
u32 flags or perhaps a checksum?,
u32 fileOffset,
u32 fileSize,
u32 fileType?
}
Title: Re: The Console Tool (by Low Lines)
Post by: huckpie on January 30, 2012, 07:44:27 pm
That might be because a lot of their NDS games are targeted at children. I would personally like to hack their DBZ games though...

Yeah I heard they did a good job at the Dragon Ball GBA games. BTW, did the package format use any sort of compression? Aluigi at Xentax told me that it made use of some crazy compression format that he couldn't find out, but i dunno.

And another thing: Does your tool have full (as in read/write) support for Grand Theft Auto archives? Most of them peeps who'd like to mod the OSX port of the game are looking for an IMG tool for Macs besides running the Windows tools under Wine.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 31, 2012, 06:42:07 am
There were a few files that had seemingly valid LZ77 headers, however they won't decompress due to negative offsets. Point is unless you know what data you are looking at, there's no real way of knowing if it actually is or not. If you got Aluigi to look at it, I'd put his word above mine, but the above is what I saw when I looked at it myself.

ctool just reads GTA IMGs, and files can also be exported. Editing support is planned, just not on my to do list atm.
Title: Re: The Console Tool (by Low Lines)
Post by: huckpie on January 31, 2012, 08:23:57 pm
There were a few files that had seemingly valid LZ77 headers, however they won't decompress due to negative offsets. Point is unless you know what data you are looking at, there's no real way of knowing if it actually is or not. If you got Aluigi to look at it, I'd put his word above mine, but the above is what I saw when I looked at it myself.

ctool just reads GTA IMGs, and files can also be exported. Editing support is planned, just not on my to do list atm.

Oh, and what about the code above? And I hope you won't mind, but I also have an NDS 3D model format that you might like to peek at:
http://www.mediafire.com/?ix1s9i7m03a21ak

I dunno, maybe we can add these just for the sake of completeness.
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 31, 2012, 09:09:20 pm
Just glancing at the data in that file, it looks like it's either compressed or was badly decompressed as the readable strings are in pieces...
What might help is stating what game it is from, and what means did you go through in order to get this file, otherwise its just garbage data atm.

[edit]

Since Google is so efficient I figured out what you are trying to do myself...File is from Strawberry Shortcake: The Four Seasons Cake, and looking at some of the files in the games filesystem, I'd say with confidence that anything with CMPR at the start of the file is compressed. Unless you know how to decompress such files you basically can't figure out what they contain.

[edit 2]

It turns out files are simply LZ77 compressed with a CMPR header to identify files are compressed.
http://www.mediafire.com/?6aefye39k9rseu2 (http://www.mediafire.com/?6aefye39k9rseu2)
Title: Re: The Console Tool (by Low Lines)
Post by: huckpie on January 31, 2012, 10:46:21 pm
So what kind of model file is it?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on January 31, 2012, 10:53:29 pm
So what kind of model file is it?

A custom model format with the magic id "SCEN", my guess an in-house format for The Game Factory company.
Title: Re: The Console Tool (by Low Lines)
Post by: huckpie on January 31, 2012, 11:00:29 pm
A custom model format with the magic id "SCEN", my guess an in-house format for The Game Factory company.

Which would be a bummer unless if the format isn't that hard to decode. From what I saw when I opened it on Notepad, bone names show up as strings.

EDIT: Is the PAK format found on Digimon games any similar to Webfoot's?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on February 03, 2012, 09:58:27 am
The N3D Scene File is basically made up of three parts. The first part holds materials, these can optionally include a string path to a TEX file stored in the rom.
The second part holds the model data, which starts with the skeleton and then the geometry data which I'm sure (although haven't really looked) is just packed geometry commands. I haven't really looked at the final part yet.

I've tried breaking down the model data, however I as of yet haven't figured out how it's broken up.

Code: [Select]
header {
u8[4] magic,
u32 materialOffset,
u32 modelOffset,
u32 offs3,
u8[20] unknown
}
materialObject { // repeats until end of materialSection
u32 offset?,
u8[16] name,
u32 flags,
u8[256] texturePath,
u8[44] unknown
}
modelSection {
bones/objects,
geometry_data
}
offs3Section {

}

Here's my attempt at breaking down the modelSection in your angel.n3d file...
Spoiler:
Code: [Select]
$0678 u32 01000000 1
$067C u32 0A090000 $090A
$0680 u8[16] 00000100000000000000000000000000
$0690 u32 BC060000 $06BC
$0694 u8[16] 576F726C640000000000000000000000 World
$06A4 s32 7BF9FFFF -1669
$06A8 s32 23050000 1315
$06AC s32 BAEFFFFF -4166
$06B0 u32 2E080000 $082E
$06B4 u8[16] 73220000B00F00000200000008080000
$06C4 u8[16] 0000000000000000780600002C4C0000
$06D4 u32 F4060000 $06F4
$06D8 u8[16] 726F746174655F616C6C000000000000 rotate_all
--------------------------------------------------------------------------------------------------------------------
$06E8 u8[12] 010000000000010100000000
$06F4 u32 00080000 2048
$06F8 u32 08080000 2056
$06FC u8[16] 0000000000000000BC06000000000000
$070C u32 44070000 $0744
$0710 u8[16] 43656E746572526F6F74000000000000 CenterRoot
--------------------------------------------------------------------------------------------------------------------
$0720 u8[36] 0100000001000102023F000000000000AA0D0000B2FFFFFF00040000FDFFFFFF00040000
$0744 u32 00080000 2048
$0748 u32 00080000 2048
$074C u8[8] 0000000000000000F406000040080000
$075C u32 A0070000 $07A0
$0760 u8[16] 4C656674486970000000000000000000 LeftHip
--------------------------------------------------------------------------------------------------------------------
$0770 u8[48] 0100000001010103FA3E000069FEFFFFA30000009A01000002F8FFFFDBFFFFFF4D0700006E000000480100008D020000
$07A0 u32 00080000 2048
$07A4 u32 00080000 2048
$07A8 u8[16] 00000000000000004407000000000000
$07B8 u32 F0070000 $07F0
$07BC u8[16] 4C6566744B6E65650000000000000000 LeftKnee
--------------------------------------------------------------------------------------------------------------------
$07CC u8[36] 0100000001000104763E00005D030000000000000000000001000000F7FFFFFF86FFFFFF
$07F0 u32 00080000 2048
$07F4 u32 00000000 0
$07F8 u8[16] 0000000000000000A007000000000000
$0808 u32 00000000 $0000
$080C u8[16] 4C656674466F6F740000000000000000 LeftFoot
--------------------------------------------------------------------------------------------------------------------
$081C u8[36] 0100000001000105F23D00006B0700000000000000000000280000001800000002030000
$0840 u32 00080000 2048
$0844 u32 00080000 2048
$0848 u8[16] 0000000000000000F40600003C090000
$0858 u32 9C080000 $089C
$085C u8[16] 52696768744869700000000000000000 RightHip
--------------------------------------------------------------------------------------------------------------------
$086C u8[48] 01000000010101067A3D000069FEFFFFA300000066FEFFFFFE070000250000004D070000B1FFFFFF2DFFFFFFD8020000
$089C u32 00080000 2048
$08A0 u32 00080000 2048
$08A4 u8[16] 00000000000000004008000000000000
$08B4 u32 EC080000 $08EC
$08B8 u8[16] 52696768744B6E656500000000000000 RightKnee
--------------------------------------------------------------------------------------------------------------------
$08C8 u8[36] 0100000001000107EA3C00005D0300000000000000000000FFFFFFFF0900000086FFFFFF
$08EC u32 00080000 2048
$08F0 u32 00000000 0
$08F4 u8[16] 00000000000000009C08000000000000
$0904 u32 00000000 $0000
$0908 u8[16] 5269676874466F6F7400000000000000 RightFoot
--------------------------------------------------------------------------------------------------------------------
$0918 u8[36] 0100000001000108663C00006B0700000000000000000000D8FFFFFFE8FFFFFF02030000
$093C u32 00080000 2048
$0940 u32 00080000 2048
$0944 u8[16] 0000000000000000F4060000740C0000
$0954 u32 8C090000 $098C
$0958 u8[16] 43656E7465725370696E650000000000 CenterSpine
--------------------------------------------------------------------------------------------------------------------
$0968 u8[36] 0100000001000109FA3B0000A900000026000000000000000100000004000000D8FFFFFF
$098C u32 00080000 2048
$0990 u32 00080000 2048
$0994 u8[16] 00000000000000003C09000000000000
$09A4 u32 DC090000 $09DC
$09A8 u8[16] 43656E7465725370696E650000000000 CenterSpine
--------------------------------------------------------------------------------------------------------------------
$09B8 u8[36] 010000000100010A5E3B00007E0200000000000000000000000000000000000004FFFFFF
$09DC u32 00080000 2048
$09E0 u32 00080000 2048
$09E4 u8[16] 00000000000000008C0900001C0B0000
$09F4 u32 2C0A0000 $0A2C
$09F8 u8[16] 43656E7465724E65636B000000000000 CenterNeck
--------------------------------------------------------------------------------------------------------------------
$0A08 u8[36] 010000000100010BDA3A00001B030000000000000000000000000000000000005E010000
$0A2C u32 00080000 2048
$0A30 u32 00080000 2048
$0A34 u8[16] 0000000000000000DC09000000000000
$0A44 u32 7C0A0000 $0A7C
$0A48 u8[16] 43656E74657248656164000000000000 CenterHead
--------------------------------------------------------------------------------------------------------------------
$0A58 u8[36] 010000000100010C563A0000C00100000000000000000000000000000000000023FFFFFF
$0A7C u32 00080000 2048
$0A80 u32 00080000 2048
$0A84 u8[16] 00000000000000002C0A000000000000
$0A94 u32 CC0A0000 $0ACC
$0A98 u8[16] 68616972000000000000000000000000 hair
--------------------------------------------------------------------------------------------------------------------
$0AA8 u8[36] 010000000100010DD23900006A050000000000000000000000080000000000004BF9FFFF
$0ACC u32 00080000 2048
$0AD0 u32 00000000 0
$0AD4 u8[16] 00000000000000007C0A000000000000
$0AE4 u32 00000000 $0000
$0AE8 u8[16] 68616972310000000000000000000000 hair1
--------------------------------------------------------------------------------------------------------------------
$0AF8 u8[36] 010000000100010E4E39000036060000000000000000000000000000000000009EFEFFFF
$0B1C u32 00080000 2048
$0B20 u32 00080000 2048
$0B24 u8[16] 00000000000000008C090000C80B0000
$0B34 u32 780B0000 $0B78
$0B38 u8[16] 4C65667453686F756C64657200000000 LeftShoulder
--------------------------------------------------------------------------------------------------------------------
$0B48 u8[48] 010000000101010FDA380000A0020000BCFFFFFF9D020000AF04000042FCFFFFDDFBFFFFE1FEFFFFACFEFFFFA9020000
$0B78 u32 00080000 2048
$0B7C u32 00000000 0
$0B80 u8[16] 00000000000000001C0B000000000000
$0B90 u32 00000000 $0000
$0B94 u8[16] 4C656674456C626F7700000000000000 LeftElbow
--------------------------------------------------------------------------------------------------------------------
$0BA4 u8[36] 010000000100011046380000420300000000000000000000FEFFFFFFF1FFFFFF86000000
$0BC8 u32 00080000 2048
$0BCC u32 00080000 2048
$0BD0 u8[16] 00000000000000008C09000000000000
$0BE0 u32 240C0000 $0C24
$0BE4 u8[16] 526967687453686F756C646572000000 RightShoulder
--------------------------------------------------------------------------------------------------------------------
$0BF4 u8[48] 0100000001010111DA370000A0020000BCFFFFFF63FDFFFF5103000042040000DD03000000000000C60000009C020000
$0C24 u32 00080000 2048
$0C28 u32 00000000 0
$0C2C u8[16] 0000000000000000C80B000000000000
$0C3C u32 00000000 $0000
$0C40 u8[16] 5269676874456C626F77000000000000 RightElbow
--------------------------------------------------------------------------------------------------------------------
$0C50 u8[36] 01000000010001123E370000420300000000000000000000020000000F00000086000000
$0C74 u32 00080000 2048
$0C78 u32 00000000 0
$0C7C u8[16] 0000000000000000F4060000C40C0000
$0C8C u32 00000000 $0000
$0C90 u8[16] 726F636B000000000000000000000000 rock
--------------------------------------------------------------------------------------------------------------------
$0CA0 u8[36] 0100000001000113FA360000540000003C000000FCFFFFFFFFFBFFFFE4FFFFFF65030000
$0CC4 u32 08000000 8
$0CC8 u32 00000000 0
$0CCC u8[16] 0000000000000000F406000000000000
$0CDC u32 00000000 $0000
$0CE0 u8[16] 616E67656C5368617065000000000000 angelShape
Title: Re: The Console Tool (by Low Lines)
Post by: Romsstar on February 03, 2012, 11:56:49 am
Any news on the MMD format from Digimon World?
Title: Re: The Console Tool (by Low Lines)
Post by: Low Lines on February 03, 2012, 07:18:58 pm
No, there were flaws in my assumptions with the MMD format that I just can't figure out no matter how many hours/days I put into it. I've put up my findings so that other people with more time and motivation on their hands can try figuring them out. I might take another look at formats I tried figuring out later on after I've taken my brain off them for a while, that sometimes works, but it sure as hell won't happen anytime soon.
Title: Re: The Console Tool (by Low Lines)
Post by: huckpie on February 03, 2012, 08:34:00 pm
Sounds interesting. In fact I'm eager to know when will the format be hacked so that I could port Strawberry into another game.  :P
Title: Re: The Console Tool (by Low Lines)
Post by: Koetsu on February 07, 2012, 12:26:30 am
I'll say it. I've wanted to see if these DS models could be animated provided the correct files. Seeing all this progress on the digimon 3D models, think you could look into the DS stuff sometime?
Title: Re: The Console Tool (by Low Lines)
Post by: danke on February 10, 2012, 04:24:35 pm
Any ideas on these? http://www.mediafire.com/?7xn6gqvayz5p8sp

They seem to be archives of some sort.

They are from Kaizoku Sentai Gokaiger for the DS.
Title: Re: The Console Tool (by Low Lines)
Post by: huckpie on February 19, 2012, 09:03:48 am
Any news on the model formats?
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on March 17, 2012, 09:34:05 am
Hi Lowlines,
I wanted to note a "bug" with 31b (I didn't read any documentation so sorry if it's already known).
When viewing nsbmd with multiple models inside, CT show them all at the same time and there is no model selector.
Is the zoom still missing as well??

(http://img36.imageshack.us/img36/297/bugjz.png) (http://imageshack.us/photo/my-images/36/bugjz.png/)

Another little problem is that if the texture is loaded alone it look like this (wrong palette):
(http://img267.imageshack.us/img267/7243/55769560.png)

But as soon as the 3D model is loaded as well, it correct itself (correct palette):
(http://img826.imageshack.us/img826/2230/testofk.png)

UPDATE:
Sadly some more bugs in 31b.
When opening the same files (NCLR/NCGR/NCER) in 31a, everything is fine and show correct bank names but when opened in 31b, there is something wrong with the tile indexes and the bank names doesn't show.
(http://img163.imageshack.us/img163/6180/26236718.png) (http://imageshack.us/photo/my-images/163/26236718.png/)



Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on April 04, 2012, 01:59:21 pm
Back again with another little bug report.
When opening many many tabs, the cross to close the tabs disappears.
Maybe while fixing that, you can add "tab scroll arrows" like Firefox to scroll the tabs left and right (not repositioning).
Title: Re: The Console Tool (by Low Lines)
Post by: viperzerof-2 on May 08, 2012, 11:51:14 am
low lines would it be possible to add this format? its ace combat xi for the IOS it was released in 09. I have the texture format working already if I could get the models it would really help me out

http://www.gamefront.com/files/21672452/pack.rar
Title: Re: The Console Tool (by Low Lines)
Post by: henke37 on June 22, 2012, 08:52:06 pm
I have found some strange NANR files in AJ, they seem to have a "TAAU" section in them. You have any idea what that's all about?
Title: Re: The Console Tool (by Low Lines)
Post by: nix_aether on November 11, 2012, 11:25:23 pm
HELP! is anyone still using this tool
I just found out about this model ripping thing a couple of weeks ago and I downloaded this tool from the official webpage but when I use it almost all menus are disabled, I'm able to see all the models from games but I can't zoom in or out for example, or hide the vectors.
Title: Re: The Console Tool (by Low Lines)
Post by: VicVergil on November 12, 2012, 02:14:28 am
Hello everyone. Excuse me for the noobish question, but I need help badly about this:
Is there a tool to edit the texture inside the BMD0/nsbmd model, and modify it as a regular tile/map NCLR file?
You have the Death Note DS and Ninokuni DS which so have lots of images stored as 3d models.
Because I'm at a loss what to do...
And I don't know how to use Crystaltile2, no matter what I do... although I did some work on uncompressed GBA games with TileEd..

Thanks in advance... :)
Title: Re: The Console Tool (by Low Lines)
Post by: FAST6191 on November 12, 2012, 06:52:31 am
In short not really-
newer versions of Tinke have some fairly good extraction/viewing/information abilities
http://code.google.com/p/tinke/
You might also want to check out MKDS course modifier as it is getting some nice 3d abilities but that is more focused on injection and viewing.

Some of the DS 3d formats align quite well with 2d formats so you can use proper tile editors like crystaltile2 or direct something like tiledGGD- http://code.google.com/p/tiledggd/ to get at them as you would. However some of the equally valid formats ( http://nocash.emubase.de/gbatek.htm#ds3dtextureformats to say nothing of the wrappers for them) do not resemble traditional 2d formats. I have a very rough overview in my GBA and DS hacking docs http://www.romhacking.net/forum/index.php/topic,14708.0.html but it does not say much I have not already said.

Basically although most tile editing is nothing compared to the likes of a full image editor given time and effort you can do it all. 3d model editing save for times where very well known formats supported by PC programs (quite rare even on the PC) or can be snatched from hardware by openGL/directx debuggers are nowhere near that level and especially not on the DS. I should also note DS developers use material colours and lighting quite a bit in lieu of doing proper textures and given the low res nature of everything on the 3d there it is not always obvious what is what.

Naturally you have a hex editor which can do an awful lot but that is usually an abstraction too far.

@nix_aether I try not to have java on my systems which means this project I tend to follow as closely as I like so I could be wrong but I believe the early NSBMD stuff was sidelined to be rewritten for the newer versions which has probably led to what you are seeing.
Title: Re: The Console Tool (by Low Lines)
Post by: nix_aether on November 12, 2012, 10:49:28 pm
@FAST6191
Ok, so is there any update of this program comming soon? is the creator alive? xD
because I really think that this tool is WAYYYYYYY better than MKDS and all the others, I think is the best one ever done
Title: Re: The Console Tool (by Low Lines)
Post by: Auryn on November 14, 2012, 11:33:45 pm
@Ghanmi: Learn to use Crystal Tile 2. As you see above in my last post (glove), CT2 is capable to edit the 3d textures very well.
You maybe have to get familiar with some formats, compressions and maybe just paste a file in hex mode inside a NSBDM.
CT2 is capable to edit the NCGR/NSCR (NCLR is the palette and nor the tile nor the map) :)
Your problem will probably be that many images use NCER and not NSCR so you will probably not get very far.
By the way, I wonder what make you think you will be successfully translate Death Note or Ninokuni if you don't even get a table done. Don't you think you should maybe start with something much smaller and maybe in english??
Just to prove something: http://gbatemp.net/threads/ninokuni-shikkoku-no-madoushi-translation-project.310214/ (http://gbatemp.net/threads/ninokuni-shikkoku-no-madoushi-translation-project.310214/), you really think you are the first one that attempt to translate Ninokuni??
Oh, I just remembered that most graphics in Ninokuni are compressed in a non standard way so you have the same problem as you other project. When the time will be right, the right persons will come together and translate even Ninokuni.
By the way, the same apply to the Death Note games: http://gbatemp.net/threads/death-note-games-translation-project-need-help.259246/ (http://gbatemp.net/threads/death-note-games-translation-project-need-help.259246/) and http://gbatemp.net/threads/death-note-nds-games-translation.318208/ (http://gbatemp.net/threads/death-note-nds-games-translation.318208/)
Sorry to be harsh but romhacking / games translation is not the same as translate a TXT file.


@Nix_Aether: Console tool is a good tool but doesn't yet support editing so it's probably not good for your needs.
Lowlines is working on a new version of CT but he doesn't say when it will be ready.