News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The Console Tool (by Low Lines)  (Read 219942 times)

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
The Console Tool (by Low Lines)
« on: June 18, 2009, 12:08:29 am »
For the sake of archiving, I'll post links to older versions here. Just be aware that Version 3.0 onwards is considered the main (non-demo) release.

---- Demo Releases ----
Version 0.1 24/01/10
Version 0.2 24/01/10
Version 2.0 25/08/10
Version 2.5 (Mini) 06/11/10

------- Releases -------
Version 3.0 29/11/10
Version 3.1 (WIP) 04/12/2010

------- Latest Releases (2/1/2012) --------
Console Tool 3.1b (Windows)
Console Tool 3.1b (Linux)
Console Tool 3.1b (Mac OS X)
Console Tool 3.1b (All-Platforms *Updated*)

[update 7/3/2011]

This site now has an official home on my website (http://llref.emutalk.net/projects/ctool/). Be sure to check it out :p

Quote from: Original Message
Argh my 5 pages of rambling is gone...Oh wells :p
To be honest it's kind of good timing for me because lately my project has kind of warped into something bigger recently. For starters it is no longer just an NDS Tool, as I've been busily adding support for many systems which so far include PSX, Gamecube, Wii, PSP, and will have PS2 as well when I bother to get round and make a copy of a PS2 game and read that too.

What I've also started working on in the last couple of days, is something like the successor of Tile Molester :) This will mostly be useful for reading data from older systems which don't have a File Table (ie GBA), and you will be able create a custom file table for such games which will make it much easier to manage them :) It's all still a WIP though and I'm trying to rework the way my tool works so like I can have a temp/cache folder which would simplify reading compressed/encrypted data etc And like most things when you change something, you generally have to rework a few others and stuff.

So to start off the rebirth of my thread I have a few picys of all the above :p
Note: Ohs and I've been have fun rendering the console icons in Flash :) Some are just for show atm


Edit: Will start posting my images as links now since I got growled *cries*

« Last Edit: January 02, 2012, 02:52:23 am by Low Lines »

Lilinda

  • Hero Member
  • *****
  • Posts: 4539
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #1 on: June 18, 2009, 12:13:31 am »
Low Lines, can you downsize that pic or make it into a URL?
Retired moderator/staff member as of July 14th 2016

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: The Console Tool (by Low Lines)
« Reply #2 on: June 18, 2009, 02:18:56 am »
Friggin' awesome on all counts. 83
In the event of a firestorm, the salad bar will remain open.

Killa B

  • Hero Member
  • *****
  • Posts: 1153
  • Fallen Angel
    • View Profile
    • Killa B
Re: The Console Tool (by Low Lines)
« Reply #3 on: June 18, 2009, 02:23:03 am »
You could always use an ImageShack thumbnail. :D


I always dreamed of doing a Pokemon hack

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #4 on: June 18, 2009, 03:39:57 am »
Bah, that's work! Besides I upload all my images to my website (though I've been too lazy to update the project page). You all can suffer and actually have to click on my links *evil laugh*

KaioShin

  • RHDN Patreon Supporter!
  • Hero Member
  • *****
  • Posts: 5697
    • View Profile
    • The Romhacking Aerie
Re: The Console Tool (by Low Lines)
« Reply #5 on: June 18, 2009, 08:59:54 am »
Can your tool replace files inside gcm images (Gamecube ISOs)? If so, awesome :thumbsup:
All my posts are merely personal opinions and not statements of fact, even if they are not explicitly prefixed by "In my opinion", "IMO", "I believe", or similar modifiers. By reading this disclaimer you agree to reply in spirit of these conditions.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #6 on: June 18, 2009, 10:37:34 am »
Not yet, it only goes as far as opening them into a treeview. But that'll definitely be something I'll have working eventually. Modifying Wii Games/WADs might not happen though :p All the data is hashed and what not, which also include a signed check by Nintendo which can't be faked without knowing their private key. That being said, hacking them would be possible, just they wouldn't pass checks or anything, and doing that is kinda complex :p

Vanya

  • Hero Member
  • *****
  • Posts: 1379
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #7 on: June 18, 2009, 12:40:47 pm »
Lookin' awesome! A successor to Tile Molester sounds really refreshing right about now! Thanks for all the effert on this I can't wait to try it. ^_^

Klarth

  • Sr. Member
  • ****
  • Posts: 484
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #8 on: August 29, 2009, 12:50:04 pm »
Any progress on this?  I'm really interested in Console Tool and hope that it'll be available before the end of the year so I can include it in the doc I'm writing.  Also, what kind of code signing does Wii use?

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #9 on: August 29, 2009, 11:46:16 pm »
Meh...sorry, I started doing Uni since mid-year, so most of my projects are basically suspended indefinitely or until I am really bored :p

You should check out the info at WiiBrew if you want to know more about the Wii. That's only one page, but if you use the search bar on the site you should find just about anything you want to know about the Wii. Am assuming you know everything on the Wii is also encrypted.

Signing wise, I know how it all signs stuff and that, but like I'm not smart enough to implement it. Was needing it to make a Wii Save Game editor (which I made specifically to edit Pokemon Ranch Data), and it's not rocket science modifying stuff, but the signing screws you up :S Save games for instance use Elliptic Curve Digital Signature Algorithm (ECDSA), which is a real pain to try and code (and beyond me at the moment), though I understand how it works and all.

[edit]

Actually I'm kinda bored today :p I think I'll work on it a bit!!

[edit2]

This probably isn't very nice for those who want to see progress. But I've kind of started a new project (as in a fresh program), and begun with a solid browser tool. This is completely custom since Java doesn't natively support pretty much any of the typical Windows Explorer functions, and I'm using the FamFamFam Silk icons for all the files/folders etc.

I figured this was a better choice than having the button menu in the earlier version I did, since I can just specify a custom file filter to only display certain files etc The "Select" button would be like you copy button, you pick a file (say a modded file you want to import), and then you can import that into a container file such as an iso or whatever :p It loads much quicker than that crappy file chooser window too. You can also add directories to the drop down list at the top for quick and easy access.

Also it's not visible but the tree view only loads what is visible in the tree, since scanning your whole file system every time you load up the app would take forever, it was a pain in the arse to implement but I think it'll make accessing container files, especially ones with shitloads of files in them, load much quicker.

The file information is actually in the tree model except it gets clipped when it gets rendered (ie you have a folder "D:\Emulation\Games" but it gets clipped to "Games"). Makes referencing and stuff much easier as I don't have to have a separate array working alongside it.

Anyways here's a screenshot...And I'll get stuck on re-adding a lot of the stuff from the older versions soon :p Am really trying to stick to a solid GUI so I won't have to change it every time I find a new shortcut/trick to make my code better.

« Last Edit: September 01, 2009, 12:19:27 pm by Low Lines »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #10 on: November 06, 2009, 07:27:54 am »
Okay its been a while since I last updated this project. I've been rather busy with my University studies so I haven't had much time to work on it. That said, I'm not in my 3 month holidays and back working on it again!

The GUI has had a lot of improvements since, making it a lot more stable and less glitchy, and you should be able to open files that are inside files that are even compressed, without having to extract nothing.

For the moment I'm focusing on re-adding the common NDS formats to the library and doing up GUIs for each type. I am still not sure how I plan to add support for custom plugins, so I'll leave that alone for now.

I've also been working on some Gamecube formats as well, but I'll add those later.

Lastly if anyone can share with me some information on LZSS compression, it would be greatly appreciated! All the documentation I've found seems to only talk about the process, and not how to read the operation bits :/

More to come soon, will post some screenshots up later of NDS formats running.

nonme345

  • Full Member
  • ***
  • Posts: 157
    • View Profile
    • JPN Translations
Re: The Console Tool (by Low Lines)
« Reply #11 on: November 06, 2009, 08:34:15 am »
Lastly if anyone can share with me some information on LZSS compression, it would be greatly appreciated!

Here's a possible source:
So You Want To Be a Hacker? Part IV: Compression Formats

Tauwasser

  • Hero Member
  • *****
  • Posts: 1392
  • Fantabulous!!
    • View Profile
    • My blog
Re: The Console Tool (by Low Lines)
« Reply #12 on: November 06, 2009, 09:02:39 am »
What platform do you need info for? NDS, Wii? I head those were pretty special cases.
The regular LZ77 compression mentioned in the handbooks already is LZSS, yet, there seem to be new files that contain "LZSS" as magic bytes and don't quite follow the traditional scheme.

cYa,

Tauwasser
« Last Edit: November 06, 2009, 09:08:35 am by Tauwasser »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #13 on: November 06, 2009, 09:18:09 am »
Wells in Pokemon Colosseum/XD there are these FSYS formats, which are a type of archive. I can open up those, but alot of files in those had the "LZSS" magic at the start of the file :p

I also recently stumbled across Thakis's work on Gamecube File Formats, and his documentation is clear enough for me to be able to interpret it :) His work includes Gamecube Model Files (BMD) with animations, Image Files (BTI), video files (THP), and Gamecube Archive Files (RARC). I've already implemented RARC along with Yaz0 compression, though I haven't added it to the GUI yet.

I've also been looking at some of the generic PSX formats as well among other things...

@nonme345
That links looks promising! I'll have a look at it later once I get these NDS formats added.

[edit]

Okay so adding those NDS formats hasn't quite happened yet :p I've been fixing up the GUI more. As you can see, the file browser has had a fair bit done to it, as it looks prettier and I've also added the icon scheme as well as highlighting the text blue to indicate a file is compressed (which is present in the Tree View Browser as well). It will also scan the start of every file to see if there are any Magic codes as well as compare file type extensions in an attempt to make it fairly compatible.

Currently you can view the hex of any file through the browser and open up NDS roms and NARC files. But it's just a matter of adding more formats really.


[edit 2]

Going back to compression formats now, I noticed that there are LZ77 files where the Reserved Bits equal to 1 and not 0 like they normally do. I can say for certain that some of these files are definitely LZ77 compressed because some even have the "LZ77" magic id at the start of the file.

The problem is that sometimes comes across a Copy Operation that points to a negative offset. There are a lot of these files inside the Kingdom Hearts 358-2 Days game. They have a ".z" at the end of the file name, and if you open up some of them, there's an obvious Magic id near the start of the file of what's it is compressing.

Possibly this is a variant of LZ77 Compression, which manipulates the operation bits differently.

[edit 3]
Okay I've wasted a couple of hours trying to figure out how this variant of LZ77 works with little success. The only thing I can tell for certain is the the copy bits is "minus 1" as opposed to the type "minus 3" value. Was using a compressed NCGR file from kingdom hearts and trying to match up the constant data with the output.

If anyone can figure out how this variant works, I'm all ears. For now my program will detect it but will refer to it as "LZ??" and thus won't even try to decompress these files since all it will do is cause errors!

The ".z" files in Kingdom Hearts appear to be a combination of LZ77 and this variant as some of the files will extact no problem while others will just spit the dummy on me >_<

[edit 4]

Going back to the NDS formats now, I'm up to re-adding NSCR and NCER. NSCR is pretty straight forward, however NCER is a bit tricky.

From what I've gathered, there are two different NCER structure types, indicated by a byte flag I like to refer to as the "Bank Type".

Type 1 is the structure I figured out ages ago when I started this project, it groups cells (a cell being a block of tile data of varying size that makes up a part of a sprite) into sprite frames. And seems to include to extra sub sections in the file (LBAL and TXEU) that hold frame information such as it's name.

Type 0 seems to be radically different. First off the bank data indicates that each bank object is 0x10 bytes long, so it now includes another 8 bytes of data I'm not sure about what it does. Like type one, there is an offset that points to another section at the end of the Bank Area. In type 1 this area is the cell data area (0x8 bytes long), but in type 0 this data is different again appearing to be 0xC bytes long. Then after this area is ANOTHER area which looks more like tile offset data with varying lengths.

It's going to take a while to figure this one out :p

On a side note, while I was looking through the kingdom hearts files, I can say for certain that "D2KP" files are a grouped combination of several of these generic formats (ie palette, graphic and either cell & animation or map/screen). However as these files are also ".z" compressed, my decompressor won't always unpack them properly :(

[edit 5]

To help with decoding formats, I thought I'd try and add multi-file loading support, and I'm quite happy to say that it was very VERY easy to do with the file access structure I use in my program. Every time a particular file type (as generic not format specific; ie pal, tile, map, etc), it will remember this file as the last opened one of its type. Then when you load up a file that works in combination with this one it will check to see if this file exists and then load it as well. And given the way my program accesses files, it won't matter if the files are stored in two separate places or even compressed...as long as the access path is valid, it should load it :)

[edit 6]


A site for sore eyes! This was one of the first bunch of graphic files I tested way back when I started this project. It's a nice feeling to actually be catching up to where I was the first time :)

[edit 7]


Okay reading sprite sheets (NCER files), is still a pain (mostly because it's a format I've worked out almost entirely on my own and obvious am no expert at it). I've got them working up to the same point I had it in my old version, and I'm just trying to fix all the problems there are with it to make it more compatible.

Aside from the main problem of there being two different file structures for NCER files, there is also the problem with handling non-tiled graphics such as those found in Super Princess Peach. When it references to graphic data, it assumes the data is already stored in memory in a 8x8 tile format (as in the first 64 pixels are tile 0), so I'm racking my brain at the moment trying to remember how to convert one between the other as I originally just had a function that takes all this information and outputs a buffered image! My brain doesn't want to work for me right now so I'm going to take a break from that one for the time being...

I'll start having a look at the animation files (NANR) instead :)

[edit 8]

Reading the animations are relatively easy. In another small project I was doing I had to read Wii Save Files and animate the icons in a browser...Using that knowledge, filling in the gaps an Arcnor's documentation is simple enough...there's only a few bits of data I'm not sure about yet, and can't really test them until I build the GUI to manage viewing animations.
« Last Edit: November 11, 2009, 03:40:36 am by Low Lines »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #14 on: November 14, 2009, 01:10:03 am »
I've been looking at some graphic formats found in Kirby Squeak Squad, which were partially supported in Tahaxan, being files with the magic stamps BCLA, BCGA, BCEA or BRCA. They are simply raw data files with a custom header at the top.

The significant thing though, was that I figured out *most* of the data format for the other Sprite Sheet Type I mentioned in my earlier posts. Basically the format is very similar to the one I've worked out, it's just differs slightly in some aspects, such as lacking any header areas, or a Name Label section at the end of the file.

There's still a few things I have to figure out first, but the above picy is of a BCEA file (Sprite Sheet) that follows this other format, some kind of rock dude from the Kirby game. I haven't looked at the BRCA format yet, but all the other ones have been added.

[edit]

Just a quick thing...I finally figured out how sprite sheets store palette information. The picy below shows a single sprite using 3 different 16 color palettes for individual cells (aka, kirby, the cloud, the cake).


I've also got to figure out a what a few of the unknown bytes do as for instance going through the file above, about 15 out of 133 sprites aren't positioning the cells properly, so there must be something different about them I am yet to find out...

[edit 2]

I have been adding a few more file formats, though I also spent I bit of time cleaning up my file accessing code, which I had found out didn't do all of what I expected it to do -_- This also included some speed enhancements with scanning files to detect their format.

The latest thing I've been working on is adding support for NDS sound files...starting with the very common SDAT format, which is actually more of an archive than an actual sound file. From what I understand there are 8 sub sections in these files, 5 of which represent actual files, while the other 3 contain data for combining these files etc. So what I've done is group each type of data in sub folders, using a fake file object to reference to non-sub file data that is found in the SDAT. I've tested it against a couple of SDAT files from different games, and it isn't too bad...except some files for whatever reason reference to nowhere etc. Once I fix up the pile of bugs in my code on this, I'll start working on the actual sound file formats (SSEQ, SSAR, STRM, SWAR, SBNK), which are contained inside SDAT files :)

[edit 3]

Wells I've just spent the last half of a day learning all about MIDI files...why? Because SSEQ files are a custom type of them. I thought it would be wiser learning how these little suckers work before trying to write some code to decode SSEQ files on the fly, but man...MIDI files are really quite complex!! But actually once you understand them, they're a lot easier to understand than some of the much more simpler file formats. That being said, the next thing on my list of to do things is obviously write a SSEQ decoder now :) Also Java has (wells I think) a much nicer MIDI sound library than the default one Windows. What would be nice though, is to learn where in the NDS file, it actually stores the sounds/algorithms for playing back sequences. I'm assuming it'd be some kind of generic file/data maybe even stored in the BIOS? Another thought was that they might be in the Utility.bin file, but that isn't always present in some NDS games so I dunno :/

[edit 4]

Okay it would appear the SBNK files may contain the sounds/instruments needed by the SSEQ files (Seems kind of obvious when you think about it), though there is very little documentation on them, so it might take a while to learn how they work. I haven't had a lot of success in porting SSEQ2MIDI to Java, simply because of the excessive amount of data these files contain, but I'll get there.

[edit 5]

Okay I've been adding all of the sound formats today. I found Kiwi.DS's SDAT documentation and it helped get a better understanding of it all :) However, Java has a somewhat limited Native Sound support, and it can only manipulate data in PCM format. Apparently it can play all sorts of different data formats but it only seems to do if said formats are in their proper file format, where as I'm trying to read stuff on the fly.

From what I understand, NDS sound files come in three types. PCM 8-Bit, PCM 16-Bit and IMA ADPCM. I can play the first two types directly out of the file without having to converting anything. Unfortunately the most common one is the last one (ADPCM), since it has better compression than the other two (from what I've researched). Now I don't have a lot of documentation on it, and I doubt it will be simple to covert it to PCM.

With luck there'll be a nice Java add-on I can use rather than the limited default package (as was with OpenGL), otherwise I'll have to learn about ADPCM >_<

Oh and the reason why I'm posting this, is because I have spent the last couple of hours listening to horrible static every time I try playing a STRM file, and I finally came across one in PCM 8-Bit format. By coincidence it was the Title Screen music for Kingdom Hearts 358/2 Days, and it sounded so wonderful to my ears!!! (Well what with all that static, anything was better lol)

[edit 6]

I've been having a go a writing a ADPCM decoder...and I've had a bit of success. Playback has gone from horrible static to clearly understanding but still fuzzy :p I've been using this site as a reference (it's actually quite hard to find solid documentation on ADPCM!!)

The real pain is that Java only works with signed variables so I've got to add extra code to translate stuff.

Here's a basic breakdown of my code...
Code: [Select]
outputBuffer.size(buffer.size * 4);
stepIndex = 0;
predictor = 0;
step = stepTable[stepIndex];
for (i=0; i<buffer.size; i++) {
for (j=0; j<2; j++) {
nibble = (buffer[i] >> (4*j)) & 0xFF;
stepIndex += indexTable[nibble];
if (stepIndex < 0) {
stepIndex = 0;
}
if (stepIndex > 88) {
stepIndex = 88;
}
signedNibble = nibble;
if (signedNibble > 7) {
signeNibble -= 16;
}
diff = (2 * signedNibble + 1) * step / 8;
predictor += diff;
if (predictor > 0x7FFF) {
predictor = 0x7FFF;
}
if (predictor < -0x8000) {
predictor = -0x8000;
}
step = stepTable[stepIndex];
outputBuffer[pos++] = predictor & 0xFF;
outputBuffer[pos++] = (predictor >> 8) & 0xFF;
}
}

I've been playing around with the variables a bit *hoping* I'd strike gold and it would just be perfect...but no....Swapping between lower and upper nibble first doesn't do nothing, changing the nibble shift to 5 seems to make the sample smoother but more muffled. I figure that there either if something wrong in my code (bare in mind this was ported to Java by me), or there's some Nintendo proprietary thing going on in there. Btw if it wasn't obvious, the reference table I used are the same ones I found on the website.

Luckily I've been doing some digging and game across VGMStream, which apparently has support for most Nintendo DS audio formats, so I'll check their source code and hopefully that will sort out my problem. But if anyone is an A class expert on audio decoding, be my guest :)

[edit 7]

Still not much luck for ADPCM sound :(
I've compared my code to various implementations of it and even found out that GBATek has some info about it also.
I'm still getting a heck of a lot of noise and/or jumpy playback.

I have however ascertained that the data is stored in blocks (size is specified in the file header), and that the number of samples in each block indicates 4 "unused" bytes as such means that at the start of every block it updates the 3 variables with the first 4 bytes of data.

Heck for all I know it could be simply Java and how it handles data (unsigned/signed, etc), I don't have enough understanding of sound playback to know myself >_<
« Last Edit: November 20, 2009, 05:27:48 am by Low Lines »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #15 on: December 13, 2009, 09:30:06 pm »
Being the holidays for me, I've been doing all the stuff I've been wanting to do, which included a Stargate marathon ;D

Nonetheless I've still been working on this project off and on, except it's been slow because I'm trying to make sure the GUIs work properly with no bugs :D

Lately my focus has been in re-adding the modeling functionality, which was present in my older versions, thus making the current build finally beyond the point in which it's origins ended up. Textures are up and running with a working GUI, and I've started on the modelling GUI, which is going to be more user friendly than my old one :)

Here's a screeny of some dumb tree from Pokemon from a texture file. I'm planning on adding exporting functionality later (as in to PNG) since its just a matter of exporting a buffered image.


I've also started adding preliminary ISO capabilities, however currently my sector reading code runs a bit slow (especially for PSX games in raw BIN format).

[edit]

I forgot how much data modelling files have :-\
So far I've got partial implementation done for display 3D models. That said, only some of the ones I've tried are displaying properly, though I'm also trying to construct a genericish model viewer at the same time :p

To get a better understanding of the NDS generic model format, I've been figuring out how data is stored in "BMD" files from Super Mario 64 DS. They are basically a simplified version of the NSBMD format, which I figure means they were possibly make with either a custom program or an older version of Nintendo's SDK. I still haven't figured everything out yet, but below is the model for fat mario :)


[edit2]

There are several bugs I had worked out in my older version which I have not yet been able to work out yet, and for whatever reason, I seem to have lost my solution since...In other words, I have to re-figure-out what I did last time from scratch.

It's good in a way though because I seem to have a lot better understanding now than I did then, simply because I've spent a lot more time working on it, like the fact that my app renders stuff a lot quicker now than it did before, particularly with large 3D maps.

I took a screenshot of Peach's Castle (Exterior) from Super Mario 64 DS. The holes in the textures is a combination of an error in my 4x4 Compressed Texture code, and a bug in my vector code centered around the scaling factor which I recall tormented me the last time I worked on it.


Hopefully, I'll figure out the problems so I can get started on Animations :)
« Last Edit: November 17, 2010, 08:44:35 pm by Low Lines »

Tauwasser

  • Hero Member
  • *****
  • Posts: 1392
  • Fantabulous!!
    • View Profile
    • My blog
Re: The Console Tool (by Low Lines)
« Reply #16 on: December 19, 2009, 03:49:06 pm »
It's looking pretty good again :D
Have you made any additional changes to the browser? What other stuff is not working that was working in the old version?

cYa,

Tauwasser

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #17 on: December 20, 2009, 12:41:59 am »
Wells once I get these couple of fixes in the 3D done, everything will be working from the old version, but it's already far more advanced in its own respect. I suppose the only other thing I'd have to add to make it truly surpass the old version is adding the ability to export graphics to PNG etc :p

The model browser itself can disable/enable Objects, Polygons and Materials along with partial highlighting functionality by changing the vertex color to red. I'm using Maya as a basis for the UI though personally I barely know how to use Maya -_- You can also zoom in/out and perform a simple rotate of the model with the mouse, but they both need a little more work. I want to add some sort of camera control later rather than just editing the raw translate/rotation/scale variables.

The browser has had very little changed lately since I've been working mostly on specific file types, but I'm working towards adding a live preview thumbnail which would display graphics, video, sound, models and maybe more.

So as of right now supported formats are...

Archive Formats:
RARC Archive (Gamecube)
ISO Image (Partial Support)
Nintendo DS Roms
NARC Archive (Nintendo DS)
WARC Archive (Wii - Initial Support)

Graphics/Palettes:
BCGA (Kirby Games
BCEA (Kirby Games)
BCLA (Kirby Games)
NANR (Initial Support)
NCER (Partial Support)
NCGR
NCLR
NSCR

Sound/Video:
MIDI
SDAT and its sub-types (Partial Support)

3D Modeling:
BMD0 (commonly known as NSBMD)
BTX0 (commonly known as NSBTX)
Super Mario 64 DS BMD files

Along with file scanning, you should be able to see a decent amount of file support in most Nintendo DS games, unless they use a lot of custom formats.

I still need to update parts of the GUI, with some updates to graphic handling I came up with while working on the 3D stuff, along with a much needed overhaul of the hex viewer, which I have been neglecting for ages, but possibly after I've done all that as well as solidify support the all the above formats (and probably a few more by then), it might be worth releasing so other people can have a play around with it :) Keep in mind though, it would only support viewing stuff and exporting it, because I'm trying do a bunch of different things at once, I'm leaving actual file editing to later, since it will require a massive amount of work to build GUIs for it :p I've got at least two months of holidays left, so hopefully I can achieve most of the above in the time I have before I get back to Uni :)

My current goal at the moment, is to get one of the Pokemon maps fully rendered, which would include loading multiple models at once and positioning them on the area map, and adding collision mapping :) Though I also want to get Model Animations working too!

In case anyone is interested in reading up on some Pokemon ROM Editing notes, here's the page I'm using as a reference.

[edit]

Okay I figured out what I was doing wrong...and as soon as I did, I remembered I had done it before too (déjà vu? lol). In NSBMD files, each object has a Transform Flag, and basically I was reading only one byte instead of two (hence causing some flags to result in zero). Changing this fixes the pivot/rotation factors, however there are still some other things as well. The below is old Petey, now looking much better than he did this morning, however a bit of his face is missing. Now I know I have fixed this before because I've still got the original screenshot I took a year ago, so now its all a matter of figuring out what I did to fix it. On another note, there are also still a bunch of variables I haven't implemented (mostly because I don't know what they do), which affect models from other games, such as materials for models from Phantom Hourglass. Link sadly also looks like a flag pole at the moment too -_-


[edit 2]

The problem with models like petey is that each polygon also seems to have an initial scale factor as well, which is why the first chunk of the mouth appears as though its missing, it actually is there, just not scaled properly. So far I haven't been able to isolate the exact bit of data which specifies the initial scale, but by not resetting the overall base scale value to 1 it seems to fix it on the second refresh of the screen. Not a proper fix though, and would no doubt be prone to errors once more things are added.

[edit 3]

Taking a break from models, I've started looking at model animations :) Although kiwi.ds's code for reading NSBCA files is far from incomplete, it's given me a start. It's formatted in very much the same way as NSBMD and NSBTX files.

The basic structure up to where I am is as follows...

BCA0 Header (0x10 bytes)
 - Section Offsets (since there is only one section, this is only a single 4 byte offset)
 - JNT0 Section (contains 3 sub sections which are present in both NSBMD and NSBTX files, namely UNKNOWN, DATA, NAMES)
------------------------------------------
Following this each animation jumps to its own section that begins with a J_AC (0x4A004143) magic stamp.
Each J_AC store the initial settings for each animation as well as points to places in the latter part of the file relative to the start of the J_AC. The offsets for every J_AC point to the same two places in the file.

kiwi.ds only got as far as the J_AC sections, which I figure is because they have a really weird way storing information.

For each object in a J_AC section, it seems to hold the base settings for Translation, Rotation and Scaling, as well as the Object # and a currently unknown byte (which always seems to be zero).

The settings a referenced using a 2 byte flag which is as follows...

xxZYXBSCxRZYXBTx

if T, R, or S equal to 0, data is present for each of their corresponding types (ie Translate, Rotation, Scale).
Both T and S control the following 4 bits to their left, being B, X, Y, Z.
R controls C.

if the corresponding flag equals to 1 a DWORD follows (except for Scale XYZ which use two DWORDs).
otherwise two WORDs and a DWORD follow.

This is how I've interpreted the data, which seems to fill in all the blanks in kiwi.ds's code, and so far has all data that follows the flag present and accounted for. So now I just have to figure out what the hell it all does :p

This isn't a very good explanation, but hopefully it sheds some light on the currently unknown NSBCA file :p

More to come once I figure that out.

[edit 4]

Animations are a pain.
When a flag is set, the data that follows are the values that don't change throughout the animation, essentially if say a character is walking to the left, the Y value may remain constant for the whole animation. But when it's not set, 8 bytes follow which reference to what happens for each frame in the animation. Breaking it down, the first two bytes always seem to be zero, and the last four bytes are an offset relative to the start of the J_AC section, which seem to always point to somewhere after the second offset found in the J_AC header.

The first offset section is made up of 6 byte chunks, the first two bytes are some kind of id or number and the last 4 bytes make up two word size signed values or a single signed dword. So I figure this is some kind of matrix table.

The second offset section seems to hold a variety of different data. Most often the first chunk seems to hold multiple two byte chunks, the first byte incrementing by 1, but this varies.

The problem I am having is that there doesn't seem to be a constant size for data referenced from the second section. Assuming references don't overlap, it seems that the third byte in the above mentioned reference data specifies how the data is stored, and more importantly right now...how big it is.

Say the animation is 30 frames and the Translation X value changes throughout it? So assuming the first section is the matrix table, you would need at least 30 bytes to properly reference which matrix from the table needs to be loaded into the current matrix. However I've seen data which which is 32 bytes, or even less than 30 bytes which is somehow meant to represent the entire 30 frames for that particular variable. Then there's also the possibility that the CAN be a zero sized first section as well (meaning there is no matrix table). There's at least 7 different types which so far have made no sense to me at all :(

So like I said, animations are a pain, but I'll figure it out eventually :)
« Last Edit: December 24, 2009, 10:32:07 am by Low Lines »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #18 on: December 31, 2009, 10:16:48 am »
Lately I've been doing a bit of GUI cleanup as wells as updating the code with some of the new stuff I've been doing with the 3D Modeling section. Right now I am determined to get Cell-Based Sprites 100% working. I've managed to fix a few problems I've been having with them, getting most sprites to display properly now, but there's still a few bugs with non-tiled sprites. While adding a decent grid layout to the GUI I've realized what the rest of the data in cells actually do, I'm just not quite sure how they work yet :p


The above is probably one of the most complex sprites I've come across yet (having a whooping 38 cells in it), and as you can see, there are errors. The problem lies in the last bit of unknown data found in each cell. For this sprite, the value is 2, which somehow is meant to shift everything to the left more. The grid is made up of 4x4 64x64 blocks, but with bigger sprites such as this one, cells can be positioned to stick outside this range, meaning the maximum size for a sprite of this format is 384x384, 192 pixels in any given direction from the center point (0,0). Once you take into account the size and position of the grid, alot of sprites I originally thought were displaying properly are actually wrong because they aren't being centered, which is what the last bit of data is somehow supposed to do.

So once I figure out how to position the cells properly and fix the bugs with non-tiled sprites, Cell-Based Sprites should work 100%

[edit]


Fixed. The unknown data is basically 4 flags, two of them describe flipping horizontally and vertically and the other two flags  specify whether their respective axis uses a different shift table.

Each cell has three variables that make up the amount of shift is added from the base grid position, normally these are just multiplied by 16, 4 and 1 respectively, however when the X or Y flag is set, the first shift variable uses a different set of positions instead. It's rather hard to explain, but basically whatever I'm doing seems to fix all problems with cell positioning in every Cell Sprite I've tried. There are probably still a few bits missing here and there, but until I find a file that uses them, I can't really do much.

The only other major problem I'm having is with referencing tiles from a non-tiled image. There are 4 types of image files. Firstly there is the 8-bit image, which I'm pretty sure is always non-tiled. Then there is there 4-bit tiled image which is the most common one. The last two are 4-bit non-tiled. One is basically treated as a single image (typically with a width of 256), where smaller images are all packed in neatly into a rectangle.

The last type, which is obviously the one I'm having trouble with is basically multiple images that sequentially follow each other. For example, the first image is 32x16, the second 16x16, and the third 16x32.

So far I've only come across images of this type in the game Digimon Championship. The NCER files that use them are also interestingly use the other banking method, which has alot more unknown data. Hopefully I'll figure it out eventually :p
« Last Edit: November 17, 2010, 08:46:49 pm by Low Lines »

Spikeman

  • Hero Member
  • *****
  • Posts: 1063
  • *unce unce unce*
    • View Profile
    • None at the moment, check out my Last.fm page instead?
Re: The Console Tool (by Low Lines)
« Reply #19 on: January 02, 2010, 07:42:57 am »
Just want to say keep up the good work man. This is one of the coolest projects being worked on in my opinion. :)
Open Source Hacking Projects: Guru Logic Champ, Telefang 2, (Want more? Check out my GitHub!)