News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The Console Tool (by Low Lines)  (Read 250835 times)

Mauron

  • Submission Reviewer
  • Hero Member
  • *****
  • Posts: 551
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #140 on: September 27, 2010, 03:27:44 am »
If you don't mind my asking, how are you handling the variations between compression formats of the same type?

Are you making fifty million variations of an LZSS routine, or do you have one routine that can take variations into account?
Mauron wuz here.

Zerox

  • Jr. Member
  • **
  • Posts: 16
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #141 on: September 27, 2010, 03:32:28 am »
Low Lines I hate to be a bother but is there any chance you could compile your program into a .exe file as opposed to .jar?

I realize that your intent is full multiplatformability but I would love to try something with a .exe version of the program and I think it might fix some issues my computer seemed to have. I'm not asking for new features just the same as your newest demo only as a .exe file if that's possible.

BRPXQZME

  • Hero Member
  • *****
  • Posts: 4572
  • じー
    • View Profile
    • The BRPXQZME Network
Re: The Console Tool (by Low Lines)
« Reply #142 on: September 27, 2010, 03:40:52 am »
It wouldn’t fix anything. Java don’t swing that way.
we are in a horrible and deadly danger

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #143 on: September 27, 2010, 03:49:03 am »
@Mauron
At the moment, it only decompresses the one NDS uses, I haven't actually dwelled too much into other stuff that use LZSS mostly because I don't understand them well enough. But when I get to that point I'd definitely try and write it to allow for variants, the iffy part is detecting compression in a file properly.

@ZeroX
I don't think Java exports to exe, I know it can be done, but its a lot of mucking around and as you said, it will then only work on Windows. So that'd have to be a no there. The only way you could do it would to port the code to another programming language, which I'm not going to do since I'm quite comfortable with Java and I think I've already given myself enough work to do.
If you are having issues running the program, by all means post them up so I can see what's going wrong.

@BRPXQZME
Was that a joke? Because I found that funny, "java don't swing that way" haha.

BRPXQZME

  • Hero Member
  • *****
  • Posts: 4572
  • じー
    • View Profile
    • The BRPXQZME Network
Re: The Console Tool (by Low Lines)
« Reply #144 on: September 27, 2010, 04:08:26 am »
Well, I wrote a post in an attempt to explain how compiling Java to a Windows executable, while possible (namely, through GCJ), will cause more problems than it solves. The “why” behind it is essentially that the JVM is the only implementation of Java with the entire API done and working, and the only implementation of Java with the language compiling and running relatively error-free. But translating the entirety of the details in lay terms was taking more than two paragraphs, which IMHO causes more confusion than it clears up.

In the end, I had a pun in there somewhere and couldn’t resist throwing it away, so I deleted the rest of the post. I’m a sucker for groaners; it runs in the family ;)
we are in a horrible and deadly danger

Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #145 on: September 27, 2010, 08:40:03 am »
Wrote a quick batch routine and exported all single frame sprites (normal, shiny, gender, forms) for BW to PNG :) Some of the new ones look awesome! I'm not going to put them up since this image ban is on, and I also doubt I'm the first to extract the images, but its an odd feeling, going back several years I was one of those noobs following fileb and all those other hackers as they hacked the games :)
You are correct. The palettes at the end are alternate palettes for Acreus (#493 = files 9860-9879) by the way.

Quote
Also with the nmcr and nmar, this is still an theory but seeing how the ncer and nanr handle animations of a pokemons limbs separately, I think these last two files are like layer on top of that. So the nmcr would position each body part (which are a group of cells), and the nmar would manage the animation for them. I'm still only up to adding NCER cells so I can't test my theory yet, but due to the similarities in the file structures and the way pokemon sprites are animated, I think it's highly likely :)
I suppose I'll put my info so far on the NCER format here, if you need it or want to compare notes;
Code: [Select]
/*
 * NCER:
 * <NitroFile("RECN",version,fileSize,headerSize,numSections)>
 * CEBK cells;
 * LABL labels; // not always present.
 * UEXT uext; // ?? only has one value, 0 or 1. not always present.
 */
/*
 * CEBK; Cell Bank. header size: 0x20
 * <NitroSection("KBEC", sectionSize)>
 * u16 numCells;
 * u16 extendedCells; // 0 or 1. if 1, uses extended cell data
 * u32 unknownConst; // always 0x18 ?
 * u16 padding; // is it?
 * u16 flags;
 * u32 partitionOffset; // if 0, no partitions. rel to section start.
 * u32 padding; // is it?
 * u32 padding; // is it?
 * CellData[numCells] cells; // CellDataEx if extendedCells == 1
 * OAMData* oams;
 * u8* padding; // may not be present
 * PartitionData[numCells] partitions; // partitions the graphics for each cell
 *
 *
 * flags: ABCDEFGH IJKLMNOP
 * A->M: only seen 0 here
 *    N: if 1, uses sub-images
 *   OP: all tile indices in OAM data should be left-shifted this much  (and then right-shift 1 if 8bpp image)
 */
/*
 * CellData:
 * u16 numOAMs;
 * u16 something; // ????
 * u32 firstOAMOffset; // rel to first OAMData of section
 */
/*
 * CellDataEX:
 * <CellData>
 * s16 xMax;
 * s16 yMax;
 * s16 xMin;
 * s16 yMin;
 */
/*
 * OAMData:
 * s8 xpos;
 * u8 xflags;
 * s8 ypos;
 * u8 yflags;
 * u16 data; // &0x07FF = tile index. >> 11 = palette index(/offset?)
 *
 * xflags: ABCDEFGH
 *  AB: 1st index of dimension matrix
 *   C: ??
 *   D: ?? only seen 0
 *   E: ?? only seen 0
 * FGH: ?? PKMN BW has 111 or 011 here; which is the only game I've seen non-000
 *
 * yflags: ABCDEFGH
 *  AB: 2nd index of dimension matrix
 *   C: 1 -> mirror Y
 *   D: 1 -> mirror X
 * EFG: ?? only seen 000
 *   H: ??
 */
/*
 * PartitionData:
 * u32 start; // in pixels
 * u32 length; // in pixels
 */
The dimension matrix indices are the 'width' and 'height' on your old(?) format page.
'sub-images' means the graphics data is non-tiled, and the referenced OAMs are sub-images of the complete graphics. The start offset remains the same however.

This used to work for most games, however for this latest pokemon game, the OAMs in a cell do not seem to be put in the correct place any more. (unless they do not use the NCER, which I find unlikely, or the NMCR/NMAR can correct that) There's about 4 pixels between the OAMs =/

I'm available if you want to collaborate on this by the way. I know I'll be continuing my own investigations in any case. =)

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #146 on: September 27, 2010, 09:57:14 am »
Oh goody some fresh docs to go over :)

Am not used to calling them OAMs :p I did notice though that when you load them into my old app you do see that 4px gap you spoke of, but I'm thinking it is just a mis-assumption of the format cos it sounds awfully stupid to use a second file to fine tune positioning.

I've uploaded the handful of docs I've been working on as of late including the NCER one, note that I'm still using my WIP site so links may change in the future...It's been yonks since I've really looked over the NCER format, and I was more taking stabs at it back then. I don't think I've made any real changes to it as of it, but will do as I work more on the format.

Mauron

  • Submission Reviewer
  • Hero Member
  • *****
  • Posts: 551
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #147 on: September 27, 2010, 11:35:06 am »
Low Lines, thanks for the feedback. Much appreciated.
Mauron wuz here.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #148 on: September 27, 2010, 08:51:14 pm »
Here are my preliminary notes on the NMCR format...
Code: [Select]
====================================================================
Nitro Mapped Cell Resource (NMCR) ??
====================================================================
0x0000    0x4    Magic (RCMN|0x52434D4E)
0x0004    0x4    Constant (0x0100FEFF)
0x0008    0x4    File Size
0x000C    0x2    Header Size (0x16)
0x000E    0x2    # Sections (Always 1)
--------------------------------------------------------------------
0x0000    0x4    Magic (KBCM|0x4B42434D)
0x0004    0x4    Section Size
0x0008    0x2    # Items
0x000A    0x2    Contstant (0xBEEF)
0x000C    0x4    Offset 1 (Index Table) - Relative to Section + 0x8
0x0010    0x4    Offset 2 (Data Block) - Relative to Section + 0x8
0x0014    0x8    Padding?
--------------------------------------------------------------------
Repeats * # Items (Index Table)
--------------------------------------------------------------------
0x0000    0x2    Count 1
0x0002    0x2    Count 2 (Count 1&2 always match?)
0x0004    0x4    Data Offset - Relative to Offset 2
====================================================================
Repeats * Count 1 or 2 ?? (Data Block)
====================================================================
0x0000    0x2    NANR Bank #?
0x0002    0x2    Signed?
0x0004    0x2    Signed?
0x0006    0x1    Always 32?
0x0007    0x1    NCER Bank #?
Am using tsutarja and zekrom as my basis for testing. Overall the file structure appears pretty straight forward except for the Count 1 and Count 2 values. Because there are two, it makes me think these represent X&Y or something similar, but the fact that these variables are separate and so far have been identical in every file I've looked at makes me wonder. The highest value of the NANR Bank # seems to match up with the number of NANR banks in the associated file and the NCER Bank # is does not seem to be higher than the total number of NCER banks in its associated file as well.
« Last Edit: September 27, 2010, 09:05:33 pm by Low Lines »

Zerox

  • Jr. Member
  • **
  • Posts: 16
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #149 on: September 28, 2010, 02:45:10 am »
Okay I realize this may have been a bit rude to even try but anyway Low Lines you mentioned that the first version of Console Tool used a heavy version of openGL and the newer one used a more lightweight version or something along those lines. So I downloaded the JDK and tried swapping .class files under formats from the new version to the old one. I wasn't really successful there. And then I tried swapping all 4 jogl related files from the old version to the new version in hopes to recreate the heavy openGL environment so I could use 3DVia Print Screen to capture the model from Console Tool (despite the fixing I had to do after doing such).

Your program is the only thing that's even been able to display KH models properly with any accuracy:



But this is only in the newer version of Console Tool. I was able to use 3D Via Printscreen on the previous version to capture models and textures but the mesh is terribly disfigured in the last version as well as the uv maps being totally destroyed. Though being able to capture the model even without UVmaps in a normal pose would help me a lot. Granted a real export feature would be loads better but I've already asked and you said I had to wait on that.

The real reason I asked about converting it to .exe was because I figured that maybe then I could capture the model easier. My only problem has been with your first version of Console Tool and it seems that it sucks all the memory out of my computer and drags it to a halt. Perhaps this is why you changed the way openGL was rendered.

Anyway I guess my question in all this is; If it's not a hard task could you possibly impliment proper format information reguarding DS models into the old Console Tool or perhaps make the new one use the same or similar openGL as the last one?

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #150 on: September 28, 2010, 06:01:07 am »
Let's see if I can explain it a little...Now some of this may be incorrect/poorly explained so if there's a smarter Java person, your welcome to clarify.

In Java there are two ways objects are rendered, heavyweight and lightweight. Heavyweight which is the older of the two basically draws things directly to the screen. This means each heavy weight object has its own "window" and I think the biggest problem you tend to face is when you want to layer heavyweight objects, because it doesn't work too well.

Lightweight was sort of like the solution to the layering issue, and in simple terms lightweight a drawn using a heavyweight as its base. The core GUI components are I think subclasses of the Window class which is a heavyweight object. The Window class is what is used to draw the GUI onto the screen. A lightweight object would be like a button or a textfield. They don't need their own window, and logically they sit inside one anyways (poor explanation I know, I'm sorry). I think another important point to make is that the updating (re-rendering) of lightweight objects is all handled by the heavy weight container.

But basically the old ctool drew the 3D directly to the screen, and if you noticed, nothing ever sat on top of the viewport, because it would get hidden behind it on refresh. And because it was a heavyweight object, and not to mention very poorly managed in terms of refreshing, it choked up memory since it was consistently refreshing the viewport by timed manual calls.

The new ctool uses a lightweight version which, basically removes the need to manually refresh it, cuts back on CPU dependencies and allows for other components to overlap it without needing any special coding needed to make it work. In general when you program in Java, the better use you make of the built in APIs, the better your program, aka you don't try to tame a lion by shoving a stick up its arse, because that will just make it angry :p

The reason why replacing the dlls didn't work is because the type of component the viewport is, is determined in the source code, ie new HeavyweightPanel, or new LightWeightPanel (not actual names but you should get what I mean), so it cannot be changed outside of the original java classes, since the package I give out is an already compiled version, same thing applies with C/++/# I think. Unless you actually implement support for render engine switching, you are basically stuck for options.

Essentially changing it between heavyweight and lightweight is just a matter of changing what the viewport is created as, as well as any references to it since Java is object orientated, but it will mean all the stuff that floats over the top may get overlapped, and a more complex refreshing system will be needed, and in general a bit more work involved to get it to display properly. Right now I want to keep away from mucking with the GUI code, since there are quite a lot of problems with how I've implemented and structured my classes which I'm now aware of since doing some Java at uni. I have to go through the whole thing and sort out the GUI better, which will take a lot of time, which I don't really have while I'm still studying. It's why I'm working on the Compact version so I can work on it in smaller modules and keep the more GUI intensive version for a major release later on once I sort out the problems I have with the code.

As for fixing up the older version with the fixes I know now, I actually don't have that complete version anymore, as back then I didn't have any sort of version control system for major changes.

Going back to the formats stuff, I've been going over the NANR file today and I've figured out a bit more of the file structure as well as having a few more assumptions. For one, I've been wondering where the rotation information for pokemon sprites are stored, because if you have opened up a set (NCLR, NCGR, NCER & NANR) you would notice that the animations have a bunch of frames but next to no movement. My assumption is that the NANR not only store NCER bank ids but also some form of transformations in the second chunk of data found in the file. From what I can tell, the data in the second block comes in three sizes determined by the value of the Data Type (4bytes, 8bytes & 16bytes). This is where I am assuming where the presumed transformation data is stored.

Added my updated notes on the NANR to my site, and below is a printout of Tsutarja's frontal NANR (hopefully its readable...). Just looking over the Frame Data, it looks like there are some signed values in there, though I'm kind of curious why this 0xBEEF constant keeps popping up, perhaps a stopper tag? I've seen it in various Nitro formats so I'm sure its not an actual value or anything :S

Tsutarja's NANR

I've got assignments to work on right now, but I'll have a go at analyzing the frame data later in the week...

Zerox

  • Jr. Member
  • **
  • Posts: 16
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #151 on: September 28, 2010, 04:09:29 pm »
Ah okay I understand a lot better now so thank you for taking some time to explain stuff to a newbie like myself. I won't bother trying to replace/swap anything anymore. I'll just wait patiently for a new version.

That's interesting about the Pokémon sprites and it sounds reasonable. Though I don't know anything that I could help with there so I'll just go back to waiting quietly.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #152 on: September 28, 2010, 05:53:19 pm »
Zerox you are consistent at the very least, I'm quite a fan of consistency (though I'm not encouraging you to actively nag lol).

Its not the model viewer but its a quick snapshot of where I'm up to...


I've added support for reading Adobe Color Tables (.act) and Windows Palettes (.pal), which will be the two main options for exporting palettes to native formats. If anyone has another format they particularly want that's either got some docs on it or is easy to implement let me know.

The GUI is currently missing Toolbar buttons, which will be mostly file management options such as exporting and creating a new file from scratch, but what I'm basically aiming for here is to have both read/write support done for each file type before moving on to the next. I already have a basic display going for NCGR and NCERs but once I'm done with palettes, I'll move onto them. Then its NANRs and then models okay? I'm not making any promises or setting any dates, but hopefully that sounds a little better than "waiting patiently" without a clear indication of when :p I'll also start releasing it in modular versions so people can actually make use of it (not that anyone cares much for palettes anyways XD) instead of waiting forever :s
As for exporting graphics, I want to take another crack at reading PNG files, but I'll just use the internal Java image exporting functionality for now since it's basically a one liner to implement. You lose the control I want over the structure of the file (particularly the way the palette is ordered) but it'll be way better than screen capturing Mario Kart DS sprites one at a time which I think one poor dude was doing :p

Also with compression, right now I've got it set up that when you open a compressed file, it will save a decompressed version with "_extract" added to the name into the same folder of the original file. If it proves to be an annoyance, I may just move that functionality to the toolbar, however it is handy if you have a handful of compressed files and you want to decompress them all at once...perhaps maybe some kind of batch option so then you don't have to close x number of windows afterwards :p
« Last Edit: November 17, 2010, 08:18:30 pm by Low Lines »

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #153 on: September 28, 2010, 08:23:17 pm »
Will it be able to export palettes to raw?

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #154 on: September 28, 2010, 11:11:35 pm »
Adobe Color Tables (act) are the same thing aren't they :p

Mauron

  • Submission Reviewer
  • Hero Member
  • *****
  • Posts: 551
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #155 on: September 29, 2010, 12:21:30 am »
Adobe Color Tables (act) are the same thing aren't they :p

If that's the case (I don't know one way or another), I think it would be worth having something in the program that mentions that. It would be a simple way to avoid confusion.
Mauron wuz here.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #156 on: September 29, 2010, 01:03:00 am »
The reason why I want to avoid using raw is because its like bin, it can hold just about anything and without a unique header/identifier it means that's one more thing the user has to figure out. Obviously there would have to be the option to open generic files like that, but since ACT's are a little more specific (and a less common extension), I thought it would be a better choice.

Will have to work on the help file too, thanks for reminding me :)

Zerox

  • Jr. Member
  • **
  • Posts: 16
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #157 on: October 08, 2010, 06:43:26 pm »
I was just using Console Tool to view BTX0 textures and apparently it's a container for many textures. But each has their own palette so I have to switch the texture and palette each time. My suggestion is that you have a check box option to be able to change both at once. And export function for at least textures would be helpful to me at least for getting to the KH stuff.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #158 on: October 08, 2010, 11:46:14 pm »
I was just using Console Tool to view BTX0 textures and apparently it's a container for many textures. But each has their own palette so I have to switch the texture and palette each time. My suggestion is that you have a check box option to be able to change both at once. And export function for at least textures would be helpful to me at least for getting to the KH stuff.
The only iffy bit with linking palettes to textures is that like texture at index 1 will not automatically be partnered to palette at index 1, as far as I know the pairing is done by a section in the nsbmd file it is used for. A toggle option sounds like a good idea though.

Been busy busy busy with uni assignments, have to complete a website redesign, prepare a power point presentation and finish the report by the end of next week, so my buddy and I are flat out getting the job done :p After that I only have 3 exam days and then I'll be able to get stuck into this project again :)

Zerox

  • Jr. Member
  • **
  • Posts: 16
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #159 on: October 13, 2010, 02:06:35 pm »
Ah okay. Well thanks to some help from Barubary I was able to export textures. So I guess there isn't quite as urgent a need for an export feature in your program for textures yet I guess.