News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The Console Tool (by Low Lines)  (Read 252672 times)

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #200 on: November 17, 2010, 12:05:42 pm »
Were you able to open those compressed overlay files? It turns out CrystalTile can actually decompress those just fine.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #201 on: November 17, 2010, 08:03:51 pm »
@Rhys
...What DS hacking tools?? I was under the impression there was a general lack of such things, and of what there is, it's generally only focusing on one thing...Plus most of the tools I know of were like one-offs that haven't been updated in a while.

@Koetsu
I'm focusing on the documented formats atm.

Cleaned up the Assets code a heck of a lot, using lots of convenience methods to cut back on repeated lines, and have also enabled the formats to be opened without requiring all their assets to display. Still trying to figure out the NCER cells. I have gone over the notes Garyong posted about 5 pages back and added mirroring and palette indexing. Am having troubles extracting sub image coordinates from the tileOffset on some cells. It seems that any offsets in the 128+, 128+ location of the image use a different format...otherwise I've been getting the x and y from the first 10 bits (5 bits each) and the multiplying that by 8.


Cell 2 has a value of 0x1F1 and is located at 128,128
Cell 3 has a value of 0x219 and is located at 192,128

This is a sprite from Super Princess Peach in case anyone was wondering.

Here are my notes on specifically the cell data structure. I've written them in the same format as Garyong's so its easier to compare.

Code: [Select]
s8 ypos
u8 flags1
s8 xpos
u8 flags2
u16 offsets

*flags1 WW????FF
W = dimension index 1
F = vertical flag?

*flags2 HHYX??FF
H = dimension index 2
F = horizontal flag
**if xpos <=-64 and F=0 add 256 to xpos
**if xpos >=64 and F=1 substract 256 from xpos

*offsets PPPPPTTTTTTTTTTT
P = palette offset
T = tile offset / sub image coordinates

Rhys

  • Hero Member
  • *****
  • Posts: 706
    • View Profile
    • CN
Re: The Console Tool (by Low Lines)
« Reply #202 on: November 17, 2010, 08:29:55 pm »
I remember there being a couple of tools here and there to extract sound archives and nitro archives, and one application that let you browse a DS rom and explore archives and filetypes and replace files, afaik this is what zero day hackers used for the DS Pokemon games. There's some Pokemon-specific tools too.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #203 on: November 19, 2010, 10:58:10 pm »
Been a bit busy with other things so I've been off and on working on this, but never fear :)


And yes the playback buttons work and don't crash like the old version. It's the first thing in the program to make use of a Thread class which although I understand, I really haven't been shown properly how to use yet. From what I understand it's a core aspect of one of the next units I'll be doing next semester at uni.

A lot of focus was actually on improving the assets management which still had a few bugs that needed ironing out. As far as I can tell, it works without issues now, at least for these four data types I've implemented into the program.

Also going back to the Animation Viewer, I've also included a copy function which allows you to copy the image the clipboard so you can paste it in Paint or whatever. The only downside is you lose transparency as the image is exported to RGB and as far as I know that's a java thing and not something I can change much. At the moment you can copy an image by right-clicking on the image and selecting copy, but I will eventually include a toolbar option as well as implement it on the other graphical data types too :p Until I add support for web images (PNG, GIF, etc), that's as far as image exporting will go...but at least that ought to be better than screenshots :p

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #204 on: November 19, 2010, 11:26:18 pm »
Ooh. I like the Photoshop-style animation strip. How are the timers calculated? Does it involve the 60fps and the value specified? And it looks like the palette of the sheet can be directly edited. Is this also possible?

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #205 on: November 19, 2010, 11:35:16 pm »
Ooh. I like the Photoshop-style animation strip. How are the timers calculated? Does it involve the 60fps and the value specified? And it looks like the palette of the sheet can be directly edited. Is this also possible?
Each data type is treated as a separate thing or file in this case so you can easily make changes and see the difference since the program will reload themselves with any changes applied to assets. Palette editing is the only fully editable thing at the moment with everything else just being in viewer mode.

The timers at the moment go on the assumption that the timer value stored in NANRs are n/60 fps (which is the number frames the DS updates per second), however after looking through a few files, its probably likely there's a second variable that tells you the divider amount as well.

On the Adobe-styling, I actually got a bit peeved when I tried dragging a panel to the top or bottom of the screen in Photoshop and discovered I couldn't do it, it seems docking on all four sides is only utilized in Flash, but still...was a bit of a shock you know :p

Rhys

  • Hero Member
  • *****
  • Posts: 706
    • View Profile
    • CN
Re: The Console Tool (by Low Lines)
« Reply #206 on: November 20, 2010, 06:10:21 am »
Wow that UI looks incredible, well done! :thumbsup:

henke37

  • Hero Member
  • *****
  • Posts: 643
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #207 on: November 21, 2010, 07:16:58 am »
Well done on the GUI indeed.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #208 on: November 21, 2010, 11:41:42 am »
Had a go at adding support for NCERs with a Bank Type 1 (aka Extended/Partitioned Cells in Garyong's terms), with success it seems. There were some issues getting the right offset addresses with cells however I'm pretty certain they're fixed now. Also as of yet, I've only come across one sprite sheet (NCLR+NCGR+NCER+NANR) with an 8 bit pixel depth, so it's probably likely there will be bugs with some sprites as I've only taken it into account for graphics that are partitioned. Digimon Championship is the game I've been testing when adding support for extended cells.


One thing I noticed however (not mentioned in Garyong's notes, unless I missed it) is that the partition section of the NCER seems to have an 8 byte header that makes up two 4 byte values. The first value looks like it is possibly the maximum size of a partition, whereas the second value always seems to be 8. There also seems to be some irregular use of 0xCCCC in both NCER and NANRs as some kind of break/padder, what its significance is I would have no clue.

Rhys

  • Hero Member
  • *****
  • Posts: 706
    • View Profile
    • CN
Re: The Console Tool (by Low Lines)
« Reply #209 on: November 21, 2010, 03:45:15 pm »
It's quite interesting that the DS has standard file type for pretty much everything a DS Developer could want, it's something that competitor platforms haven't really done too much outside of basic naming conventions for executables and a specific directory structure (the PSP/PS3 comes to mind).

It makes emulating these platforms on future platforms for backwards compatibility quite interesting, as the hypervisor/emulator could analyse the game's code for library calls for accessing data from these formats and swap it out for code much more optimised for the new platform, or even for the case of the ARM7 binaries replace it altogether for games that ship with nintendo's binary (afaik all retail games still do this).

It also makes translating DS games much easier as if a game uses a known format that can be un/repacked then translating the game should be fairly straightforward with little to no ASM hacking required. It'll be interesting to see if other platforms adopt this approach in the future.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #210 on: November 21, 2010, 09:45:29 pm »
Going a little off-track from my milestones, I've been looking at the NMCR and NMAR files found in newer games, such as those in the new Pokemon DS games.


Now this above picture may look like crap, but I think, that if I had the cell gap error fixed, it would be displaying properly. After getting this far, I'm almost certain NMCRs are like a second cell mapping file that positions each animated limb (bank in a NANR file) with some offsetting added to the positioning. In the case of the Pokemon sprites there are two cell banks, which make up the two frames in the NMAR animation. As far as I can tell, there's no rotation information stored in any of these files, which leaves me to assume that its kept in next file following the NMAR.

There's also not enough data in the NMCR to allow for fixing the cell gap, so the real question is, if the cell gap is dependent on the NCER, how is it meant to be fixed XD

Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #211 on: November 22, 2010, 04:49:00 am »
The rotation data is stored in the NANR file, in the Frame Data field. I'm assuming you've already figured out some of the Frame Data format, since you've shown an animation loaded into the tool, but these are my notes on NANR so far (ignoring the generic file header and LABL and UEXT sections);
Code: [Select]
/*
 * ABNK; Animation Bank
 * <NitroSection("KNBA", sectionSize)>
 * u16 numAnimations;
 * u16 numFrames; // total number of frames. not really necessary, since they have various sizes and can only be read once the format is known from the animation.
 * u32 animationOffset; // relative to section start + 0x8
 * u32 frameRefsOffset; // relative to section start + 0x8
 * u32 frameDataOffset; // relative to section start + 0x8
 * u32 padding; // is it?
 * u32 padding; // is it?
 * Animation* animations;
 * u8* padding;
 * FrameRef* frameRefs;
 * u8* padding;
 * FrameData* frameData;
 */
/*
 * Animation:
 * u32 numFrames;
 * u16 dataType; // Type of FrameData used for this animation. 0: 2 (4) bytes. 1: 16 bytes. 2: 8 bytes
 * u16 unknownConst; // always 1 it seems
 * u32 unknownConst; // always 2 it seems
 * u32 frameRefOffset; // offset of first frameRef, relative to start of frameRefs.
 */
/*
 * FrameRef: (frame reference/pointer)
 * u32 frameDataOffset; // relative to start of frameData.
 * u16 delay; // duration in #frames. ~= (1/60)s?
 * u16 beef; // always 0xBEEF?
 */
/*
 * FrameData:
 * - Type 0:
 * u16 cellIdx; // cell of the NCER to display
 * u16 garbage; // if this is the last frame data for an animation, this is 0xCCCC. Otherwise this is the first u16 of the next FrameData. Just ignore it.
 * -------
 * - Type 1:
 * u16 cellIdx;
 * v16[5] transform;
 * s16 xDisp; // displacement of cell in x-direction
 * s16 yDisp; // displacement of cell in y-direction
 *
 * The transform is usually of this format:
 * {B, A, C, D, E}, with:
 * E = 0
 * A,B,C,D as used in the formula in the bottom of this section in GBATEK: http://nocash.emubase.de/gbatek.htm#lcdiobgrotationscaling
 *
 * However C and E can be 0xFFFF sometimes (pokemon B/W file 505, pikachu), which seem to be some special marker. Not sure yet what they mean.
 * -------
 * - Type 2:
 * u16 cellIdx;
 * u16 beef; // always 0xBEEF
 * s16 xDisp;
 * s16 yDisp;
 */
(the terminology is most likely different again)
A v16 is a 1.3.12 fixed-point number, as defined here.

MegamanX

  • Full Member
  • ***
  • Posts: 210
    • View Profile
    • X Translations
Re: The Console Tool (by Low Lines)
« Reply #212 on: November 22, 2010, 05:05:16 am »
Will this support .ob, .pb, .ab, .nbfc, .nbfs, .nbfp formats?

Curious because the game I'm looking at right now uses those formats.

Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #213 on: November 22, 2010, 06:25:43 am »
Will this support .ob, .pb, .ab, .nbfc, .nbfs, .nbfp formats?

Curious because the game I'm looking at right now uses those formats.
Extensions are only weak indicators of the content of files. I know the .nbf* formats should be just raw graphics, tile maps and palettes, which could be supported. I think it's unlikely the others will be supported however, unless you can provide example files (or mention the game you're looking at) ;].


By the way, I've unfortunately not yet had the opportunity to test the GBATEK formula for 2D tranformation. However the expected results of applying the formula to the values in the pokemon files match the actual animation too closely for it not to be a coincidence.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #214 on: November 22, 2010, 09:21:32 am »
Yeah it's a colossal amount of data that needs to be processed to get the end result, I mean the complete animation of one sprite is made up of 6 files at least.

I actually had the NANR bank types in my implementation (excluding knowing what the extra data did), however I must have thought it was wrong and removed it. Silly me :p

At the moment I've been trying to figure out the cell gap problem, and my current assumption is that if the vflag == 3 an offset is added based on the x & y coordinates, much like how with the Digimon World sprites (which I think are vflag == 2) add/subtract 256 if the x coordinate is <=-64 or >=64. I can get sprites to display within about an 80% accuracy but I'm doing it the long way so it's sort of trial and error atm. The key thing that helps support my assumption is that when you load up everything up to the NMCR file, the body parts line up better with the feet generally just above to horizontal center point (red line in my gui).

@MegamanX
I'll add support for any formats with a clear indication of what they do (ie structured header, documented, etc), but my priority is getting the more generic ones supported first, specifically right now I want to have all the Nitro and Nitro3D formats supported. At the moment, it can open NCLR (as well as the other stated palettes), NCGR, NCER, NANR, NSCR, NMCR and NMAR with varying levels of functionality, with the most supported being the first 5. As far as my current milestones go, I've only got to add support for NARCs and probably DS Roms, plus finish up the drag and drop ui, before I'm going to be happy releasing a build so people can start testing it :p  Though as I've said I've kind of sidetracked at the moment trying to fix errors in the NCER and NANR, depending on how they go I'll get stuck into the rest later.

[edit]

Not having much luck with the cell gap bug. There is almost always two conflicting cells that if you fix one, it breaks the other, so perhaps there could be another variable involved, such as the u16 value in the bank header? The issue though with that is that other NCERs have values stored here also.
« Last Edit: November 22, 2010, 05:57:15 pm by Low Lines »

Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #215 on: November 22, 2010, 07:14:16 pm »
I'm quite positive the 'cell gap' (=the gap between OAMs in a Cell) is inherent in the NCER files, and should thus somehow be mended in the NCMR/NMAR files (I'm guessing with position scaling). Using more GBATEK specifications, I believe I've nearly got the complete NCER format (there's only two unknown values left);
Code: [Select]
/* CEBK section:
 *
 * <NitroSection("KBEC", sectionSize)>
 * u16 numCells;             // number of cells defined in this CEBK
 * u16 extendedCells;        // if 1, cell data is CellDataEX instead of CellData
 * u32 cellDataOffset;       // relative to section start + 8
 * u32 flags;                // 2 LSBs: left-shift all tile indices this much.
 *                           // 3rd LSB: if 1, use `sub-images' instead of normal
 *                           //           addressing mode
 * u32 partitionDataOffset;  // offset to partitioning data header.
 *                           // partitioning data is only present if this is non-0
 * u32 padding;
 * u32 padding;
 * CellData[numCells] cellData; // is CellDataEX[numCells] if extendedCells == 1
 * // no padding in-between; cell data is properly aligned
 * OAMData* oamData;
 * u8* padding;              // may be empty
 * // --- partition data header starts here. is not present if partitionDataOffset==0
 * u32 maxPartitionSize;     // not sure; could be something else.
 * u32 headerSize;           // partition header size, offset of first partition
 *                           // relative to this section, or just constant 8
 * PartitionData[numCells] partitions;
 */

/* CellData:
 *
 * u16 numOAMs;
 * u16 unknown;
 * u32 firstOAMOffset; // relative to first OAMData
 */
/* CellDataEX:
 *
 * <CellData>
 * s16 xMax;
 * s16 yMax;
 * s16 xMin;
 * s16 yMin;
 */

/* OAMData:
 *
 * u16 OBJAttribute0;
 * u16 OBJAttribute1;
 * u16 OBJAttribute2;
 * (see also GBATEK, http://nocash.emubase.de/gbatek.htm#lcdobjoamattributes)
 *
 * Attribute0: ABCDEFGH IJKLMNOP
 * IJKLMNOP : Y position (signed, range [-128,127])
 *        H : R/S flag (Rotation/Scaling flag)
 *   if R/S flag = 0:
 *        G : OBJ Disable. (if 1, do not display) ignore.
 *   else
 *        G : Double-Size flag. (if 1, double-sized display reactangle) ignore, unless some
 *                developer purposefully used a larger/smaller clipping rectangle for displaying
 *                the rotated/scaled sprites properly for some obscure reason.
 *   endif
 *       EF : OBJ mode. 00 -> normal.
 *                      01 -> semi-transparent.
 *                      10 -> 'OBJ Window' (some sort of masking; see GBATEK).
 *                      11 -> Invalid.
 *                      Should be 00 for NCER values.
 *        D : Mosaic flag. Should not be set in NCER.
 *        C : Color Depth. 0 -> 16 colors, 1 -> 256 colors. (can technically be ignored, since the depth is also known from NCGR/NCLR file)
 *       AB : OBJ Shape. 00 -> square
 *                       01 -> horizontal
 *                       10 -> vertical
 *                       11 -> Invalid.
 *                       
 * Attribute1: ABCDEFGH IJKLMNOP
 * HIJKLMNOP : X position (signed, range [-256, 255]. read as unsigned, subtract 0x0200 if X>=0x0100)
 *   if R/S flag = 0:
 *       EFG : unused.
 *         D : Horizontal flip.
 *         C : Vertical flip.
 *   else:
 *     CDEFG : R/S Parameter selection. Should not be set in NCER.
 *   endif
 *        AB : OBJ Size. (resulting OAM size depends on this and OBJ Shape from Attr0)
 *       
 * Attribute2: ABCDEFGH IJKLMNOP
 * GHIJKLMNOP : Tile index
 *         EF : Priority relative to BG (usually not set in NCER, but sometimes is)
 *       ABCD : Palette index
 *       
 *
 * OAM size table:
 *
 * Size  Square   Horizontal  Vertical
 * 0     8x8      16x8        8x16
 * 1     16x16    32x8        8x32
 * 2     32x32    32x16       16x32
 * 3     64x64    64x32       32x64
 *
 */

/* PartitionData:
 *
 * u32 start;   // (in pixels)
 * u32 length;  // (in pixels)
 */
I find it unlikely (although not impossible) the unknown value in the CellData can fix the gap, since values at 'broken' cells also appear for non-'broken' cells. (was this the value you mentioned? I'm interpreting 'bank header' as the CEBK section header however, since the CEBK is a singular Cell Bank / Bank of Cells ;])
The other unknown value has to do with the partitions, so that's certainly not the fix-it-all value.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #216 on: November 22, 2010, 08:28:43 pm »
My terminology sucks, in fact I'm going to go and fix all my naming right now. In MY terms, a cell is a bank and an OAM is a cell..Whenever I first started my notes, that's how I put it down and that's how it's kind of stuck >_<

You are clearly much better at figuring this stuff out than me, and it didn't even occur to me to look at GBATek :p

Just out of curiosity have you built a viewer along with your specs?

Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #217 on: November 23, 2010, 08:58:57 am »
My terminology sucks, in fact I'm going to go and fix all my naming right now. In MY terms, a cell is a bank and an OAM is a cell..Whenever I first started my notes, that's how I put it down and that's how it's kind of stuck >_<
Heh, I know the feeling. I probably still have some mis-named comments somewhere in my code as well; rarely anybody get them right the first time. =)

Quote
You are clearly much better at figuring this stuff out than me, and it didn't even occur to me to look at GBATek :p
I probably only have more experience at these things ;]
I started looking at DSTEK/GBATEK for the capabilities the DS/GBA have for rotating sprites, when I also noticed these very familiar format. Also, since these Nitro formats are built-in, it's more likely the Nitro formats use data structures the DS/GBA need in their memory anyway.

Quote
Just out of curiosity have you built a viewer along with your specs?
Yes, although it only displays NCER+NCGR+NCLR or NSCR+NCGR+NCLR triplets. It's a Windows-only viewer, partly because I dislike designing Java UIs, partly because I like the binary data handling of C# better.
I've not yet made it open-source, because I wanted it to support more formats before I did (and currently only the recently re-written NCER class is not mostly ugly code). The included Glycerin.dll should be accessible by other .NET apps though, which contains the file format classes.

Lexou

  • Jr. Member
  • **
  • Posts: 9
  • a mantaray
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #218 on: November 23, 2010, 12:34:44 pm »
Well, sir, Low Lines, I must say your app is the best game explorer I've seen. Especially thanks to the cross-platform-ness.

Now, my only problem is that whenever I open this and start saerching, even just in the /browser/, the task uses up 98% of my dual processors even though my RAM is only at 20% (like it is normally).
I'm not sure if my computer is acting all shitty, or your app is too big, but it'd be great to have this working without bugs/ huge lag.

Also, does the extract button actually work or was the app lagging too much for the dialog to even OPEN.

Lastly, .nsbca type animations don't seem to load. Again, I'm not sure if it's computer doing this or whatever.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #219 on: November 23, 2010, 12:36:39 pm »
Which version are you referring to?