News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The Console Tool (by Low Lines)  (Read 252381 times)

Lexou

  • Jr. Member
  • **
  • Posts: 9
  • a mantaray
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #220 on: November 23, 2010, 12:42:44 pm »
Which version are you referring to?

Well, the demo 1 and 3 .zip files from your sig apparently couldn't be opened by 7-zip, so I guess I'm using Demo #2.

BTW, kudos on the fast reply.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #221 on: November 23, 2010, 01:05:05 pm »
Demo 1&2 were designed with the file browser at the core, so whenever you start the program, it first scans your system to get the drives and the contents of the top level folder in each. Also due to the way I programmed it, each file reference that is created, also is scanned for file type matchups (as in reading the actual header for magics, compression, etc), which can also create laggyness. The only reason for a very high cpu usage like that is some kind of routine looping constantly and as far as I remember, only the older (demo 1) model viewer does that.

On my system, I think my hard drive is badly fragmented or something, because it takes a colossal amount of time to load up the browser now (which it didn't originally), though I also have networked drives including ones connected to very slow computers. These are all issues I want to address when I eventually redo the file browser, which in the current version, just uses the default browser dialog created in java (looks quite nasty and doesn't blend well with system look and feels). Threading is one thing I have not used at all when file browsing, which would probably dramatically resolve alot of the lagging issues.

As for the NSBCA, I got up to the point where I could fully understand the file structure however I had no way to assign assets in the older versions. Its on my list of things to do though.

All demo zips are working, perhaps they didn't download properly for you?

Lexou

  • Jr. Member
  • **
  • Posts: 9
  • a mantaray
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #222 on: November 23, 2010, 01:41:42 pm »
Well, I redownloaded Demo 3, and apparently, 3rd time's a charm.

This one doesn't take up all my CPU, but it doesn't work. I've tried dragging and dropping, opening with the app... It always seems to just stop working afterwards.
I have the latest java platform, don't worry about that.

I'll try and redownload all of them and try each, see if the CPU usage was just a download problem.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #223 on: November 23, 2010, 01:45:46 pm »
Demo 3 only handles a a few of the 2D Nitro formats (was an experiment to help improve my class design), otherwise it just closes. Redownloading also won't fix any CPU issues.

Lexou

  • Jr. Member
  • **
  • Posts: 9
  • a mantaray
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #224 on: November 23, 2010, 01:55:01 pm »
Surprisingly enough, it did !

The interface has a different look, and it's a whole lot better. Actually I think I was using Demo 1, hence all the problems.

I have two things to point out though, when viewing a model, I can't seem to zoom, is there a control for it ? Secondly, the "Save as..." option doesn't work. Is it because it hasn't been implemented yet ?
See, I could extract the models with a printscreen ripper, but I don't want to lose bone/animation info.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #225 on: November 23, 2010, 02:10:37 pm »
Exporting hasn't been implemented, and to be honest, just getting the damn things to display properly is an achievement in itself :p

3D controls are done with a three button mouse...
left - pivot/rotate
right - move center point (up/down)
middle (hold and swipe left/right) - scale

From what I've been told, the ram ripping programs don't work on the demo 2 model viewer due to the way it renders stuff.

I swear though...the moment my app is able to export DS models, I'm going to be in for a world of hurt @_@

Lexou

  • Jr. Member
  • **
  • Posts: 9
  • a mantaray
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #226 on: November 23, 2010, 03:34:55 pm »
What do you mean ? From the moment it's able to export, then it should be smooth sailing, am I wrong ?

BTW, how do I make it render textures ? It's just a white mesh, in it's current state.

Also, maybe I didn't say this enough, but bear in mind that your game exploring app is the absolute best, and no other comes close to rivaling with it.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #227 on: November 23, 2010, 03:56:18 pm »
Wells with more support, comes more users, and from what I can tell, hacking 3D models is a big want atm, so this thread will inevitably get alot of the same questions asked over and over, either because people don't bother to read (which is understandable with 16+ pages), or it simply just isn't clearly answered anywhere :p I'll have to write up an FAQ too for the next version :p

With the textures, if they aren't part of the NSBMD file, they won't load. This is because I tried implementing the assets manager but then gave up (and then had a several month break from the project before releasing demo 2 without changes), otherwise it should have reverted to the last opened texture by default.

I'm sorry if I came across a bit iffy, I'm all for answering questions. The way I see it, is if people have to ask questions, that means I'm not doing my job as the designer, but also, all I want to do right now is get everything I've achieved to this point all working in the same build (which is not riddled with the leftover garbage of all previous versions), so I can remove the demos and so there isn't all this confusion as to what is and isn't working anymore. I am getting there slowly though :)

Lexou

  • Jr. Member
  • **
  • Posts: 9
  • a mantaray
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #228 on: November 23, 2010, 04:13:47 pm »
Yeah, I'm sorry if I posted already answered questions, but as you said, the thread is pretty long, and each of your posts are (very interesting nonetheless but still) walls of text.


You don't come across iffy at all, actually. I am truly grateful that you answer all (quite a bunch, looking back) of my questions, and with such speed, too.

I completely respect what you do as a software developer, and, unlike most fans, I can actually be patient from time to time.

I'll stop assaulting you with questions, now, cause that's one of (if not the) best way to deconcentrate the person who's making the software.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #229 on: November 24, 2010, 12:35:06 am »
Filb appears to have links to large number of the animated pokemon sprites (forgive me if this isn't news...).


I wonder if they ripped them manually, or if someone out there actually has a complete spec :p
After doing some random pokemon ids and getting 404s I'm guessing that these were ripped manually otherwise it makes no sense why they aren't all uploaded to the site.

@Status
Currently working on the archive UI. NARC and DS Roms are supported and load super quick since file format checking can be skipped until being called by the render component. Have had to tweak the FileAccess class a bit as for some odd reason some files return an access denied error when opened using the RandomAccess method, possibly because said files had uncommon characters (such as &) in the filenames.

@Garyong
Your app doesn't even load for me :'( Interesting name for it though :p

Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #230 on: November 24, 2010, 05:02:56 am »
Filb appears to have links to large number of the animated pokemon sprites (forgive me if this isn't news...).


I wonder if they ripped them manually, or if someone out there actually has a complete spec :p
After doing some random pokemon ids and getting 404s I'm guessing that these were ripped manually otherwise it makes no sense why they aren't all uploaded to the site.
I find it quite unlikely they figured out the format. They probably took them from this topic (or a page with all the GIFs from that topic), or some other source of manually ripped animations.

Quote
@Garyong
Your app doesn't even load for me :'( Interesting name for it though :p
Whoops, packed the wrong .exe  :banghead:. Use this instead, although it doesn't contain the .XML with the documentation for the .dll (which you can copy from the other link if you want it).

Lexou

  • Jr. Member
  • **
  • Posts: 9
  • a mantaray
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #231 on: November 24, 2010, 06:02:54 am »
I find it quite unlikely they figured out the format. They probably took them from this topic

Funny you should mention tSR, that's where I plan on submitting model found through Low lines' app ! I am currently ripping levels from the wind waker games, as you can see here.

BTW, are you Barubary ?, it said "by Barubary" in the glycerin map viewer. If so, then hello, fellow tSRian.

Tauwasser

  • Hero Member
  • *****
  • Posts: 1392
  • Fantabulous!!
    • View Profile
    • My blog
Re: The Console Tool (by Low Lines)
« Reply #232 on: November 24, 2010, 09:18:56 am »
I find it quite unlikely they figured out the format. They probably took them from this topic (or a page with all the GIFs from that topic), or some other source of manually ripped animations.

Indeed loadingNOW figures the format out and had to crack the key they are encrypted with, or so I heard. http://pokeguide.filb.de/

cYa,

Tauwasser

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #233 on: November 24, 2010, 11:57:04 pm »
I don't believe Black & White have very many files encrypted with encryption used in the other DS Pokemon Games :p Still I think people are crazy if they're trying to rip all those animated sprites manually...I mean yeah back in the days, 151 pokemon is little work, but 649...that would take forever!!

I had a brief heart-attack yesterday as I had been backing up my files before I went to sleep and so I immediately wiped all the stuff off my hard drive (had about 250,000+ files and 8000+ folders on it) when I woke up and then realized I had also deleted the folder with my program in it. So I copy it back across from my backup disk and reload it up in Netbeans and for a while I had thought it hadn't updated all the class files I had made changes to since the last backup I did just over a week ago. There's nothing worse than getting heaps done in a short amount of time and losing it in an instant. So I'm there scanning through a Deleted File Recovery app, trying to pick out all the changed files one by one, when I then realize that the files I had changed WERE updated, just some of them I had moved into different folders in the project manager and so the old ones were still there from the previous backup...Stupid me >_<

The core parts of the Archive UI are done, the code behind it completely rewritten (and much much less code) though I quite liked the old design so I've pretty much kept it looking the same. Still have to add file extracting functionality but other than that, it does the bare minimum job for now. I want to maybe add support for different viewing modes such as like thumbnails and file filtering etc, but it's not my priority right now as I would like to move on to finishing off the drag and drop (plus fill in the menu bar with options) and then *finally* get stuck into the 3D :crazy:



Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #234 on: November 25, 2010, 06:00:55 am »
Still I think people are crazy if they're trying to rip all those animated sprites manually...I mean yeah back in the days, 151 pokemon is little work, but 649...that would take forever!!
You're underestimating the power of fanboyism. ;P

Quote
I had a brief heart-attack yesterday as I had been backing up my files before I went to sleep and so I immediately wiped all the stuff off my hard drive (had about 250,000+ files and 8000+ folders on it) when I woke up and then realized I had also deleted the folder with my program in it. So I copy it back across from my backup disk and reload it up in Netbeans and for a while I had thought it hadn't updated all the class files I had made changes to since the last backup I did just over a week ago. There's nothing worse than getting heaps done in a short amount of time and losing it in an instant. So I'm there scanning through a Deleted File Recovery app, trying to pick out all the changed files one by one, when I then realize that the files I had changed WERE updated, just some of them I had moved into different folders in the project manager and so the old ones were still there from the previous backup...Stupid me >_<
May I suggest some versioning system like Subversion? It'll prevent such (near-)mistakes, helps you keep track of what you've done when (if you get into the habit of writing 'proper' commit messages), and allows you to easily go back to any previous version of the tool or single files.
Assembla has a free private Subversion (or git, if you prefer that over SVN) repository package available. (I'm assuming 2GB is ample space for the tool's source)


@Lexou: I am indeed =)


[EDIT]

You've probably figured most of this out already, after you've posted your preliminary specs on page 10, but there are my notes on the NMCR anyway (or rather its MCBK section);
Code: [Select]
/* MCBK format:
 * <NitroSection("KBCM", sectionSize)>
 * u16 numMappings;
 * u16 beef;             // constant 0xBEEF
 * u32 mapHeadersOffset; // relative to section start + 8
 * u32 mapDataOffset;    // relative to section start + 8
 * u32 padding;
 * u32 padding;
 * MapHeader[numMappings] maps;
 * u8* padding;          // probably empty, since sizeof(MapHeader)=8
 * MapData* data;
 */

/* MapHeader format:
 *
 * u16 numParts;        // Number of MapData's used
 * u16 numAnim;         // the number of unique NANR entries used
 * u32 firstPartOffset; // relative to mapDataOffset
 */

/* MapData format:
 * u16 animIndex; // index in NANR
 * s16 xpos;
 * s16 ypos;
 * u8 unknown;    // 'const' 0x20 or 0x21. some flags?
 * u8 priority;   // low = draw in front, high = draw in back. (I think, although it seems to coincide with
 *                // the order of the data anyway. Certainly not NCER index though ;])
 */

Posting the NMAR format would be silly, since it only has an ABNK and LABL section. The ABNK section is the same as the NANR ABNK section, but the first unknown constant in an Animation is 2 instead of 1.

This means the 'cell gap' must be fixed by the .dat, or I was wrong and the NCER/NANR actually do contain enough data to fix it. =/
« Last Edit: November 25, 2010, 01:58:08 pm by Garyong »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #235 on: November 25, 2010, 11:04:19 pm »
After you posted your findings on the nCER I had some of the same assumptions with the data (particularly the priority value), however I didn't see the point in posting another 'wall of text' with 2 changes which would honestly have been guessing on my part so :p It'd better having two perspectives on something anyways before coming to a conclusion.

Does anyone know of any other games that have the NMCR/NMAR files in them? That'd be the best way to rule out if the cell gap is stored in the extra file or not. They have nitro structures so I can't see them as being custom files specifically for the pokemon games, but in saying that, it would kind of be weird to have 6/7 files all nicely structured and one without as part of your sdk if you know what I mean. I can live an hope at least :p

I've had no previous experience with versioning, nor have I been taught how to properly do it, but I've created an account, uploaded a copy and got Tortoise SVN or something on my computer so, at the very least I should have a back up from here on :p

[edit]

Just looking through the a folder in black/white, the trainer back sprites (a\0\6\4) are also done using this 7 file thing, including the raw data file, so maybe I'm wrong :p They have the cell gap too btw.

[edit2]

After going through the whole a folder in the rom again, I think I might have found the odd one out :)
In a/1/7/0 the last 7 files in the archive make up the animated sprite of the female professor displayed in the game introduction. Yes it still has the unknown raw data file, but interestingly unlike all other animated sprites (basically every animated sprite used in battles) there doesn't seem to be any cell gap, or at least as far as my viewer can tell (it only displays the first frame when viewing NMCRs atm).


Also after looking at a few of the raw data files, this is of course just a guess, but perhaps they are a type of custom packed commands like how you can program in overworld events?
« Last Edit: November 26, 2010, 06:04:53 pm by Low Lines »

Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #236 on: November 28, 2010, 11:06:02 am »
 :banghead:

I do believe I have a solution to the 'cell gap' that is so obvious, it almost must be true.
It has to do with the 'Double-Size' flag in an OAM data, which indicates that the rendering rectangle should be double the normal size (which is the size of the OAM), since the OAM may be rotated/scaled. You may feel it coming, but the X and Y in each OAM are actually the X and Y position of its rendering rectangle. Since the rectangle's centre stays in the same location, the X and Y in the OAM are half the size of the OAM less than when the Double-Size flag would be 0.

In other words, for a quick fix, for each OAM add half its size to its location whenever the Double-Size flag is 1. (since you still need the centre of the OAM when rotating it, the rest should be the same code)
Some quick tests confirm the cell gap to be gone in the Cells; I'm currently working on my own NANR rendering, so I cannot be sure of anything else. (Of course, I could also have accidentally only tested files in which this particular fix worked, while others need something else to be fixed)
« Last Edit: November 28, 2010, 11:16:38 am by Garyong »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #237 on: November 28, 2010, 05:08:34 pm »
I knew it!!
I've been glaring at that flag for ages, trying stuff like dividing the x&y by two and getting the remainder etc and all sorts of dumb ideas with little luck. I also thought it was weird how your specs said it shouldn't be set and yet pretty much every animated sprite had it set so :crazy: Another conflict I've seen is generally the eye oam has its object mode set to semi-transparent according to your specs. I'm also seeing the R/S Flag set in some places too (typically most pokemon sprites, but some human sprites don't have it set for all oams).


I've been caught up in a the archive export gui, specifically an option dialog similar to how Windows Explorer works when you are copying/moving files, because I was getting sick of switching between versions to extract stuff :p At the moment you can't specify the save location (which is defaulted to the directory in which the source file is located), as I'd rather wait until I make my own "Save-as" dialog instead of using the built in one with lots of tweaks.


Garyong

  • Jr. Member
  • **
  • Posts: 15
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #238 on: November 28, 2010, 06:55:20 pm »
I knew it!!
I've been glaring at that flag for ages, trying stuff like dividing the x&y by two and getting the remainder etc and all sorts of dumb ideas with little luck. I also thought it was weird how your specs said it shouldn't be set and yet pretty much every animated sprite had it set so :crazy: Another conflict I've seen is generally the eye oam has its object mode set to semi-transparent according to your specs. I'm also seeing the R/S Flag set in some places too (typically most pokemon sprites, but some human sprites don't have it set for all oams).
I never said it should not be set, only that it could be ignored (since I believed it only affected the clipping rectangle) ;]
The R/S flag should coincide with OAMs that are rotated in the NANR animations (and should be set for the Double-Size bit to have that meaning), although they may just have set the flag for all OAMs

The 'Semi-Transparent' flag means that some sort of blending is applied with the background layer (not with other OAMs!). I suppose it could mean that the .dat contains a u16 for each semi-transparent OAM with the blending data. It may also be the unknown u16 in a CellData, but that would mean the blend factor would be equal for the complete cell (which wouldn't be very surprising).
For completion's sake, this would most likely be the format of the u16: (from GBATEK, obviously)
Code: [Select]
4000052h - BLDALPHA - Alpha Blending Coefficients (W)
Used for Color Special Effects Mode 1, and for Semi-Transparent OBJs.
  Bit   Expl.
  0-4   EVA Coefficient (1st Target) (0..16 = 0/16..16/16, 17..31=16/16)
  5-7   Not used
  8-12  EVB Coefficient (2nd Target) (0..16 = 0/16..16/16, 17..31=16/16)
  13-15 Not used

For this effect, the top-most non-transparent pixel must be selected as 1st Target, and the
next-lower non-transparent pixel must be selected as 2nd Target, if so - and only if so, then
color intensities of 1st and 2nd Target are mixed together by using the parameters in BLDALPHA register,
for each pixel each R, G, B intensities are calculated separately:

  I = MIN ( 31, I1st*EVA + I2nd*EVB )
'OBJs that are defined as 'Semi-Transparent' in OAM memory are always selected as 1st Target ..., and are always using Alpha Blending mode'


.... I suppose I should get used to calling the OAMs OBJs. (OAM = OBJ Attribute Memory, apparently  -.-)

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #239 on: November 28, 2010, 07:55:58 pm »
No OAMs are good, no need to over complicate things :p An object could mean just about anything (Object is the base class type for everything in Java for instance), but most people should know 'OAM' has at least something to do with sprites.

Why the eye OAM needs to do all that is beyond me, the ONLY thing I can think of it being possibly used for is when you have those battle effects where the body gets inverted colors or something along those lines. Like a pass-through on certain effects or something (or the reverse). Must be frustrating to have a nearly complete spec and not know the final little bits -_- Like those 3 missing jigsaw pieces that are missing from your puzzle preventing you from finishing it :p

[edit]

Time for a release 8)
I haven't got round to fixing all the bugs in the UI as I found there were actually quite a lot more issues than I thought. Some bits have been disabled to prevent any wacky effects, but by rights, there should not be an instance where the program will simply stop working *fingers crossed* There may be some lags when processing a large number of files as no threading is currently used, but compared to the older versions, it should pail in comparison.

Along with exporting stuff from archives, I've also added the multi-select functionality that was used in the older versions for loading assets. Every time you select a file of a specific type (Palette, Image, etc) it will remember it and anywhere where that file is displayed (atm only in the Archive viewer) it will have a little link icon next to the file to indicate this file is part of chained action. So for instance if you wanted to load an NCER, you'd select all its assets first (NCLR, NCGR) and then double click on the NCER and it will load all of them together. Once I make my own file browser, this functionality will be added but for now it's only available in the archive viewer.

I've also included an updated help file (*docx) in the zip folder with a start on explaining the new interface. I'm open for any suggestions to make the document more useful than what it probably currently is.

So again, a quick run down of what you can do with it:
  • UI drag and drop
  • Savable Layout settings (Automatic)
  • Hex Viewer
  • DS LZ77 and RLUncomp support (automatic extract was turned off)
  • Full palette editing support (NCLR, ACT/RAW, Microsoft PAL, Usenti PAL)
  • NCGR support
  • NCER support
  • NSCR support
  • NANR support (rotation isn't supported yet)
  • NARC and DS Rom support
  • Partial support for NMCR/NMAR
  • NCGR,NSCR, and individual NANR frames can be copied to the System clipboard by right-clicking on the image and selecting "Copy Image"

Hopefully it will work as smoothly for everyone as it does for me.

Download: Console Tool 3.0

[edit2]

Ack just found a bug -_- You can't view files in a compressed archive, it will just start reading from the start of the archive file. This happens because the caching was being overridden. Will be fixed in the next release...
« Last Edit: November 28, 2010, 11:54:04 pm by Low Lines »