Let's see if I can explain it a little...Now some of this may be incorrect/poorly explained so if there's a smarter Java person, your welcome to clarify.
In Java there are two ways objects are rendered, heavyweight and lightweight. Heavyweight which is the older of the two basically draws things directly to the screen. This means each heavy weight object has its own "window" and I think the biggest problem you tend to face is when you want to layer heavyweight objects, because it doesn't work too well.
Lightweight was sort of like the solution to the layering issue, and in simple terms lightweight a drawn using a heavyweight as its base. The core GUI components are I think subclasses of the Window class which is a heavyweight object. The Window class is what is used to draw the GUI onto the screen. A lightweight object would be like a button or a textfield. They don't need their own window, and logically they sit inside one anyways (poor explanation I know, I'm sorry). I think another important point to make is that the updating (re-rendering) of lightweight objects is all handled by the heavy weight container.
But basically the old ctool drew the 3D directly to the screen, and if you noticed, nothing ever sat on top of the viewport, because it would get hidden behind it on refresh. And because it was a heavyweight object, and not to mention very poorly managed in terms of refreshing, it choked up memory since it was consistently refreshing the viewport by timed manual calls.
The new ctool uses a lightweight version which, basically removes the need to manually refresh it, cuts back on CPU dependencies and allows for other components to overlap it without needing any special coding needed to make it work. In general when you program in Java, the better use you make of the built in APIs, the better your program, aka you don't try to tame a lion by shoving a stick up its arse, because that will just make it angry :p
The reason why replacing the dlls didn't work is because the type of component the viewport is, is determined in the source code, ie new HeavyweightPanel, or new LightWeightPanel (not actual names but you should get what I mean), so it cannot be changed outside of the original java classes, since the package I give out is an already compiled version, same thing applies with C/++/# I think. Unless you actually implement support for render engine switching, you are basically stuck for options.
Essentially changing it between heavyweight and lightweight is just a matter of changing what the viewport is created as, as well as any references to it since Java is object orientated, but it will mean all the stuff that floats over the top may get overlapped, and a more complex refreshing system will be needed, and in general a bit more work involved to get it to display properly. Right now I want to keep away from mucking with the GUI code, since there are quite a lot of problems with how I've implemented and structured my classes which I'm now aware of since doing some Java at uni. I have to go through the whole thing and sort out the GUI better, which will take a lot of time, which I don't really have while I'm still studying. It's why I'm working on the Compact version so I can work on it in smaller modules and keep the more GUI intensive version for a major release later on once I sort out the problems I have with the code.
As for fixing up the older version with the fixes I know now, I actually don't have that complete version anymore, as back then I didn't have any sort of version control system for major changes.
Going back to the formats stuff, I've been going over the NANR file today and I've figured out a bit more of the file structure as well as having a few more assumptions. For one, I've been wondering where the rotation information for pokemon sprites are stored, because if you have opened up a set (NCLR, NCGR, NCER & NANR) you would notice that the animations have a bunch of frames but next to no movement. My assumption is that the NANR not only store NCER bank ids but also some form of transformations in the second chunk of data found in the file. From what I can tell, the data in the second block comes in three sizes determined by the value of the Data Type (4bytes, 8bytes & 16bytes). This is where I am assuming where the presumed transformation data is stored.
Added my updated notes on the NANR
to my site, and below is a printout of Tsutarja's frontal NANR (hopefully its readable...). Just looking over the Frame Data, it looks like there are some signed values in there, though I'm kind of curious why this 0xBEEF constant keeps popping up, perhaps a stopper tag? I've seen it in various Nitro formats so I'm sure its not an actual value or anything :STsutarja's NANR
I've got assignments to work on right now, but I'll have a go at analyzing the frame data later in the week...