News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The Console Tool (by Low Lines)  (Read 254785 times)

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #180 on: November 07, 2010, 08:56:00 pm »
Its specifically for the NDS version. The first byte in the file should either be 0x10 or 0x11 iirc, and I also think I implemented it to open files with the LZ77 magic stamp as well (found in some NDS games ie Super Mario 64 DS).

The program will just close if it doesn't know how to open a file, but I was pretty sure the auto decompressor occurs before that point though so there should still be a newly created file in the folder for any valid LZ77 compressed files.

Actually now that I'm actually going back over the code, RLU compression should also be available too btw @_@ And also it definitely should decompress the file before closing...
You can always PM me the file and I'll have an look and see what's wrong :)
I sent you the files with this problem before. They were overlay files. Apparently not all of those files are compressed, but the ones that are compressed always have problems. I just tested them with DSDecmp and some decompress larger than they should be or not at all. I know for sure compressed overlay files begin after overlay 0671 and there are dozens of compressed files after that. ConsoleTool Mini couldn't decompress any more than DSDecmp either. And those files begin with 00 00. I don't know if they even are LZ77 compression.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #181 on: November 07, 2010, 09:15:06 pm »
Oh right, I forgot >_<

I'd say none of them are compressed as just scanning through the files there was also a RLU compressed file in there as well, you generally don't see a mixture of compression formats in batched files. It just happens to be that some of these overlays fit the header criteria of compressed files. The dead giveaway that they aren't is the massive arse gap between the compressed file and the decompressed file size. If it's anything over like a 10x increase or something, it's doubtful they were meant to be decompressed, 1.4KB > 4 MB is kind of pushing it I think :p

Without knowing the actual file structure of an overlay file though it's kind of hard to say anything for sure :p I have tried filtering the compression criteria a little but in doing so you might actually miss the odd file that actually IS compressed heavily so its a bit iffy :p
« Last Edit: November 07, 2010, 09:27:04 pm by Low Lines »

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #182 on: November 07, 2010, 09:53:44 pm »
Well I stopped getting decompressed files at overlay 0769. There should be more since these files are ports of files from the GBA game. Overlays 0971 past 0769 are the game's main sprite files which use a game-specific file format. I've already got that part covered and I know there could be more files. I'm sure the overlays could just contain any graphic or text for the game which I know is inconvenient. Maybe there is a pattern of sorts?

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #183 on: November 07, 2010, 10:08:33 pm »
Were these overlays pulled from an archive file? Easiest way to look for a pattern is to have them all in a single file and look at it using a combination of tile molester and a hex editor. Going through 1114 files that have no obvious file structure would be a nightmare. There also ought to be some kind of index file somewhere which might shed some light on your problem. What game are these from anyways?

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #184 on: November 07, 2010, 10:12:09 pm »
Yes, these did come from an archive. I got them from Rockman EXE Operate Shooting Star. When I looked through the game with a tile viewer, I figured out that all sprites were arbitrarily compressed since there are 1 or 2 types of sprites uncompressed for no reason.

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: The Console Tool (by Low Lines)
« Reply #185 on: November 07, 2010, 10:15:22 pm »
What exactly is in the "overlay" folder, anyway? I use NDSTool to extract DS ROMs into a directory structure and I always get two subfolders: "data" and "overlay". The "data" folder usually has the scripts and graphics, but "overlay" has a collection of BIN files of uncertain provenance. Are the files in the "overlay" folder usually graphics of some kind?
In the event of a firestorm, the salad bar will remain open.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #186 on: November 07, 2010, 10:56:49 pm »
@Koetsu
Sprites and text are typically the two file types most likely to get compression since they are typically the bigger files when compared to their counter parts.

@Ryusui
God it's been a while since I've used NDSTool...Okay now I'm on track with what Koetsu is trying to do. I'd assume "overlays" have a specific use much like how there's always a utility.bin file in the root folder of every game. Though I thought much of this stuff was all generic (built by the SDK) and all the game data was kept in the files located via the fat. If this isn't the case and there are custom files in other areas of the rom then that's annoying -_- Will have a look more into this when I go over the NDS rom format :p

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #187 on: November 07, 2010, 11:16:03 pm »
Yeah. In the previous game port, all the files were put into their own archive. This time, it seems the files in cmn and jp are probably screens and text while overlay is everything else like maps and sprites.

MegamanX

  • Full Member
  • ***
  • Posts: 210
    • View Profile
    • X Translations
Re: The Console Tool (by Low Lines)
« Reply #188 on: November 08, 2010, 02:13:14 am »
I don't want to take this that far off-topic, (if I am) but I unpacked a DS rom recently and the overlay folder is empty. 

I have a data folder with nothing but sub-folders with all the graphics from the game and then the empty overlay folder.  Aside from that, there's a few BIN files not in either folder (arm7, arm9, y7, y9, banner, and header).  That normal?  Haven't messed with other DS roms before so I don't have much clue yet.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #189 on: November 08, 2010, 02:26:09 am »
No I think that's normal, I would think that when a developer creates a new project, it automatically builds all the default settings/folders etc, but as each developer would have their own way of doing things, not all parts of it might get used, it'd be like having 4 USB slots on your computer but you only ever use 2 of them. As for the bin files, those are straight dumps of blocks of data from the header of an NDS rom, these hold stuff like the actual game code and information used for identifying your game ie when you load in a game into your DS it shows up with an icon and the name of the game.

MegamanX

  • Full Member
  • ***
  • Posts: 210
    • View Profile
    • X Translations
Re: The Console Tool (by Low Lines)
« Reply #190 on: November 08, 2010, 02:35:28 am »
Thanks for the quick reply.  Appreciate it.

I'm trying to learn a bit about DS roms here because there's only one game I give a hoot about.  As of right now, I can change all the graphics and repack it and play it.  Unfortunetly, it's slowly looking as if it won't be an easy task.

Thanks again.

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: The Console Tool (by Low Lines)
« Reply #191 on: November 08, 2010, 02:57:14 am »
You won't have to fiddle with the arm* files unless you plan on actually hacking the game's code. As for the directory structure, it can work with you or it can work against you. Right now, I'm working on a project where all the script files were neatly grouped in a single folder called...drumroll, please..."script". Mind you, almost all the other resources are in folders with bizarre abbreviations, so it'll take a bit more effort to find the pics that need modification. Adding complexity to matters is the fact that everything is wrapped up in proprietary (and compressed!) archive files. Fortunately, I've already written up a tool for cracking open those archive files. Recompressing them is another matter, but the neat thing is, I can reinsert files without compression if I so choose. Ryusui: 1, tricky programmers: 0.
In the event of a firestorm, the salad bar will remain open.

MegamanX

  • Full Member
  • ***
  • Posts: 210
    • View Profile
    • X Translations
Re: The Console Tool (by Low Lines)
« Reply #192 on: November 08, 2010, 03:20:14 am »
Good to know I can eliminate something from anything I want to do.

I WISH I had a script folder.  Just about everything has it's own sub-folder.  Armor, battle, bg, enemy, saiga, weapon, world, etc.  I found the font because there's a font folder.  That was the first thing I edited just to fool around and the edit appeared when I repacked and played.  Just wish there was a script folder.  Haha.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #193 on: November 12, 2010, 02:59:51 pm »
As it turns out, adding support for drag and drop rendered a transferhandler class I got off the net for handling drop and open (files off your desktop) useless so a bit of time was spent firstly trying to figure out how the whole DND system worked and as to how and where the file drop was located. A bit hectic but it's all working and I've found and fixed a few more bugs in there drag and drop interface :)

The next step was transferring the palette editor from the mini version, which is mostly just a copy and paste (hooray) but I've still got to build a few editor panels etc.
The main focus for most of this week has been the hex editor, which is kind of just a "viewer" at the moment. Displaying the file data is pretty straight forward but for the rest of the time I was playing around with the APIs to see what was achievable in terms of text highlighting etc. There's a built api for adding custom highlighting in which you specify a start and end position within the text as well as a painter, however as far as I know this only works if you want to apply different background colors for blocks of text. What I was also wanting to do is set different foreground (text colors) as well, but the best thing I've found so far is something called the AttributedString class, which allows you to set attributes to blocks of text (ie font, size, color, background, etc) however it can only be used when or if you are painting text onto the screen. Basically it means that you have to write your own custom paint class that firstly simulates all the default stuff of your normal textcomponent, plus applying the extra attributes. The rendered text also does not match the same smoothness of the text fields which ruled out a the idea of simply drawing over the text. I'm also having some issues filtering the text selection also, hence why atm it is just a hex viewer and not a hex editor. But other than that, the UI works pretty well, which is based on Hex Workshop if anyone has used it before. I'm going to put it aside and come back to it later once I've studied up on JTextComponents a little more :p


Oh and I also got swapping the order of tabs in the panels working...You'd be surprised at how satisfying it is to be able to drag the tabs around (like in photoshop) and it just reorders them for you :)

[edit]

Cleaned up the UI a bit as well as fixed a bunch more bugs I've found and have made a start on the palette/tile image editors. The major milestone for this point is that I've got the assets working, which manages linking files together (ignore the empty Assets panel, I forgot to remove it). It allows you to change say the selected palette for an image to any other palette that is currently opened, and will load the last opened palette if the selected palette has been closed. It should also allow for live updating too :)

« Last Edit: November 17, 2010, 08:14:52 pm by Low Lines »

Koetsu

  • Jr. Member
  • **
  • Posts: 45
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #194 on: November 16, 2010, 12:05:46 am »
Aren't those Pokemon sprites animated? Is it specified in the format?

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #195 on: November 16, 2010, 11:28:06 am »
@Koetsu
There are four different sprites for each Pokemon, two for the front and two for the back, one being the animated sprite while the other is solid single cell sprite like how they were done in older DS Pokemon games.

@Update
The palette editor has been added with improvements on the version from the Mini app I posted a few weeks ago, namely the ability to drag and drop colors effortlessly (which took MANY hours to get working right), as well as input values either using the RGB textfields or as a hexidecimal value. When you switch between tabs, you can instantly see the changes made in the graphics, which is great up until the point where you are switching tabs constantly as it gets annoying. I haven't implemented the windowed mode yet, so tab swapping will have to do for now :p

I've been a little concerned about how CPU intensive the app performs when messing with UIs such as the color sliders and general component resizing. At he very least it doesn't cause the window to display "not responding" as does photoshop or flash if you play with the sliders too much :p Once windowed mode is supported it will probably be even more CPU intensive with each window watching other windows for changes :[ Interestingly though I realized that the mini version I posted actually uses a constant 40% of my CPU when displaying a palette (reminiscent to the old 3D renderer from the first version), but at least now it only knocks up CPU with dragging events.


Next up, the tile/graphic editor :)

[edit]

It turns out you can't always view NCGR's on their own as some have been packed in a way that if you don't have the associated NSCR or NCER it won't display properly, particularly with files that have a 257 tiled flag byte. So instead I've gotten the NCER working. My notes on decoding the cell data was definitely overcomplicated as changing the x/y byte to signed (see Garyong's notes) positions cells about 4px more centered than what I had achieved, though it still doesn't fix the gap with the new Pokemon sprites yet. Am still in the process of going over it, but the nostalgia is hitting again so I thought I might put up a picy of Coronamon (which was like one of the first sprites I was working on back when this project started for those who don't know). I've ditched the grid UI from the old version in favor of a blend between the Adobe Rulers and the Flash scene scrollPane (where you start in the middle of a scrollable area).

« Last Edit: November 17, 2010, 08:11:12 pm by Low Lines »

MegamanX

  • Full Member
  • ***
  • Posts: 210
    • View Profile
    • X Translations
Re: The Console Tool (by Low Lines)
« Reply #196 on: November 17, 2010, 04:23:35 am »
I downloaded the consoleToolMini in your sig, just to look at it because I have free time, and when I finally was able to get it running, everytime I drag a DS rom onto it, it just closes.  I have no idea why that is. 

Can anyone help me out?  This is the first java thing I've actually ran that wasn't on some website so I'm really clueless.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #197 on: November 17, 2010, 06:43:48 am »
@MegamanX
It opens four different palette formats and also NCGR, plus should automatically extract compressed files.

The Mini version was incredibly limited as to what it could do, and has mostly been an experiment to get a few concepts figured out (such as exporting/converting formats) that I was having troubles with. The complexity of the UI would also be (I think) alot more work to write code for. As of yet, I have not been posting regular builds so any new stuff I've been doing is still private. Sorry for any confusion :s

The first version did the most stuff, and would also open files as hex (iirc) if it doesn't know the file structure.
The second version focused on constructing the Adobe-styled UI as well as 3D rendering as well as a more user friendly file browser.
The third version (which was the mini) is as I stated above, very basic, and although was intended as a quicker way to put on file support, ended up just been an experiment (which has honestly helped clear my head so to speak). First version to support the export feature as well as a more structured file assets management.

The current (next) version I'm working on aims to do all of what I have achieved so far in terms of functionality, I also intend to start releasing progressive builds once I have a certain amount of stuff implemented, namely 2D Graphic and Archive support.

Current Functionality
- Drag & Drop UI (missing dock layout, minimize support, display options)
- Drop file opener
- Basic file browser (Mini version browser)
- Hex Editor (viewer only)
- Support for NDS LZ77/LZSS and RLUncomp compression
- Full support for editing 4 different palette formats (NCLR, Adobe Color Table ACT/RAW, Microsoft Palette, Usenti Palette)
- Partial support for NCGR
- Partial support for NCER

Current Milestones
- Finish Drag & Drop UI
- Internal Frames / windowed mode (not a priority)
- Right-Click Popup Support
- Advanced file browser (not a priority)
- Hex Editor (not a priority)
- Support for Graphic Editing/Exporting
- Support for 2D Maps (NSCR)
- Support for Cell/Sprites (editor is not a priority since there aren't any formats to covert to)
- Support for Cell/Sprite Animations
- Support for Archive browsing/editing (editing not a priority though)
- Support for 3D formats (not a priority)

So hopefully that's all very clear now for anyone who is wondering :)

MegamanX

  • Full Member
  • ***
  • Posts: 210
    • View Profile
    • X Translations
Re: The Console Tool (by Low Lines)
« Reply #198 on: November 17, 2010, 07:43:05 am »
For me, nothing would happen.  The program literally closed.  Nothing new would happen and I would have to relaunch it.  So I wasn't able to do anything with it.  Haha.

I can't wait for the full thing like everyone else that has posted.

Rhys

  • Hero Member
  • *****
  • Posts: 706
    • View Profile
    • CN
Re: The Console Tool (by Low Lines)
« Reply #199 on: November 17, 2010, 08:08:33 am »
I'm impressed you've managed to do so much already, once those features are added I don't think there'll be a need for any other ds hacking tools apart from this one