News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The Console Tool (by Low Lines)  (Read 252456 times)

Klarth

  • Sr. Member
  • ****
  • Posts: 492
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #20 on: January 02, 2010, 04:29:39 pm »
Agreed.  Though to be honest, 95% of my interest is geared towards PSX CD-image building and extraction so I won't have to code it myself.

Gemini

  • Hero Member
  • *****
  • Posts: 2024
  • 時を越えよう、そして彼女の元に戻ろう
    • View Profile
    • Apple of Eden
Re: The Console Tool (by Low Lines)
« Reply #21 on: January 02, 2010, 05:21:05 pm »
Can't you already do that with pixel's Cd-Tool? I'm pretty sure it can even manage correctly virtual file systems (if they aren't too complicated, that is).
I am the lord, you all know my name, now. I got it all: cash, money, and fame.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #22 on: January 02, 2010, 05:33:03 pm »
Agreed.  Though to be honest, 95% of my interest is geared towards PSX CD-image building and extraction so I won't have to code it myself.
Yeah I had started working on that, except the code I came up with for reading PSX BIN files was really slow (because you have to jump to a sector, read part of it and then read the next sector until you have your file), which is really bad for my automatic file detection because the only way I know of identifying an iso is by reading sector 16 which is the primary header or whatever, so any files bigger than 16 sectors will get checked and relying on the ".bin" extension is useless because its a VERY popular file extension for MOST files in games -_-

What I've found is that its much quicker to load load large files into Java's memory than to be constantly accessing it through file access methods, however Java also gradually builds up wasted memory space, so over time especially when accessing large files, it will run out of memory. I've already increased said maximum memory to 256MB which seems to be enough even for accessing some of the larger files from NDS games, if anyones used Tile Molester, they would know it's limited to files at a maximum size of 64 MB, thus making it less useful for direct editing of a lot of the new game images.

Klarth

  • Sr. Member
  • ****
  • Posts: 492
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #23 on: January 03, 2010, 12:33:25 am »
Can't you already do that with pixel's Cd-Tool? I'm pretty sure it can even manage correctly virtual file systems (if they aren't too complicated, that is).

Hmm...I suppose I could write a general purpose script to dump and rebuild PSX games via Cd-Tool.  Just that I'd hate to have to use LUA.  On the other hand, it could save me a lot of time.  I'll probably go this route.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #24 on: January 03, 2010, 04:09:27 pm »
Hmm wells I was "hoping" I would have a nice little preview of sprite animations working by the end of the day, but no...GIF images, I never knew how much of a pain it is to read them (let alone write them from scratch). I spent all night trying to make head or tail of the Wiki documentation on LZW compression, before finally understanding how it works!! It's very vague I think and the examples have horrible paragraphing which makes it even harder to line stuff up!!

So tomorrow, I'm going to try and get GIFs supported, and THEN write an exporter so I can show off animations :)
Also after doing some research, I think that maybe adding support for APNGs might be an idea too...I mean they aren't heavily supported yet but wells...look at PNG :p

The last thing I'd like to make note about today, is correcting myself on some of the notes I've been making on the NCER format. For starters the mappable range is actually 512x512 and not 384x384, and also the so called "fix" I came up with actually did nothing ... increasing the mappable range was what fixed the crocodile sprite I showed earlier. All NCERs that follow the offical format work fine, it's just the custom Kirby BCEA ones that are playing up, where like the bottom half of a character is positioned above its head, with no obvious errors in my code -_-

Ohs and I've been doing a bunch for fixes on the GUI, such as properly implementing Byte Order swapping/detecting (so you can reading both types without having to worry too much about it), and I have made a start on the Hex Viewer, starting with some dumb bugs that have haunted it for the past year etc.

Time for sleep...and I shall dream about GIFs laughing at me :'(

creaothceann

  • Hero Member
  • *****
  • Posts: 2619
  • SPINESHARK
    • View Profile
    • creaothceann's website
Re: The Console Tool (by Low Lines)
« Reply #25 on: January 03, 2010, 06:25:24 pm »
I spent all night trying to make head or tail of the Wiki documentation on LZW compression, before finally understanding how it works!! It's very vague I think and the examples have horrible paragraphing which makes it even harder to line stuff up!!
Yeah, I tried to understand it at one point and gave up because of the documents. :P

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #26 on: January 04, 2010, 12:30:10 am »
Wells LZW, from what I understand is basically a command based compression method, as opposed to say LZ77. Data is broken up into 9 bit lengths and read one at a time. The first code in any data should always be 0x100 which is the command called CLR. Codes that are between 0x00-0xFF are uncompressed colors, and codes between 0x100-0x107 are commands. I'm still in the process of understanding how each command works but even my crappy explanation makes better sense than the "proper" ones I've found on the internet. Once have it figured out, I'll try and explain how I did it :)

[edit]

I think I'm starting to understand LZW better now. For a gif, the initial table of values is from 0x0-0x101...0x0-0xFF obviously being your 256 colors, and 0x100 and 0x101 are reserved for "CLR" and "END" commands respectively. Any code greater than 0x101 points to the extended part of your table which gets updated as you go through the file. A code value of 0x102 basically means the first color 0 repeated twice, a value of 0x103 is the first color and the second color (ie 0, 1) and so on.
The really confusing part in all the documentation is all this "string" crap they go on about, like one author claimed that every code could be translated into a string, even though string character don't start until 0x30 in terms of bytes, which makes no sense to me. So instead I'm using a double byte array for my "string table". I'm no expert, but honestly how can comparing strings be reliable when doing something like this? Anyways with luck the route I'm taking will just work and I can move on and get GIFs out of the way...

I also attempted at adding PNG support (in which to further add APNG support), but again I got stuck on the compression part, DEFLATE being significantly harder for me to make sense of compared to horrid little LZW.

However I really like the file structure of the PNG, it stores each piece of data in its own chunk, allowing for custom chunks to be added without crippling support with earlier versions. In the future I think making up my own custom formats using this chunk structure may be a much better way at manipulating files such as grouped ones like NCLR, NCGR, NSCR, NCER, NANR. My app works in a similar way when handling multiple types of data, it checks if combinable files are presently stored in memory and then loads them together. Also having a custom format, would allow me to retain a lot of the data that often gets lost when converting custom formats to pc based ones, especially with ones like mappers and sprite cells. It's just an idea at the moment though.
« Last Edit: January 04, 2010, 02:19:54 pm by Low Lines »

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: The Console Tool (by Low Lines)
« Reply #27 on: January 04, 2010, 02:31:54 pm »
http://en.wikipedia.org/wiki/DEFLATE

Doesn't seem that complicated...if you've ever worked with LZ compression and Huffman coding, combining the two makes perfect sense.
In the event of a firestorm, the salad bar will remain open.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #28 on: January 04, 2010, 11:10:57 pm »
http://en.wikipedia.org/wiki/DEFLATE

Doesn't seem that complicated...if you've ever worked with LZ compression and Huffman coding, combining the two makes perfect sense.

It's mostly that I get stuck at a particular point in the process because I don't understand. I haven't implemented any form of Huffman yet, so that's all new to me, but I do understand how the string table works, but what I don't understand is how I generate it and read from it in the decompression process. I probably should be doing it the other way around (compression then decompression), but wells :p

Here's the data of a PNG IDAT chunk representing a one pixel image with an RGB of (237, 28, 36).

Code: [Select]
0x185763782BA302000327012E
The first two bytes are ZLIB flags and the last 4 bytes are a custom CRC of the decompressed data.

The ZLIB flags tell me 3 things. Compression mode is DEFLATE, Window Size is 2048, and there is no "preset dictionary" present before the actual data starts. So therefore I move to byte 3 (0x63) which is the start of my DEFLATE block.

The first three 3 bits are the block header. The first bit indicates whether this is the last block (which it is), and the other two specify how the data is stored (which is method 1). After this point I am stuck. I know I'm supposed to read each bit one at a time and travel down the tree until I get a value, but I don't know how to start it.

I'm using this document as my reference. I have been using the Wiki reference too, but some of their notes on this stuff is a pain to understand (like with LZW).
Assuming my process is right up to this point, compression type 1 uses fixed Huffman codes (see link). I see the information in the documentation (Section 3.2.6) but I'm not sure how I'm supposed to use it etc. Like am I already supposed to have a partial table generated? It may seem simple to some (or sadly most), but it's new to me and I haven't looked at it long enough to go crossed-eyed and see the hidden picture :p

KaioShin

  • RHDN Patreon Supporter!
  • Hero Member
  • *****
  • Posts: 5698
    • View Profile
    • The Romhacking Aerie
Re: The Console Tool (by Low Lines)
« Reply #29 on: January 05, 2010, 04:39:02 am »
To decompress Huffman you need the Huffman tree. It must be stored with the compressed data somewhere, probably in some sort of header. If you have the tree the rest will be easy.
All my posts are merely personal opinions and not statements of fact, even if they are not explicitly prefixed by "In my opinion", "IMO", "I believe", or similar modifiers. By reading this disclaimer you agree to reply in spirit of these conditions.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #30 on: January 12, 2010, 05:20:55 am »
I managed to get GIFs mostly decompressed, just there's a bug in my code that stuffs up LZW decompression at some point and that's as far as I got with it :/

Putting aside my war with the compression formats...I've started working on some gamecube stuff. GCM files should now work and Yaz0 compression has been added along with RARC archive support. My main focus however is adding support for gamecube models. I figure that in order to understand NDS models better, I should see how a different model format works (a more complex one), and then build my model viewer based on that.

So far I've added support for most texture types, S3TC, I4, I8, IA4, IA8, CI4, CI8 and partial support for RGB5A3 and RGB565. My next step is understanding all the model data (which there is ALOT of...). Expect more to come :)


[edit]

Okay...this is my first attempt at drawing gamecube models...It's a hat!! Can't you tell??


I'm using what notes thakis has posted on the formats, though I'm pretty much at the point where I have to figure things out on my own. Having taken hours just writing up the code to interpret about 80% of the file, I just wanted to see my progress...but obviously there's still a long way to go ;D Here's what the model above is SUPPOSED to look like if anyones interested in comparing :p

« Last Edit: November 17, 2010, 08:37:27 pm by Low Lines »

BRPXQZME

  • Hero Member
  • *****
  • Posts: 4572
  • じー
    • View Profile
    • The BRPXQZME Network
Re: The Console Tool (by Low Lines)
« Reply #31 on: January 13, 2010, 10:24:23 am »
Mmm... polygon goulash....
we are in a horrible and deadly danger

creeperton

  • Hero Member
  • *****
  • Posts: 604
    • View Profile
.
« Reply #32 on: January 13, 2010, 12:49:52 pm »
.
« Last Edit: November 16, 2015, 12:15:29 am by creeperton »

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #33 on: January 14, 2010, 07:56:00 pm »
Okay not so meshy anymore :p


I've been updating the model GUI a little as well, and reworked my model viewer to operate in a chunk like style instead of the piles of custom code I had for viewing NDS models specifically -_- Hopefully I can figure out why these gamecube models are being stretched out :)
« Last Edit: November 17, 2010, 08:38:13 pm by Low Lines »

Vanya

  • Hero Member
  • *****
  • Posts: 1787
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #34 on: January 15, 2010, 12:16:03 am »
Have you released a demo yet?

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #35 on: January 16, 2010, 02:19:47 am »
Yay progress!


99% of the problem was me not understanding the documentation :p I found an easy to interpret Floating Point Calculator on Google and from that I realized that "**" apparently stands for "to the power of". I also looked through Thakis' bmdview2 source and finally understand how he reads Fixed Point Values as well. So basically now simple models (that don't require Joints etc) display properly and I am feeling relieved to not be staring at meshies all day.

I actually only spent like 15 minutes on figuring out my problem with the Gamecube models, as I've been overhauling my NDS model code more and gradually improving the model GUI as well.
« Last Edit: November 17, 2010, 08:38:59 pm by Low Lines »

Ryusui

  • Hero Member
  • *****
  • Posts: 4989
  • It's the greatest day.
    • View Profile
    • Tumblr
Re: The Console Tool (by Low Lines)
« Reply #36 on: January 16, 2010, 02:43:35 am »
This is going to be fricking incredible when it's finally released.
In the event of a firestorm, the salad bar will remain open.

Celice

  • Hero Member
  • *****
  • Posts: 644
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #37 on: January 16, 2010, 03:43:18 pm »
Awwwweeeeeessssoooommmeeee.

Penance

  • Jr. Member
  • **
  • Posts: 87
    • View Profile
Re: The Console Tool (by Low Lines)
« Reply #38 on: January 16, 2010, 03:59:29 pm »
Wow, completely astounded. Can't wait to play around with this.

Low Lines

  • Full Member
  • ***
  • Posts: 173
    • View Profile
    • lowlines
Re: The Console Tool (by Low Lines)
« Reply #39 on: January 17, 2010, 09:28:38 am »
Sorting through all the data in these Gamecube Files is seriously hectic, mostly because a lot of things point to another offset which then points to ANOTHER offset before you "might" even get the data you want. Before you know it, you have 100 lines of code purely for seeing this data so you can get your head around it.

Anyways, I've halted work on them for the moment because I figured if I could get my Plugin Concept operational, it would make decoding files with massive amounts of data like this much quicker...or at least I hope it will...


This is VERY much still a work in progress, so bare with me here. Now the idea is that plugins are stored in XML files made up of 5 or more sections. Firstly the header, which has all the information needed to recognize the file and give it a custom icon etc. The second part is a file structure tree that maps the contents of the file. Variables that rely on other variables will use a simple byte code to reference, test, jump to offsets and whatever.
The next section is for extended code, and/or I may put all byte code here for neatness or whatever later.
The fourth section is for external variables. Sometimes the data you want has to be decoded before you can actually make use of it etc.
The last section is simply the read/output function. It gathers up the data needed to load the file using my generic data structures into the app.

Now the idea is that once I have this up and working, you'd be able to write your plugins via the GUI using the window above. So far my plugin tool can load up the NCLR plugin, and seems to actually check for the PMCP section (which isn't always present) and omits it from the tree. But it's buggy as hell and my byte code decoder is VERY basic atm so there's still work to do on it.
« Last Edit: November 17, 2010, 08:39:56 pm by Low Lines »