Romhacking.net

General Category => News Submissions => Topic started by: RHDNBot on February 03, 2015, 05:48:49 pm

Title: Utilities: Compress Tools upgraded to version 3.0
Post by: RHDNBot on February 03, 2015, 05:48:49 pm
Update By: Bregalad

For people looking to compress their data, Compress Tools is very versatile and support many compression algorithms, in addition to allow users to add their own.

The features are :
- Use input scripts as data (more versatile than binary files)
- Input data is defined as a set of randomly-accessible small data blocks (rather than a big file of binary data)
- Output to either assembly language or binary files
- Support non-ASCII encodings, support for unicode (UTF-8) input
- Can analyse data using all algorithms to find the best suited compression algorithm easily
- Open source, users can add their own compression / decompression algorithms to the existing ones

The program has been rewritten entirely in C++, bugs has been fixed, and it is now easier to use. Enjoy !

RHDN Project Page (http://www.romhacking.net/utilities/882/)

Relevant Link: (http://www.romhacking.net/utilities/882/)
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: Pennywise on February 03, 2015, 06:24:54 pm
Does this tool support Konami's RLE compression?

I always had a hell of a time getting D's Graveyard Duck tool to run with python.
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: Bregalad on February 04, 2015, 03:10:27 am
Not currently, but that would be straightforward to add if the algorithm is documented.

Besides, I'd like to avoid distributing proprietary codecs for obvious reasons.
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: 90s Retro Gamer on February 06, 2015, 12:35:24 am
Bregelad,

I downloaded the software and scanned through the readme.

Quote
3) The decompression algorithms for most existing tools are too complex, too slow and needs to much RAM for the NES which has only 2k of RAM and 32k of ROM.
   Even algorithms made for the Commodore 64, which also have a 8-bit 6502 processor, requires the usage of large RAM blocks which are absent on the NES.
   
4) Most existing tools just applies a compression algorithm on data, and doesn't do analysis.
   CompressTools does full analysis on all compression algorithms, to allow the user to see in a second which algorithms work the best and which ones works the less. Some algorithms code stuff on various numbers of bits, I even made it compress using ALL possible numbers of bits and return the most efficient one automatically. In other words, CompressTools doesn't just applies algorithms blindly, it will really analyse which variant of an algorithm works the best for YOUR data.
According to the SMW-Construction forums, Kirby Superstar (SNES/SFC) is difficult to ROM-hack because HAL used 7 layers of compression. Can your tool decompress those layers?

Another question I have is, can this tool compress let's say a .jpg image-file into a format that is readable for, oh let's say, NES/FC ROMs? For example, can I COMPRESS an image and OUTPUT it into an NES format and INSERT/PASTE it as background-scenery in a Super Mario Bros. 3 Sky-World level?

Thanks for any response.
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: Revenant on February 06, 2015, 01:49:03 am
According to the SMW-Construction forums, Kirby Superstar (SNES/SFC) is difficult to ROM-hack because HAL used 7 layers of compression. Can your tool decompress those layers?

There are already at least two other tools that can (with 8/16-bit Kirby games specifically in mind):
http://www.romhacking.net/utilities/1004/ (by me, a command-line utility; source available here (https://github.com/devinacker/exhal))
http://www.romhacking.net/utilities/600/ (by Sukasa, a VB.NET DLL)

It's also exactly the same as some compression that EarthBound uses (for graphics? I have no idea) which I assume has had some tool support for it for ages now. (It's also extremely similar to Pokemon G/S/C's compression, as a fun fact!)

"7 layers of compression" is also some real wacky bullshit (ahem), it's really just the usual RLE and LZ77 with a few bells and whistles, and it's been documented and supported by other stuff for a while by now. It shouldn't be too hard to create a plugin based on my implementation if someone wants to (but keep in mind it has a maximum input size of 64kb :P)
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: Bregalad on February 06, 2015, 03:34:11 am
@90s Retro Game :
This program was meant mainly to compress data, not to decompress it. Compress Tools was developed for homebrew and translators that needs to compress back their script to fit a ROM.

Although, by exchanging encode() and decode() functions you could apply decompression to some data, but once again implementing proprietary compression algorithms used in commercial games is out of the scope of the maintained part of the project : Anyone can add their own extensions that does this if they'd like to : I will not merge them back with the main project because of the potential copyright issues.

What you describe (converting JPEG graphics to NES graphics) is completely impossible and has nothing to do with what this tool aims for. No offence but it seems you have absolutely no idea what a JPEG image is and how NES graphics works internally.
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: 90s Retro Gamer on February 06, 2015, 05:55:45 am
@90s Retro Game :
This program was meant mainly to compress data, not to decompress it. Compress Tools was developed for homebrew and translators that needs to compress back their script to fit a ROM.

Although, by exchanging encode() and decode() functions you could apply decompression to some data, but once again implementing proprietary compression algorithms used in commercial games is out of the scope of the maintained part of the project : Anyone can add their own extensions that does this if they'd like to : I will not merge them back with the main project because of the potential copyright issues.

What you describe (converting JPEG graphics to NES graphics) is completely impossible and has nothing to do with what this tool aims for. No offence but it seems you have absolutely no idea what a JPEG image is and how NES graphics works internally.

I know what JPEG images are. JPEGs are compressed images, as opposed to raw bitmaps, hence why they have smaller file sizes. No, I do not the exact mechanics of NES graphics. I know that they utilize "tiles" and one can use Tile Layer Pro to edit the pixels in each "tile". I thought it was possible to convert JPEGs into NES backgrounds, as long as it was shrunk small enough to fit the limited resolution of an NES "screen" but perhaps I was mistaken.
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: tc on February 09, 2015, 08:22:57 pm
Better to use plain old BMP for that. NES graphics have no need for JPEG's compression.
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: Marat on February 13, 2015, 07:16:49 am
I tried compress script, but an error goes out for some reason.
What is wrong?
(http://savepic.ru/6735181.png)
(http://savepic.ru/6726989.png)
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: Bregalad on February 13, 2015, 10:15:40 am
Your input file should be encoded in UTF-8, it appears you are using another encoding (probably ISO-8859-5), so you should first somehow convert it into UTF-8 before processing.

What actually happens is that the last character, 0xc9, makes it belive it's the start of a multibyte UTF-8 sequence, and "eats" the following quote, thus reporting a missing quote.

If this bothers you, you could remove the UTF-8 decoding code in main.cpp at lines 315-330 and simply replace by the statement
Code: [Select]
c = s[i++];, that should work (although I cannot guarantee it will).
Title: Re: Utilities: Compress Tools upgraded to version 3.0
Post by: Marat on February 13, 2015, 01:45:58 pm
Your input file should be encoded in UTF-8, it appears you are using another encoding (probably ISO-8859-5), so you should first somehow convert it into UTF-8 before processing.

Thank you!
Everything worked.