News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Utilities: Compress Tools upgraded to version 3.0  (Read 4913 times)

RHDNBot

  • Guest
Utilities: Compress Tools upgraded to version 3.0
« on: February 03, 2015, 05:48:49 pm »
Update By: Bregalad

For people looking to compress their data, Compress Tools is very versatile and support many compression algorithms, in addition to allow users to add their own.

The features are :
- Use input scripts as data (more versatile than binary files)
- Input data is defined as a set of randomly-accessible small data blocks (rather than a big file of binary data)
- Output to either assembly language or binary files
- Support non-ASCII encodings, support for unicode (UTF-8) input
- Can analyse data using all algorithms to find the best suited compression algorithm easily
- Open source, users can add their own compression / decompression algorithms to the existing ones

The program has been rewritten entirely in C++, bugs has been fixed, and it is now easier to use. Enjoy !

RHDN Project Page

Relevant Link: (http://www.romhacking.net/utilities/882/)

Pennywise

  • Hero Member
  • *****
  • Posts: 2249
  • I'm curious
    • View Profile
    • Yojimbo's Translations
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #1 on: February 03, 2015, 06:24:54 pm »
Does this tool support Konami's RLE compression?

I always had a hell of a time getting D's Graveyard Duck tool to run with python.

Bregalad

  • Hero Member
  • *****
  • Posts: 2637
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #2 on: February 04, 2015, 03:10:27 am »
Not currently, but that would be straightforward to add if the algorithm is documented.

Besides, I'd like to avoid distributing proprietary codecs for obvious reasons.

90s Retro Gamer

  • Full Member
  • ***
  • Posts: 149
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #3 on: February 06, 2015, 12:35:24 am »
Bregelad,

I downloaded the software and scanned through the readme.

Quote
3) The decompression algorithms for most existing tools are too complex, too slow and needs to much RAM for the NES which has only 2k of RAM and 32k of ROM.
   Even algorithms made for the Commodore 64, which also have a 8-bit 6502 processor, requires the usage of large RAM blocks which are absent on the NES.
   
4) Most existing tools just applies a compression algorithm on data, and doesn't do analysis.
   CompressTools does full analysis on all compression algorithms, to allow the user to see in a second which algorithms work the best and which ones works the less. Some algorithms code stuff on various numbers of bits, I even made it compress using ALL possible numbers of bits and return the most efficient one automatically. In other words, CompressTools doesn't just applies algorithms blindly, it will really analyse which variant of an algorithm works the best for YOUR data.
According to the SMW-Construction forums, Kirby Superstar (SNES/SFC) is difficult to ROM-hack because HAL used 7 layers of compression. Can your tool decompress those layers?

Another question I have is, can this tool compress let's say a .jpg image-file into a format that is readable for, oh let's say, NES/FC ROMs? For example, can I COMPRESS an image and OUTPUT it into an NES format and INSERT/PASTE it as background-scenery in a Super Mario Bros. 3 Sky-World level?

Thanks for any response.
« Last Edit: February 06, 2015, 05:56:29 am by 90s Retro Gamer »

Revenant

  • Full Member
  • ***
  • Posts: 203
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #4 on: February 06, 2015, 01:49:03 am »
According to the SMW-Construction forums, Kirby Superstar (SNES/SFC) is difficult to ROM-hack because HAL used 7 layers of compression. Can your tool decompress those layers?

There are already at least two other tools that can (with 8/16-bit Kirby games specifically in mind):
http://www.romhacking.net/utilities/1004/ (by me, a command-line utility; source available here)
http://www.romhacking.net/utilities/600/ (by Sukasa, a VB.NET DLL)

It's also exactly the same as some compression that EarthBound uses (for graphics? I have no idea) which I assume has had some tool support for it for ages now. (It's also extremely similar to Pokemon G/S/C's compression, as a fun fact!)

"7 layers of compression" is also some real wacky bullshit (ahem), it's really just the usual RLE and LZ77 with a few bells and whistles, and it's been documented and supported by other stuff for a while by now. It shouldn't be too hard to create a plugin based on my implementation if someone wants to (but keep in mind it has a maximum input size of 64kb :P)

Bregalad

  • Hero Member
  • *****
  • Posts: 2637
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #5 on: February 06, 2015, 03:34:11 am »
@90s Retro Game :
This program was meant mainly to compress data, not to decompress it. Compress Tools was developed for homebrew and translators that needs to compress back their script to fit a ROM.

Although, by exchanging encode() and decode() functions you could apply decompression to some data, but once again implementing proprietary compression algorithms used in commercial games is out of the scope of the maintained part of the project : Anyone can add their own extensions that does this if they'd like to : I will not merge them back with the main project because of the potential copyright issues.

What you describe (converting JPEG graphics to NES graphics) is completely impossible and has nothing to do with what this tool aims for. No offence but it seems you have absolutely no idea what a JPEG image is and how NES graphics works internally.

90s Retro Gamer

  • Full Member
  • ***
  • Posts: 149
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #6 on: February 06, 2015, 05:55:45 am »
@90s Retro Game :
This program was meant mainly to compress data, not to decompress it. Compress Tools was developed for homebrew and translators that needs to compress back their script to fit a ROM.

Although, by exchanging encode() and decode() functions you could apply decompression to some data, but once again implementing proprietary compression algorithms used in commercial games is out of the scope of the maintained part of the project : Anyone can add their own extensions that does this if they'd like to : I will not merge them back with the main project because of the potential copyright issues.

What you describe (converting JPEG graphics to NES graphics) is completely impossible and has nothing to do with what this tool aims for. No offence but it seems you have absolutely no idea what a JPEG image is and how NES graphics works internally.

I know what JPEG images are. JPEGs are compressed images, as opposed to raw bitmaps, hence why they have smaller file sizes. No, I do not the exact mechanics of NES graphics. I know that they utilize "tiles" and one can use Tile Layer Pro to edit the pixels in each "tile". I thought it was possible to convert JPEGs into NES backgrounds, as long as it was shrunk small enough to fit the limited resolution of an NES "screen" but perhaps I was mistaken.

tc

  • Hero Member
  • *****
  • Posts: 1125
  • Lum Fan
    • View Profile
    • Eon Blog
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #7 on: February 09, 2015, 08:22:57 pm »
Better to use plain old BMP for that. NES graphics have no need for JPEG's compression.

Marat

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #8 on: February 13, 2015, 07:16:49 am »
I tried compress script, but an error goes out for some reason.
What is wrong?


Bregalad

  • Hero Member
  • *****
  • Posts: 2637
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #9 on: February 13, 2015, 10:15:40 am »
Your input file should be encoded in UTF-8, it appears you are using another encoding (probably ISO-8859-5), so you should first somehow convert it into UTF-8 before processing.

What actually happens is that the last character, 0xc9, makes it belive it's the start of a multibyte UTF-8 sequence, and "eats" the following quote, thus reporting a missing quote.

If this bothers you, you could remove the UTF-8 decoding code in main.cpp at lines 315-330 and simply replace by the statement
Code: [Select]
c = s[i++];, that should work (although I cannot guarantee it will).

Marat

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Utilities: Compress Tools upgraded to version 3.0
« Reply #10 on: February 13, 2015, 01:45:58 pm »
Your input file should be encoded in UTF-8, it appears you are using another encoding (probably ISO-8859-5), so you should first somehow convert it into UTF-8 before processing.

Thank you!
Everything worked.