[Pikachumanson]

Just a hypothetical situation I was thinking about. Let's suppose I manage to crack a game's lszz(or whatever) text code and i translate the all the lines there but i don't bother to recompress which I imagine would be doing the reverse of what i did to crack the code. Would I still be able to play the rom?

Note: i would not ever do it like that, but I am just wondering what would happen because i am studying up on that format so I can take another crack at Necromancer(TG16) after I am finished translating Outlanders(NES)

[LostTemplar]

Depends. Either the game already supports uncompressed data, or you'll have to write a new routine that circumvents the decompression routine. Otherwise it probably won't work.

It's fairly standard with newer games (maybe SNES and later) to just reinsert data uncompressed because it's too much hassle to recompress it - writing a compressor is usually a lot harder than writing the decompressor - and the space is there anyway. Not only that, but it's usually a lot faster in-game because decompression takes a lot of time, whereas copying obviously  is pretty fast. For example, I've seen games that take a few seconds to decompress their tilesets each time you change the area, which is fairly noticeable on virtually-no-load-times consoles like the SNES.

[Pikachumanson]

So if a game has it's items and menus uncompressed but the intro and towntalk is compressed would you say it may support uncompression? Especially on a TG16 game made in 1987-88?

[FAST6191]

LostTemplar said pretty much everything I would have but I should note the GBA and DS often do not play the nicest with formerly compressed data as they have formal methods in BIOS and in the SDK but their compression methods are extremely well documented so that is not a problem. Some aspects of the DS (mainly overlays which is nice as the BLZ they use can be annoying to work with but I have seen several instances of file/archive level stuff) have little flags in headers and such to tell it is is compressed and will work from there.

I do wish to add though "recompression" is not necessarily the reverse for, as LostTemplar said, writing a compatible compression tool be be a right pain to do well (among the GBA and DS we even have things that are BIOS compatible (the main sort of compression there) and not even with quirks/undocumented stuff but are better than the stock/traditional compression). If it is LZSS like (if it is not Huffman or bit packing it probably will be a variation on this*) you can fake compression and just insert the "uncompressed" flag however often it needs to be inserted, it is not quite as fast as a straight copying in some cases (straight copy probably uses DMA where this will likely be the slightly slower conventional memory commands and lots of them) but it can also get a nice speed boost and save having to mess around with assembly**.

*I am not sure about RLE off the top of my head. It will probably come down to the given implementation but as it is seen as a special case of LZ it should not be that bad.

**most compression I see will have a magic stamp/indicator of some form so the ASM is not always so bad; you subvert the opening to be a IF ELSE arrangement with the IF being a jump to the usual decompression and the ELSE being a copy routine, bonus too is a lot of compression or decompression commands will include a file size anyway and you just construct a read or DMA in a handful of instructions which you should be able to easily find a place for.

[Pikachumanson]

Rle i can handle but for this game it will either dictionary or lz(probably huffman) and I have to figure out the intro which is compressed with non-contiguous text, in which i some reading and experimentation to do in the future. Thank you both.

[Zoinkity]

Results are case-by-case depending on the game, and occationally certain files that don't use the normal loader will hardcode a decompressor call.  Detection won't be automated unless there's an indicator or an expected filesize or something.  It does need some reference to know it's compressed, after all.

You might want to set a breakpoint on the loaded data to see what the decompressor expects, then backtrace to see how the decompressor is called. 

[KC]

Not only that, but it's usually a lot faster in-game because decompression takes a lot of time, whereas copying obviously  is pretty fast.

That is not necessarily correct for all systems. On the SNES you can access the data within a few clock cycles, but on other systems you have to wait for the disc drive to return the data. There, compression can save you A LOT of time, as obviously the drive is much slower than just moving stuff around in RAM a bit. The decompression time is often almost negligible compared to the time it would take to read the amount of data you saved with it.

Many hackers do seem to take pride in being able to reproduce a game's compression methods, too.

[LostTemplar]

That is not necessarily correct for all systems. On the SNES you can access the data within a few clock cycles, but on other systems you have to wait for the disc drive to return the data. There, compression can save you A LOT of time, as obviously the drive is much slower than just moving stuff around in RAM a bit. The decompression time is often almost negligible compared to the time it would take to read the amount of data you saved with it.

You're of course right.

Many hackers do seem to take pride in being able to reproduce a game's compression methods, too.

True as well.

[KingMike]

If you know how to program your own utility, RLE is probably one of the simpler methods to program a recompressor for.
LZ is kind of a pain to write a recompressor for. Fortunately, I've for the most part been able to just reuse one I've written and modify the source a bit as needed for each game.
For text on the NES, dictionary or DTE are probably most common. Before writing a custom tool, I used Hexposure (inside DOSBox) to insert DTE text (WindHex supports DTE entry, but I recall it being pretty buggy last I used it), and manually enter dictionary entries by having the table file and entering the hex codes.

[Zoinkity]

Surely you mean LZ with huffman encoding, because LZSS is probably the easiest thing next to RLE.  Derivations of it are also the fastest to decompress.  I mean, if you're going to say dictionary formats are easier to write compressors for then I seriously envy you ;*)

It's a little off-topic, but just curious: has anyone tried more modern compression types with NES games?  Historically, LZSS derivations designed primarily for speed and which don't require buffers or large amounts of RAM came after the NES era.  LZO comes to mind as one that's open-source.  How do these perform on a NES versus more common compression formats?

[KingMike]

NES has only 2KB internal RAM and most mappers only support up to 8KB of on-cart RAM. While you could probably use LZSS, it would eat up a lot of that space.

No, I don't think dictionary is easier to write a compressor for. I think I wrote one (and then might have just, like my LZ compressor, reused as much code as possible for other projects).
But of those I used it for, many of the scripts tended to be small enough that I could bother manually inserting the text in a hex editor (Hexposure, particularly. Was a pretty decent DOS hex editor. Just too bad it won't work natively on XP.)

[Zoinkity]

What about one of the LZSS types that decompresses a stream? 

I can understand the main concern is that the source wouldn't be compressed as well and if both were resident in memory at the same time this would take much more room, but what kind of overhead would prevent you from loading in 1-16 bytes at a time (or whatever size buffer), then refilling as you decompress until EOF or size is reached?  At that point RAM isn't an issue, but cart access speed. 
I've seen this in action a bit, mostly due to high decompression speeds as opposed to RAM requirements.  Still, backtraces can read directly out of written data and in practice it's only a little worse than RLE.

Wasn't there a win16 version of hexposure as well? 

[henke37]

LZSS requires at least a few hundreds byte of look back buffer for the decompression. Otherwise it kinda loses the edge it gives with back references. You can only refer to stuff that is in the look back buffer.

[Bongo`]

...

[Zoinkity]

LZSS requires at least a few hundreds byte of look back buffer for the decompression. Otherwise it kinda loses the edge it gives with back references. You can only refer to stuff that is in the look back buffer.

Traditional LZSS, but many later variants simply use output as the lookback buffer.  Without using a 1k buffer you're left with just the output and whatever size input buffer is handiest for the system.  Another advantage is you can then reference further back than the common buffer size by changing the backtrace code format.
About the only feature then that places traditional over inline would be using fill values within (most of) the first 1k of the file, short of implementing bitcode commands or something.  At that point though it really isn't LZSS anymore.

Dictionary models still have the (vast) advantage in compressed file size though.

[henke37]

The lookback buffer is indeed nearly always a part of the output buffer. But it is still size limited for various reasons.

[DougRPG]

The game runs an algorithm to extract the data, so the data must be on a specific format. If you insert the data uncompressed, the format will be invalid, and the algorithm will do the wrong thing.
Even if the compression method supports uncompressed data you probably need to insert some control simbols to tell the algorithm what to do.

There is no magic. The processor cannot guess what you are doing by its own. The processor is not a conscient being. It's a circuit that runs intructions. It's completelly dumb, and follows exactly the instructions on the program. So if you insert the wrong data the results will be wrong and unpredictable.