New Compression Scheme

I am also thinking about how to make a compressor that will work with any data no matter how random or patterned it is. So you could just run it over and over and it will progressively compact more and more. It eventually of course would reach a point that no more compression is achievable. I don't know though maybe that is a little paradoxical.

Burrows-Wheeler is what you are looking for. Of course it's completely impractical for real time stuff in consoles as it needs n² the memory space of whatever you want to compress, but it can compress pretty much any data even when other algorithms just give up.

I've only worked with LZSS up to now. Once I get this to a point I am satisfied I could move on to other compressions. LZ77 would be a good choice for it's compatibility with older systems, LZ78 also looks interesting.

I am also thinking about how to make a compressor that will work with any data no matter how random or patterned it is. So you could just run it over and over and it will progressively compact more and more. It eventually of course would reach a point that no more compression is achievable. I don't know though maybe that is a little paradoxical.

That's simply mathematically impossible.  Even if the size of the uncompressed data is fixed, 99.6% (254/255) of all possible files can't even be compressed by one byte.

If you were to try to improve LZSS, I'd suggest allowing larger values for the backcopy distance / length values; interpret a pair that would have a backcopy length of 18 to use a third byte for the backcopy / length data.   This will give a good increase for files which have long blocks of repeated data (such as bitmapped images with small palettes). 

And the length of the file does matter.  Backcopy-based compressors perform poorly at the early parts of their run, before they've processed the maximum backcopy distance.  It's simply less likely that there will be a match within the processed bytes.

Yes, back copy compressors in their purest form doesn't do well with repetition. But there is an easy trick to solve that problem. Allow having a longer copy length than the back seek distance. Then you suddenly are able to include RLE for free.