Emulators and ROM management tools have supported zip (and even rar and 7zip in some cases) for years at this point so I am assuming it is not that.
This leaves you wanting something to compress parts of a ROM to fit more in a ROM hack or something to compress parts of a homebrew you are making.
As far as I am aware the SMS did not include any BIOS level decompression routines which means you are not going to get anything like we saw on the GBA and DS. Similarly I don't think Sega or any big library makers of the time made a format that got terribly widespread. SMS homebrew is pretty minimal so I doubt there are a bunch of nice libraries floating around similar to what we saw with the GBA and DS.
This means you are probably left making you own, or porting across something from the NES (the SMS and NES are similar enough in power) and then such things will probably be more like inspiration than porting. NES stuff, and most stuff I see on similarly powered systems, for decompression in real time whilst still doing something else is limited to some of the most basic forms of compression like run length encoding (RLE) and really short window LZ style compression (hard to manipulate 16 bits of data in the look ahead/read length aspect on an 8 bit machine without taking 10 times the number of instructions and time to do them all), maybe a basic lookup/substitution (full blown Huffman would be a bit strong) as well as classic pointer style compression for text that can handle it.
Compression gains will be similarly minimal as a result of how basic this stuff is and some kind of constraint upon it.
If you are good enough to hack it into a game then you are similarly good enough to not need a library, or indeed probably see the library get in your way more than it helps (even on later systems where we have assemblers capable of injecting code into random points in the ROM such that you could have a "library" it still takes a lot of fiddling for a lot of things). For homebrew as you are unlikely to be writing in C or something high level for this then same problem really. If more exotic compression could be achieved with half sensible results then that would be one thing but RLE and substitutions are nothing drastic to code.