It's likely to be a game-specific.
SMSPower guy Maxim made a decompressor
for a format that's supposedly used on a lot of SMS games (a RLE format the SMSPower members call the "Phantasy Star" compression, I'm guessing it was the first game they cracked that format on).
If doesn't work, the only way to find out the format would be to learn Z80, and start tracing the code.
If you haven't already asked at SMS Power, it would probably be a good idea to.