News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Nintendo DS - Weird LZ10 Compression  (Read 772 times)

Cereal

  • Newbie
  • *
  • Posts: 2
    • View Profile
Nintendo DS - Weird LZ10 Compression
« on: March 11, 2018, 10:06:22 pm »
I really can't figure this out.
I'm trying to rip the tiles from Kirby Squeak Squad. This file contains a tileset, appears to be LZ10 compressed(from the header) but I can't decompress it.

Here is the header and the first ~250 bytes
Code: [Select]
10 00 00 00 00 02 00 00 00 90 00 00 80 04 00 00
EF 3D 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1F 7C FF 7F DF 5B 7D 4F FD 3E 7D 2A D7 19 51 11
8D 08 F6 7F A0 7F 4C 7F E9 7E 60 7E E0 7D 80 69
E0 03 FF 7F 9F 7F 5A 7F F4 7E AF 7E 00 00 00 00
00 00 F6 7F A0 7F 4C 7F E9 7E 6A 7A CC 75 4D 71
1F 7C 10 32 59 01 7F 02 DF 23 FF 7F 00 00 4A 29
10 4A 73 62 18 7B E5 79 8E 2D D0 31 3F 03 F8 01
1F 7C 9E 3F FE 02 F8 01 5D 16 34 01 8C 00 00 00
1F 3B D5 15 EE 00 1B 1E FF 7F 35 26 70 19 67 00
1F 7C A6 1D 28 26 AB 2E EC 32 8F 37 00 00 8E 2D
D0 31 13 3A F6 7F 80 7F 0A 7F CB 76 4B 72 09 62
1F 7C 0D 15 93 19 37 22 9B 2A 49 0A 44 1A B1 16
E8 22 30 1F 53 23 39 1B 6E 17 D3 1B F9 23 FF 4F
1F 7C EC 27 F4 7F 8A 73 E3 5A FF 4F A5 19 65 1E
E8 22 6E 17 D3 1B F9 23 29 27 DF 6A 3F 5E 5C 45
1F 7C 94 4E D5 52 F8 52 18 5F 1A 5B 3B 5F BD 6F
FF 77 A5 19 65 1E 0E 1F B1 31 E5 79 C8 10 EF 3D
1F 7C EC 27 57 79 AF 58 AC 40 FC 7D A5 19 65 1E
8C 32 0E 1F 51 43 F7 23 B1 31 FF 67 9F 37 FF 1E
1F 7C 35 32 B9 46 3B 4F 7D 53 DE 5F A5 19 65 1E
E8 22 CA 1D 4D 1E 8F 1E 11 27 72 23 D6 23 FC 2B
1F 7C 42 08 BB 2E 58 2A 15 22 B2 1D 2C 1D C7 1C
70 3B 08 23 46 26 A3 29 02 1D 3D 05 D4 08 1F 7C
1F 7C 0D 15 93 19 37 22 9B 2A FF 4F A5 19 65 1E
E8 22 6E 17 D3 1B F9 23 B1 31 F4 35 37 3A 09 62
1F 7C 70 19 18 0E DE 2E 9F 4F 44 29 C1 21 65 1E
E8 22 29 27 EC 27 F7 2F B1 31 F4 35 37 3A EA 0C
1F 7C 8E 2D D0 31 00 00 F1 7F FF 4F A5 19 65 1E
E8 22 6E 17 D3 1B F9 23 09 62 69 7A EC 7E 6E 7F
1F 7C 3B 63 97 4E F1 3D 8E 2D 4B 21 09 19 A8 0C
1A 0D 90 00 A5 19 65 1E B1 31 F4 35 37 3A 1F 7C
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
B9 55 AB A9 AA A9 5B BB A9 BA BB 59 5B A9 9B 9B
5A AB 9B B9 BB A9 5B 95 9A 5B 5B BA 9A B9 A5 AA
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60
00 66 67 76 66 77 76 77 76 77 77 78 77 78 87 87
00 00 00 00 00 00 00 00 00 70 76 66 67 76 67 67
76 67 77 77 77 86 87 77 77 78 A8 78 A9 A8 88 8A
00 00 00 00 00 00 00 00 00 00 00 00 66 66 67 76
(Near the end after the long block of gibberish, theres the actual tile colors. Tiles are 16 colors, 16x16 size.)


As I understand, there is always a number after the first byte in the header(0x10 in this case) that defines the length of the uncompressed file, but that seems to be missing, does that mean it's not actually compressed? And that the 10 in the header means something else? Which would be unconventional because that's always the number for LZ10 compression.
But I can't correctly display the graphics either so I do believe it's compressed somehow...

FAST6191

  • Hero Member
  • *****
  • Posts: 2384
    • View Profile
Re: Nintendo DS - Weird LZ10 Compression
« Reply #1 on: March 12, 2018, 08:27:00 am »
10 is one of the LZ compression markers (11 came a bit later and 40 for an even smaller amount) on the DS.
You would be right that after that one would expect a length value.

Going further at the end of the selection there are a whole bunch of 00 and other repeating values. While the compressions seen on the DS were hardly the hyper efficient stuff they are not going to miss that, even something as basic as RLE would catch it.
Beyond that I see nothing I would recognise as flags.

10 is not a reserved value for the start of files (indeed outside the header there are no reserved anything for the DS) and there are plenty of coincidences I have seen over the years.

Unconventional tile size is where I would head next. Crystaltile2 being my main tile editor in these scenarios. Could be anything, especially for a font, but do powers of 2 (8,16,32,64,128 and 256 sort of thing) first. I know you said "Tiles are 16 colors, 16x16 size" but did you rip that from OAM readouts and its controlling registers or something?

If you are facing an unknown/custom compression (I have seen a few on the DS, Nintendo captive devs certainly not being above it) then regardless of anything else most would not leave compressed data in RAM for longer than it takes to decompress it. To that end you can dump the memory/VRAM/whatever and start looking for patterns in the compressed file and otherwise.

Cereal

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Nintendo DS - Weird LZ10 Compression
« Reply #2 on: March 13, 2018, 07:20:17 pm »
Thanks for the reply.

I was wrong about tile size, ingame the tiles are 16x16, but tiles in the file are actually 8x8 and one ingame tile is usually 4 of those together.
Another problem was however not being able to load the palette at all.
All that stuff before the actual tiles is palette data, but it only loads the first 32 bytes after the header and since they are by default mostly 00, the graphics can't be recognized at all.
Every 32 bytes from then appears to be another palette. Most tilesets use different palettes for different tiles and since I can only load 1 at a time, ripping is a bit tedious, but at least I can do it now^^
This is the debug tileset, which only uses 1 palette so I didn't have to go through the trouble of stitching it together:

The first byte appears to just define the size of the header :/  Was pretty misleading though, Tinke also recognized it as a compressed file.

Alosromax

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: Nintendo DS - Weird LZ10 Compression
« Reply #3 on: March 18, 2018, 10:47:50 pm »
As I have seen the issue of this kind of story as mentioned above, I might have brought a friend to help you....