The exact format is:
1st 2 bytes is decompressed file size
Then 1 byte for 8 compression bit flags 76543210
bit flags are read one at a time starting at 0 then 1...7
if bit flag = 0 use a sliding window that has a size of no less than 3 and no more than 18
if bit flag = 1 load a non-compressed byte from file(this is what I used for the 1st test compressor which is why file was larger)
xx yx sliding window is 2 bytes in size xx is low byte yx is high byte
switch these to yx xx
x xx is the location that the sliding window starts
y is the size of the sliding window
if y = 0x0, sliding window has a size of 3
if y = 0xf, sliding window has a size of 18
Here is a example of the compressed gfx at $84:B727:
00 04 //size
FD //compression flags, 0xfd = 1111 1101, read flags from right to left
FF //bit flag = 1, store non-compressed byte from file
00 C0 //bit flag = 0, use sliding window, 00 = low c0 = high, switch to c0 00, 0x000 is window starting location, 0xc + 3 = size of window
3F //bit flag = 1, store non-compressed byte from file
00 //bit flag = 1, store non-compressed byte from file
60 //bit flag = 1, store non-compressed byte from file
1F //bit flag = 1, store non-compressed byte from file
5F //bit flag = 1, store non-compressed byte from file
BF //bit flag = 1, store non-compressed byte from file
This is the decompressed data:
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
3F 00 60 1F 5F BF
If 0x1000 bytes are stored 7e:4000-7e:4fff then dma those bytes to vram and then finish decompressing
start back over at 7e:4000.
So I was able to create the compressor with the sliding window that creates the same
exact compression size but some of the sliding window location are different. The ones that
are different are the ones that find multiple matches of the same values.
Here is a example of $84:B727 decompressed:
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
3F 00 60 1F 5F BF 7F BF 7F BF 7F BF 7F BF 7F BF
FF 00 00 FF FF FF FF FF FF FF FF FF FF FF FF FF
F8 00 04 F8 F2 FC FA FC FA FC FA FC FA FC FA FC
7F BF 7F BF 7F BF 7F BF 7F BF 7F BF 7F BF 7F BF
FA FC FA FC FA FC FA FC FA FC FA FC FA FC FA FC
7F BF 7F BF 7F BF 7F BF 5F BF 2F DF 1F 60 00 3F
FF FF FF FF FF FF FF FF FF FF FF FF FF 00 00 FF
FA FC FA FC FA FC FA FC FA FC F4 FA F8 04 00 F8
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
7C 83 C6 39 DE 39 DE 39 DE 39 DE 39 7C 83 01 FF
DB E7 FB C7 DB E7 FB E7 FB E7 FB E7 FD C3 C1 FF
7C 83 C6 39 1E F9 BC C3 60 9F DE 39 FE 01 00 FF
7C 83 C6 39 3E F9 FC C3 C6 F9 FE 39 7C 83 00 FF
FD E3 FD C3 ED 93 CD 33 FE 01 0C F3 FE E1 E0 FF
FE 01 C0 3F FC 03 06 F9 FE F9 FE 39 7C 83 00 FF
7C 83 C6 39 D8 3F FC 03 C6 39 DE 39 7C 83 00 FF
FE 01 C6 39 2C F3 D9 E7 F7 CF F7 CF FB 87 83 FF
7C 83 C6 39 DE 39 7C 83 C6 39 DE 39 7C 83 00 FF
7C 83 C6 39 DE 39 7E 81 06 F9 DE 39 7C 83 01 FF
FF FF F7 CF F7 CF C7 FF F7 CF F7 CF C7 FF FF FF
FF FF FF FF F7 CF C7 FF F7 CF F7 CF D7 EF E7 DF
F7 CF F7 CF F7 CF F7 CF C7 FF F7 CF F7 CF C7 FF
F7 CF F7 CF D7 EF E7 DF CF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF F7 CF F7 CF D7 EF E7 DF
79 87 8D 73 3D F3 D9 E7 F3 CF C7 FF F7 CF C7 FF
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
FF C7 FB C7 AB D7 ED 93 7D 83 C6 39 DE 39 18 FF
FD 03 66 99 EE 99 FC 83 E6 99 EE 99 FC 03 00 FF
FD 83 C6 39 D8 3F DF 3F DF 3F DE 39 7C 83 81 FF
FF 03 67 99 EE 99 EE 99 EE 99 EE 99 FD 03 03 FF
FE 01 62 9D EC 9F FB 87 E3 9F EE 9D FE 01 00 FF
FE 01 62 9D EC 9F FB 87 E3 9F EF 9F F7 0F 07 FF
FD 83 C6 39 D8 3F DE 31 D6 39 DE 39 7D 83 83 FF
DE 39 DE 39 DE 39 FE 01 C6 39 DE 39 DE 39 18 FF
FD C3 D9 E7 FB E7 FB E7 FB E7 FB E7 FD C3 C1 FF
FE E1 EC F3 FD F3 FD F3 FD 33 FD 33 7B 87 87 FF
E6 19 6D 93 FB 87 F7 8F FB 87 ED 93 E6 19 00 FF
F7 0F 67 9F EF 9F EF 9F EF 9F EE 9D FE 01 00 FF
C6 39 EE 11 FE 01 D6 29 D6 29 C6 39 DE 39 18 FF
EE 19 EE 19 F6 09 DE 21 CE 31 CE 31 D6 39 10 FF
FD 83 C6 39 DE 39 DE 39 DE 39 D6 39 7D 83 83 FF
FC 03 66 99 EE 99 FC 83 E0 9F EF 9F F7 0F 07 FF
FD 83 C6 39 DE 39 DE 39 D6 29 CE 31 FD 83 86 F9
FC 03 66 99 EE 99 FC 83 E6 99 EE 99 EE 19 08 FF
7D 83 C6 39 DC 3F 7C 83 06 F9 DE 39 7C 83 80 FF
FE 01 B2 4D 30 CF F3 CF F3 CF F3 CF F9 87 81 FF
EE 19 6E 99 EE 99 EE 99 EE 99 EE 99 BC C3 80 FF
DE 39 DE 39 5C BB 6D 93 ED 93 A9 D7 BB C7 C3 FF
D6 29 D6 29 D6 29 FE 01 EE 11 C6 39 D6 39 38 FF
DE 39 C6 39 6D 93 B9 C7 ED 93 C6 39 D6 39 18 FF
DE 39 C6 39 6D 93 B9 C7 F3 CF F3 CF F9 87 81 FF
FE 01 8C 73 39 E7 F3 CF E7 9F CE 3D FE 01 00 FF
FB E7 F3 CF F7 CF F7 CF F7 CF F7 CF DB E7 C3 FF
FF 3F 7F 9F BF CF DF E7 EF F3 F6 F9 FA FD FC FF
F3 CF DB E7 FB E7 FB E7 FB E7 FB E7 F3 CF C3 FF
FF FF FF FF FF FF FF FF FF FF FB E7 FB E7 E3 FF
FF FF FF FF FF FF FB 87 FB 87 83 FF FF FF FF FF
compressed gfx $84:B727 from rom:
00 04 FD FF 00 C0 3F 00 60 1F 5F BF F5 7F 15 60
FF FE CF F8 00 04 F8 07 F2 FC FA 35 60 16 70 16
30 36 70 3A B0 FF 5F BF 2F DF 1F 60 00 3F F8 00
A0 FE 0F 56 70 F4 FA F8 04 00 01 F8 00 D0 9F F0
B1 F0 C3 F0 D5 F0 E7 F0 00 30 DF 7C 83 C6 39 DE
03 41 7C 83 3F 01 FF DB E7 FB C7 10 01 15 21 F7
FD C3 C1 FF 20 1E F9 BC C3 7F 60 9F DE 39 FE 01
00 FF 20 7F 3E F9 FC C3 C6 F9 FE 0B 01 FF 00 FF
FD E3 FD C3 ED 93 FF CD 33 FE 01 0C F3 FE E1 FF
E0 FF FE 01 C0 3F FC 03 27 06 F9 FE 39 41 00 11
D8 53 01 02 11 FE 3C 11 FE 01 C6 39 2C F3 D9 FF
E7 F7 CF F7 CF FB 87 83 F8 FF 40 80 51 2E 31 DE
39 7E 81 06 2D F9 0A 31 FF FF 78 11 C7 A1 41 00
30 1E A4 51 D7 EF E7 DF 78 11 A2 91 B8 51 F9 CF
00 C0 B8 51 79 87 8D 73 3D CA 75 01 F3 A5 21 C7
20 00 F2 CF C7 FB BF C7 AB D7 ED 93 7D 01 21 18
FF FF FD 03 66 99 EE 99 FC 4B 83 E6 23 12 03 3E
01 61 21 DF 35 02 3E 0A 11 81 FF FF 03 67 23 02
44 32 F7 FD 03 03 4F 01 62 9D EC 9F 3F FB 87 E3
9F EE 9D 2C 11 50 72 DF EF 9F F7 0F 07 2F 42 DE
31 3D D6 03 01 7D 83 83 FF 04 31 70 11 CC 04 11
1E 02 C3 D9 15 41 1A 31 FE E1 F7 EC F3 FD A3 12
33 FD 33 7B FF 87 87 FF E6 19 6D 93 FB FF 87 F7
8F FB 87 ED 93 E6 3F 19 00 FF F7 0F 67 69 02 C4
12 FE 5A 32 C6 39 EE 11 FE 01 D6 F7 29 D6 29 1A
32 EE 19 EE 19 7F F6 09 DE 21 CE 31 CE 77 02 A9
10 2F 22 04 31 D6 7B 22 FC 21 42 E0 FC 69 82 F6
22 29 CE 31 FD 83 86 79 F9 00 53 28 12 EE 19 08
FF 18 12 AF DC 3F 7C 83 98 31 80 4F 01 B2 EF 4D
30 CF F3 45 23 F9 87 81 BA DF 02 6E 43 62 BC C3
80 7F 22 5C FF BB 6D 93 ED 93 A9 D7 BB F7 C7 C3
FF D6 12 D6 29 FE 01 D7 EE 11 C6 F9 02 38 7F 02
C6 39 BF 6D 93 B9 C7 ED 93 7A 13 18 7C 7F 63 48
53 FE 01 8C 73 39 F7 01 2F E7 9F CE 3D 2C 11 FB
F7 01 C0 51 F7 DB E7 C3 0E 00 7F 9F BF CF FF DF
E7 EF F3 F6 F9 FA FD 67 FC FF F3 BB 03 94 52 F3
CF BE 03 14 00 60 16 11 E3 E3 53 87 7C 11 79 10
compressed gfx $84:B727 from tool that I posted below:
http://www.mediafire.com/file/x3i0nc0y92pgedd/BlackthorneCompressorTest2.rar/file00 04 FD FF 00 C0 3F 00 60 1F 5F BF F5 7F 15 60
FF FE CF F8 00 04 F8 07 F2 FC FA 35 60 16 70 16
30 36 70 3A B0 FF 5F BF 2F DF 1F 60 00 3F F8 00
A0 FE 0F 36 70 F4 FA F8 04 00 01 F8 00 D0 9F F0
B1 F0 C3 F0 D5 F0 E7 F0 00 30 DF 7C 83 C6 39 DE
03 41 7C 83 3F 01 FF DB E7 FB C7 10 01 15 21 F7
FD C3 C1 FF 20 1E F9 BC C3 7F 60 9F DE 39 FE 01
00 FF 20 7F 3E F9 FC C3 C6 F9 FE 0B 01 FF 00 FF
FD E3 FD C3 ED 93 FF CD 33 FE 01 0C F3 FE E1 FF
E0 FF FE 01 C0 3F FC 03 27 06 F9 FE 39 41 00 11
D8 53 01 02 11 FE 3C 11 FE 01 C6 39 2C F3 D9 FF
E7 F7 CF F7 CF FB 87 83 F8 FF 40 80 51 2E 31 DE
39 7E 81 06 2D F9 0A 31 FF FF 78 11 C7 A1 41 00
30 1E A4 51 D7 EF E7 DF 78 11 A2 91 B8 51 F9 CF
00 C0 B8 51 79 87 8D 73 3D CA 75 01 F3 A5 21 C7
20 00 F2 CF C7 FB BF C7 AB D7 ED 93 7D 01 21 18
FF FF FD 03 66 99 EE 99 FC 4B 83 E6 23 12 03 3E
01 61 21 DF 35 02 3E 0A 11 81 FF FF 03 67 23 02
44 32 F7 FD 03 03 4F 01 62 9D EC 9F 3F FB 87 E3
9F EE 9D 2C 11 50 72 DF EF 9F F7 0F 07 2F 42 DE
31 3D D6 03 01 7D 83 83 FF 04 31 70 11 CC 04 11
1E 02 C3 D9 15 41 1A 31 FE E1 F7 EC F3 FD A3 12
33 FD 33 7B FF 87 87 FF E6 19 6D 93 FB FF 87 F7
8F FB 87 ED 93 E6 3F 19 00 FF F7 0F 67 69 02 C4
12 FE 5A 32 C6 39 EE 11 FE 01 D6 F7 29 D6 29 1A
32 EE 19 EE 19 7F F6 09 DE 21 CE 31 CE 77 02 A9
10 2F 22 04 31 D6 7B 22 FC 21 42 E0 FC 69 82 F6
22 29 CE 31 FD 83 86 79 F9 00 53 28 12 EE 19 08
FF 18 12 AF DC 3F 7C 83 98 31 80 4F 01 B2 EF 4D
30 CF F3 45 23 F9 87 81 BA DF 02 6E 43 62 BC C3
80 7F 22 5C FF BB 6D 93 ED 93 A9 D7 BB F7 C7 C3
FF D6 12 D6 29 FE 01 D7 EE 11 C6 F9 02 38 7F 02
C6 39 BF 6D 93 B9 C7 ED 93 7A 13 18 7C 7F 63 48
53 FE 01 8C 73 39 F7 01 2F E7 9F CE 3D 2C 11 FB
F7 01 C0 51 F7 DB E7 C3 0E 00 7F 9F BF CF FF DF
E7 EF F3 F6 F9 FA FD 67 FC FF F3 BB 03 94 52 F3
CF BE 03 14 00 60 16 11 E3 E3 53 87 7C 11 00 10
In the compressed gfx file that my tool produced I found 2 sliding windows that have different locations than
the original compressed gfx.
1. $0033 in rom = 56 70, $0033 in my tool = 36 70
Both have the same size and same byte pattern but different locations
FA FC FA FC FA FC FA FC FA FC
in rom 7e:4056-7e:405f
in tool 7e:4036- 7e:403f
there is a total of 4 or 5 matches of this group of bytes
2. $024e in rom = 79 10, in my tool $024e = 00 10
Both have the same size and same byte pattern but different locations
FF FF FF FF
in rom 7e:4079-7e:407c
in tool 7e:4000- 7e:4003
there are a lot of matches
I know the sliding windows can start going forward or backwards and if the same size of matching bytes
are found multiple time it could ether go with the 1st time that pair is found or the last time that
pair was found. These 2 windows seem to be somewhere in between. The original compressor could have been
programed to use random locations for the patterns that are the same bytes of the same size but if that
was the case I would think more of the sliding window locations would be different than just 2 of them.
There has to be some sort of pattern that I haven't found yet. I have yet to fully look at the compressed
gfx that are > 0x10000 in size so that might help find the last piece of the puzzle. I'm not sure that
I can spend much more time on this because I have a lot going on right now but I can do a little more next
week.
Test out the tool and let me know if it is compressing the gfx size correctly and if the gfx is decompressed
correctly in rom. The tool still does not compress files larger than 0x10000 but that will be updated when I get
more time.
October 19, 2020, 07:13:49 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
I forgot to mention for the 2 sliding windows that have different locations than
the original compressed gfx I did find something that could be the missing link but
it might just be a coincidence.
For:
1. $0033 in rom = 56 70, $0033 in my tool = 36 70
Both have the same size and same byte pattern but different locations
FA FC FA FC FA FC FA FC FA FC
The pattern for mine is the 1st time it occurs and for the pattern in the
compressed gfx from the rom it is the 5th time it occurs. Well the compression byte
is 0xf8 and the bits are 1111 1000. Bit 2 is the one used for this window and it has
a bit value of 4. Is it a coincidence, idk?
For:
2. $024e in rom = 79 10, in my tool $024e = 00 10
Both have the same size and same byte pattern but different locations
FF FF FF FF
The pattern for mine is the 1st time it occurs and for the pattern in the
compressed gfx from the rom it is the 33rd time it occurs. Well the compression byte
is 0x14 and the bits are 00010100. Bit 5 is the one used for this window and it has
a bit value of 32. Is it a coincidence, idk?
I'm probably over thinking and these two things are just a coincidence but more research
definitely needs to be done. At least it produces the same compression size.