My days working on RS3 are long long ago but, if I remember correctly, the dialogue blocks are loaded directly from ROM. Do you then individually compress each dialogue chunk?
Yes, I compress each chunk separatelly. The main dialog text is divided in banks 0xFA, 0xFB and 0xFC which have its own pointer table (1024 for 0xFA and 0xFB, 768 for 0xFC); then I check the most common pair of bytes (not only chars) and create a dictionary filled with:
* 0x000 to 0x0FF -> chars and commands
* 0x100 -> END marker
* 0x101 to 0x7FF -> the most common pair of bytes up to the optimum combination of dicionary size and compression ratio.
The next step is to "compress" the script using this dictionary. This saves 115680 bytes in the script length but increases 7164 bytes because of dictionary size.
Finally, I compress the script using Huffman, with 12 bit length entries. The MSB tells if the tree entry is a "leaf" or a "branch". This reduces each bank's size roughly in a half (53% of original size).
All those steps are made in a dialog basis, i.e. from "END" marker to "END" marker.
Besides, I analyze the miscellaneous text ( items, armors, weapons, magic...) and fill the extra room in the hex table with most common couple of chars (depending on the language which is going to be translated into, the tool calculates the number of hex bytes needed for characters and the remaining up to 0xFF are filled) so those texts are compressed in a different way.
I guess I won't be resurrecting my work on RS3. It looks like its in very safe hands.