Dictionnary compression - how to create the dictionnary?

[Ridculle]

Hello everyone.

I'm french and I've been working on Samurai Pizza Cats translations (NES version).
For now, the script has been extracted and translated from japanese to french.
Now, I have to reinsert the translation, and this is where things become complicated.

Samurai Pizza Cats seems to use dictionnary compression for its script. The thing is I can't figure out how to create a new dictionnary from my own translated script.
To be more precise, what i want to find is an algorithm or a document that will help me to create a words list for my dictionnary that is specific to my script.

I hope you guys could help me or redirect me to the answer.

Regards.

[Tito Santana]

You want to find out which words in your script are used the most so that you can change them with the ones already in the game's dictionary, right?

If I understand right, then WordCounter can do what you want.

[Ridculle]

Thanks for your answer Tito Santana.

Indeed I want to find out which words are used the most. However when I say "word" I mean group of x characters (not necessarily group of characters between two blank characters).
My goal is to optimize my script compression with the best dictionnary possible. It means that items that will be within it don't have to be complete.

[Tito Santana]

Maybe this utility will do it?

[Ridculle]

This tools seems very promising, unfortunately the link is broken :S

[Tito Santana]

The 'Download This File' link? If so your refferals may be messed up because the link works ok for me.

[Ridculle]

In fact, when i try to download a file from Romhacking.NET server, i get 403 error (You don't have permission to access /utils/Martial.rar on this server.).
... and I have the same error when i try to displays pictures (which explains why i don't see logos anymore).

[Tito Santana]

It is your referrers. What browser do you use?

[Ridculle]

Iceweasel 3.5.16

[Klarth]

Use ScriptCrunch with substring settings.  It'll take a few minutes to process a large script, but you'll get what you're looking for.  PM me if you need help on how to use it.  Basically, edit one of the included configuration files and run it from the command line.

[Tito Santana]

In about:config what is your network.http.sendRefererHeader set to?

[Ridculle]

It is set to 0.

I set it to 2 and both images/links came back
July 15, 2011, 12:50:41 PM - (Auto Merged - Double Posts are not allowed before 7 days.)
Thank you both for your help!

Now I'm gonna working on finishing this bloody project!

[Gil Galad]

Here is an example to look at for ScriptCrunch. It's what I used for Dracula II since it has dictionary and I also added DTE.

Code Select Expand


[DTE]
; DTE at [80-BF]
DTEEnable=true
DTEBegin=80
DTEEnd=BF

[Dictionary]
; Dictionary enabled
DictEnable=true
DictBegin=FC00
DictEnd=FC7F
DictEntrySize=2
DictUseWholeWords=true
DictMinString=3
DictMaxString=10

[Script]
ScriptFile=Main_Test.txt
LineComments=//
BlockComments=/*,*/
LineStartComments=#
IgnoreBlocks=<,>

[Table]
OutputTable=dracula_eo.tbl
InputTable=dracula_e.tbl
OutputFrequencyTable=freq_1.txt
PerformSizeAnalysis=true

[Insert]
; DTE doesn't need the extra features for Dictionary insertion
InsertFile=
EndTag=
FixedStringLen=0
PadString=


I'm not using the bottom insert function but using all the rest. Using whole words for dictionary. Dictionary in this game is two bytes; FCxx. First byte FC says it's a dictionary entry and the number afterwards is which dictionary entry it is.