News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: What's the best way to package a CD game translation?  (Read 4525 times)

SamIAm

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
What's the best way to package a CD game translation?
« on: July 03, 2015, 03:22:02 am »
My partner and I are about to release a translation of Zeroigar on the PC-FX. I'll start a project thread about it soon, but I have a specific question first.

It appears that everyone is using xdelta or PPF for CD releases, but this is not easy for us. We have hardsubbed all of the FMV, and the changes to the ISO data structure are so extensive that an xdelta patch exceeds 130 megabytes.

On the other hand, believe it or not, the all-original insertion program we've got going, which essentially applies hardsubs to the FMVs itself (it's complicated), and all the script files and everything compress down to about 2 megabytes. Needless to say, this size is vastly preferable for distribution.

However, it doesn't look like people ever really release executables for their translations. I can understand that users wouldn't want to be downloading and running programs willy-nilly, but are executables something the community actually discourages? Is it because of operating system incompatibility? We were thinking about releasing an optional xdelta patch for non-Windows users.

What do you think?

Both the hacking and the translating are finished, by the way. While a few play-testers look over the game, we're making the readme.  This project is essentially done. 8)

FAST6191

  • Hero Member
  • *****
  • Posts: 2626
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #1 on: July 03, 2015, 04:44:05 am »
Patches were always done for the size reduction and not wanting to be distributing directly copyrighted material (which I am guessing the large FMVs would be).

Executable patches do exist. They are discouraged in the case of things like simple IPS patches but in your case you have both technical and legal reasons for wanting to. CD/filesystem based things that otherwise struggle often see this sort of thing.
If you can port your patcher, or make a patcher for the patching format you have made, to python or some other readily multiplatform language then that is great. If you want to release the source to your patcher then that is not as nice as a python thing but still good.

What is the xdelta size for the unchanged FMV but otherwise full patch? I am guessing the FMV uses rudimentary compression, or even something a bit more raw, and you can patch it accordingly. With that you could make a FMV alterer far more simply, if it is used in other games on the system then you have even made a nice tool to help others.

Pennywise

  • Hero Member
  • *****
  • Posts: 2257
  • I'm curious
    • View Profile
    • Yojimbo's Translations
Re: What's the best way to package a CD game translation?
« Reply #2 on: July 03, 2015, 10:57:00 am »
I wouldn't recommend including hardsubbed videos in a translation due to the size of the patch. It's much easier to make a regular patch for everything besides the videos, and make an optional patch people can apply for the videos.

Btw, I've discussed softsubs with the Mednafen author and I was told that it would be possible to make soft subs that would play on the emulator.

Nightcrawler

  • Hero Member
  • *****
  • Posts: 5765
    • View Profile
    • Nightcrawler's Translation Corporation
Re: What's the best way to package a CD game translation?
« Reply #3 on: July 03, 2015, 11:11:15 am »
Yes, it looks like your problem is coming from the FMVs. No patching format is going to be able to do anything about that. Since you hard coded it, you've changed nearly every frame of the video. The resulting full video will be vastly different resulting in a huge chunk of marked different data (especially if compression is involved). The reason your custom patcher is 2 megabytes is because you already know the new data ahead of time and are storing a video editing procedure to apply it. The patcher needs to reverse engineer that with only strict binary difference rules. It will never come up with the same results.

In any event, if you absolutely need hard coded subs, it probably would be more appropriate to include a utility to apply the subs to the videos.
TransCorp - Over 20 years of community dedication.
Dual Orb 2, Wozz, Emerald Dragon, Tenshi No Uta, Glory of Heracles IV SFC/SNES Translations

SamIAm

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #4 on: July 03, 2015, 11:18:09 am »
Yes, it looks like your problem is coming from the FMVs. No patching format is going to be able to do anything about that. Since you hard coded it, you've changed nearly every frame of the video. The resulting full video will be vastly different resulting in a huge chunk of marked different data (especially if compression is involved). The reason your custom patcher is 2 megabytes is because you already know the new data ahead of time and are storing a video editing procedure to apply it. The patcher needs to reverse engineer that with only strict binary difference rules. It will never come up with the same results.

In any event, if you absolutely need hard coded subs, it probably would be more appropriate to include a utility to apply the subs to the videos.

Yep, that's all correct.

I'm just wondering if an executable file is shunned by the community. I should have been clearer about that.

It sounds like it isn't, so I think that's what we're going to go with. It will probably be a root directory with the executable, a readme, and three folders: one to put your image in, one where an output patched image is created, and one with all the data files we need.

Thank you!
« Last Edit: July 03, 2015, 11:30:02 am by SamIAm »

elmer

  • Full Member
  • ***
  • Posts: 122
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #5 on: July 03, 2015, 12:13:24 pm »
If you can port your patcher, or make a patcher for the patching format you have made, to python or some other readily multiplatform language then that is great. If you want to release the source to your patcher then that is not as nice as a python thing but still good.

Haha ... the patching programs are written in C ... once upon a time, that was considered a multiplatform language!

When I have time, I'll rewrite the Windows-specific memory-mapped-file routines to use linux/osx mmap() so that the programs will be truly multi-platform.

Quote
I am guessing the FMV uses rudimentary compression, or even something a bit more raw, and you can patch it accordingly. With that you could make a FMV alterer far more simply, if it is used in other games on the system then you have even made a nice tool to help others.

The game is for the PC-FX which uses a hardware variant of M-JPEG, just like the PlayStation's MDEC (they're not compatible).

I'll be releasing the source to the tools at some point after the translation patch is released.

I've already released my modifications to Mednafen which, IMHO, improve the debugging experience. Rypheca already has those, and I'll be happy to work with her to get them incorporated into the Mednafen mainline if she wants to incorporate them.


I wouldn't recommend including hardsubbed videos in a translation due to the size of the patch. It's much easier to make a regular patch for everything besides the videos, and make an optional patch people can apply for the videos.

Btw, I've discussed softsubs with the Mednafen author and I was told that it would be possible to make soft subs that would play on the emulator.

The game contains 16 1/2 minutes of video ... 11961 frames.

Without being able to follow that story, you're missing most of the particular charm of the game.

Because the PC-FX has no copy-protection, any owner can just burn a translated CD and play the game on real hardware.

That's the goal.

And "yes", it does run in Mednafen, too ... without Mednafen's excellent debugging this translation wouldn't look anywhere near as good ... but I don't want to use any Mednafen-specific tricks like Ryphecha's soft-subs.


Yes, it looks like your problem is coming from the FMVs. No patching format is going to be able to do anything about that. Since you hard coded it, you've changed nearly every frame of the video. The resulting full video will be vastly different resulting in a huge chunk of marked different data (especially if compression is involved). The reason your custom patcher is 2 megabytes is because you already know the new data ahead of time and are storing a video editing procedure to apply it. The patcher needs to reverse engineer that with only strict binary difference rules. It will never come up with the same results.

In any event, if you absolutely need hard coded subs, it probably would be more appropriate to include a utility to apply the subs to the videos.

Every single frame of video is modified to move the original video up and then add the JPEG-compressed subtitles below the bottom ... all without any recompression of the original video, and using "best-quality" JPEG subtitles.

There's no way that xdelta can cope with that. I'm actually surprised that it does as well as it does.


It sounds like it isn't, so I think that's what we're going to go with. It will probably be a root directory with the executable, a readme, and three folders: one to put your image in, one where an output patched image is created, and one with all the data files we need.

As SamIAm says ... as long as we're not violating any community "standards", then that's what we'll do.

I'll just have to get the fully-multiplatform C code written at some point.

Thanks!

Gemini

  • Hero Member
  • *****
  • Posts: 2017
  • 時を越えよう、そして彼女の元に戻ろう
    • View Profile
    • Apple of Eden
Re: What's the best way to package a CD game translation?
« Reply #6 on: July 03, 2015, 12:23:25 pm »
There's no way that xdelta can cope with that. I'm actually surprised that it does as well as it does.
If you enlarge the look-ahead buffer size, it actually does. Most huge xdelta patches are so because people tend to use the default buffer size instead of something bigger to account for major relocation gaps.
I am the lord, you all know my name, now. I got it all: cash, money, and fame.

elmer

  • Full Member
  • ***
  • Posts: 122
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #7 on: July 03, 2015, 12:49:52 pm »
If you enlarge the look-ahead buffer size, it actually does. Most huge xdelta patches are so because people tend to use the default buffer size instead of something bigger to account for major relocation gaps.

Thanks for the advice.

We'll give that a try and see how much it helps.

[EDIT]

BTW ... even though the original video's M-JPEG bitstream isn't altered when moving the video frames up to create room for the subtitles ... the M-JPEG bytestream does have some significant alterations.

The original videos are 193MB ... so an xdelta patch of 130MB does seem to be a bit too large, especially given the opportunities that do exist for re-synching the data.

We'll have to test an xdelta without the subtitles ... we're patching 42 out of the 51 different programs that are on the CD ... that does quickly add up to a bunch of changes.
« Last Edit: July 03, 2015, 01:06:23 pm by elmer »

Pennywise

  • Hero Member
  • *****
  • Posts: 2257
  • I'm curious
    • View Profile
    • Yojimbo's Translations
Re: What's the best way to package a CD game translation?
« Reply #8 on: July 03, 2015, 02:06:30 pm »
Curious as to what your changes are. I still wish Mednafen had a GUI for its debugger. I'd even settle for a setup similar to MEKA.

elmer

  • Full Member
  • ***
  • Posts: 122
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #9 on: July 03, 2015, 03:56:47 pm »
Curious as to what your changes are. I still wish Mednafen had a GUI for its debugger. I'd even settle for a setup similar to MEKA.

Nothing particularly earth-shattering ... mainly putting in a new intermediate sized size font and reorganizing the debugging displays for those of us who don't have the eyes of an 18-year-old anymore.

I want to get source-level debugging (or at least symbolic debugging) in there ... but that's a longer-term goal.

I'd post some screenshots, but it doesn't look like I can attach them ... and I'm a grumpy enough old bastard to refuse to put stuff on 3rd-party "cloud" services that can disappear at any time.

If you have an account on PCEngineFX, you can see screenshots in the thread "CC65 and the PCE".
http://www.pcenginefx.com/forums/index.php?topic=18654.0




Pokeytax

  • Jr. Member
  • **
  • Posts: 56
    • View Profile
    • Cavespeak
Re: What's the best way to package a CD game translation?
« Reply #10 on: July 03, 2015, 04:14:57 pm »
On a similar note, how are people dealing with translation patches for encrypted works (I'm looking at PSP)? Here are what I see as my options:

1) Distribute a pre-patched ISO or practical equivalent (a file containing the entire ISO).

Obviously not optimal.

2) Distribute an executable that decrypts and patches the ISO.

Non-trivial to implement and violates the DMCA (I'm assuming).

3) Distribute a patch for a decrypted version of the file.

Requires the user to have some technical expertise (they may even have to rebuild the disk themselves). Will drive people to just download a patched ISO. But probably the best option?

FAST6191

  • Hero Member
  • *****
  • Posts: 2626
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #11 on: July 03, 2015, 04:44:40 pm »
I might have to look at the video format. Most of my dalliances with game video have been on the PC side of things and the rudiments of some things on the DS.

Anyway on the PSP thing I assume you have an encrypted eboot to contend with. I have not paid attention there for a while (last I saw was people using the PSP for black box/oracle style decryption, and this was quite some time before the PS3 coughed up the PSP keys).

Anyway yeah it is 2 or 3, and for 2 they might have to provide their own keys, if you are playing it by the book.

"Haha ... the patching programs are written in C ... once upon a time, that was considered a multiplatform language!"
I am old enough to have heard "portable assembler" before now.

BlackDog61

  • Hero Member
  • *****
  • Posts: 784
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: What's the best way to package a CD game translation?
« Reply #12 on: July 04, 2015, 11:24:37 am »
If you enlarge the look-ahead buffer size, it actually does. Most huge xdelta patches are so because people tend to use the default buffer size instead of something bigger to account for major relocation gaps.

I'm curious about which exact option is supposed to give best results here.I can see 4 size-related options in the xdelta help:
   -B bytes     source window size
   -W bytes     input window size
   -P size      compression duplicates window
   -I size      instruction buffer size (0 = unlimited)
What do people advise here, for a PSP ISO?
Can we afford to have all 4 encompass 1.4GB? (I think RAM per program islimited, but I don't know what the exact limit typically is.)

Gemini

  • Hero Member
  • *****
  • Posts: 2017
  • 時を越えよう、そして彼女の元に戻ろう
    • View Profile
    • Apple of Eden
Re: What's the best way to package a CD game translation?
« Reply #13 on: July 04, 2015, 11:47:56 am »
I remember it being -b. The latest version of Delta Patcher does it automatically for you, btw. It does miracles even with newly rebuilt ISOs.
I am the lord, you all know my name, now. I got it all: cash, money, and fame.

BlackDog61

  • Hero Member
  • *****
  • Posts: 784
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: What's the best way to package a CD game translation?
« Reply #14 on: July 04, 2015, 12:03:07 pm »
I remember it being -b. The latest version of Delta Patcher does it automatically for you, btw. It does miracles even with newly rebuilt ISOs.
Thanks!
I actually noted (only after posting  ::)) that the xdelta RHDN page contains an example of parameters... for PSP ISO.;D
It does good as I was currently building a ~4MB patch for individual files, but a patchon the ISO seems to be feasible with 1.4MB only. And of course simpler to apply.
Great! Thanks!

elmer

  • Full Member
  • ***
  • Posts: 122
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #15 on: July 04, 2015, 05:07:16 pm »
If you enlarge the look-ahead buffer size, it actually does. Most huge xdelta patches are so because people tend to use the default buffer size instead of something bigger to account for major relocation gaps.

Thanks for the advice.

We'll give that a try and see how much it helps.

Every single frame of video is modified to move the original video up and then add the JPEG-compressed subtitles below the bottom ... all without any recompression of the original video, and using "best-quality" JPEG subtitles.

There's no way that xdelta can cope with that. I'm actually surprised that it does as well as it does.

Wow ... I was really wrong!

Using an enlarged 512MB window (larger than the 350MB .iso size), the complete xdelta file has shrunk down to just 14MB in size.

The xdelta file without the subtitled video is only 40KB.

So I suspect that we'll definitely offer an xdelta version of the patch for those people that want it that way.

wyndcrosser

  • Full Member
  • ***
  • Posts: 199
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #16 on: July 06, 2015, 09:39:23 am »
To avoid the extra work, you can easily post it on an ISO site and create a blog to identify the location. The blog will be used on RHDN. That way you don't need to do any patches and Romhacking.net doesn't get in trouble for ISO related legalities.

VicVergil

  • Hero Member
  • *****
  • Posts: 716
    • View Profile
Re: What's the best way to package a CD game translation?
« Reply #17 on: July 06, 2015, 07:33:12 pm »
To avoid the extra work, you can easily post it on an ISO site and create a blog to identify the location. The blog will be used on RHDN. That way you don't need to do any patches and Romhacking.net doesn't get in trouble for ISO related legalities.

That may not be needed.

Wow ... I was really wrong!
Using an enlarged 512MB window (larger than the 350MB .iso size), the complete xdelta file has shrunk down to just 14MB in size.
The xdelta file without the subtitled video is only 40KB.

I don't think there's anything wrong with releasing such a patch on RHDN, I mean, I doubt much copyrighted data, if useful at all in this state, made it in a 14MB file :P

Pokeytax

  • Jr. Member
  • **
  • Posts: 56
    • View Profile
    • Cavespeak
Re: What's the best way to package a CD game translation?
« Reply #18 on: October 03, 2015, 01:45:19 pm »
Bumping this because I am actually tackling this now. It's a huge pain. I can see why every single PSP patch seems to have its own method of application.

I'm dealing with a large CRIWARE-packed archive that I can't recompress with the tools I have - I'm going to have to unpack, patch a large number of individual files, then repack (without compression).

It sure makes me miss fire-and-forget patching. Is a generalized patching format for PSP possible? It would have to be able to incorporate unpacking, repacking, and possibly decryption so I dunno. Even just covering CRIWARE would entail handling a bunch of outdated versions.