News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: [SOLVED] Something like xdelta that can stack patches?  (Read 1880 times)

theflyingzamboni

  • Jr. Member
  • **
  • Posts: 91
    • View Profile
[SOLVED] Something like xdelta that can stack patches?
« on: February 05, 2019, 09:33:55 pm »
SOLVED: Apparently xdelta could do that the whole time. D'oh!

I've been working on various mods for The Legend of Dragoon and created my own sort of Frankensteined patcher for the game. Right now it uses xdelta3 to create and apply diff patches to small component files, which has worked fine so far. But I've reached a point where I have two mods that need to patch the same file, and there's no way to do that with xdelta without going down the road of multiple patch versions. And that way lies madness and terrible user experience. Also, one of these mods expands the sizes of some of these files, so simple offset replacement won't work.

So is there a patch format and command line patching utility that allows for:
  • insertions/deletions and changing file size like xdelta
  • patch stacking

As an added bonus, if possible, I would like to be able test the patches before applying them to make sure they aren't going to modify the same bytes. So if someone knows of a patch format that fits these requirements, I'd also be interested in the specifications and how they could be compared.
« Last Edit: February 07, 2019, 10:16:10 am by theflyingzamboni »
ROM wasn't hacked in a day.

FAST6191

  • Hero Member
  • *****
  • Posts: 2699
    • View Profile
Re: Something like xdelta that can stack patches?
« Reply #1 on: February 06, 2019, 02:11:35 am »
"to create and apply diff patches to small component files"
The file/file system aware approach is the method most would use to get to your position. Usually in the form of a batch file, something that can rip files from a given system's ISO/ROM and then return it after patching.
You can drop down a level lower and do pointer, or possibly content, aware stuff if you must (though you will likely be coding it yourself). You usually see this for high level game editing tools but no reason you could not do it here.

Stacking theoretically dynamic/size altering patches though is mathematically tricky, even more so when you consider most such dynamic/relocation scans are computers doing something like a compression algorithm and not humans assigning meaning to sections of code.
If one is in place then you can try various things, or have various pathways (if the dynamic patch is to change but something else is static and the dynamic stuff does stuff after the static section you could do an "if [hash value] then [blah] else..." type setup, where [hash value] presumably either the base file or a known hack).
I have combined xdelta and IPS (or indeed another xdelta working from a technically static scenario) before like this.

"to make sure they aren't going to modify the same bytes"
Which will work until you repoint something and thus render something else redundant. If you are dealing with pointers that is not even a rare case. If you know you are not going to do this and thus have a limited case then so be it.

"[multiple patch versions] And that way lies madness and terrible user experience"
Other than download size possibly making it tricky make a sensible UI and you should be OK. Make sure to write a nice release notes for hackers down the line, or yourself if you care to revisit some years later, so they can know what files do what or what they might want to look at.

theflyingzamboni

  • Jr. Member
  • **
  • Posts: 91
    • View Profile
Re: Something like xdelta that can stack patches?
« Reply #2 on: February 06, 2019, 11:33:08 am »
"to create and apply diff patches to small component files"
The file/file system aware approach is the method most would use to get to your position. Usually in the form of a batch file, something that can rip files from a given system's ISO/ROM and then return it after patching.
You can drop down a level lower and do pointer, or possibly content, aware stuff if you must (though you will likely be coding it yourself). You usually see this for high level game editing tools but no reason you could not do it here.
This is what the patcher I wrote does. Other than STR and XA files, the game has two main types of data- and code-containing files: MRG files and BPE files. The MRG files are composed of a bunch of smaller data files indexed by an LBA table at the head of the MRG file. Some MRG files are composed of MRG files themselves, up to a few layers deep. Since I can just change the address and file size in the LBA table, these are the files I can alter the size of if I need to. In my case, it's just text overflow, rather than code.

The BPE files are byte-pair encoded by block, usually 2048 bytes per decompressed block, sometimes less. When I mod these, I decompress only the blocks I'm interested in, then recompress them. Due to where many of these files are loaded in RAM, I don't change their sizes.

So the process I use is: extract game files from disc -> decompress blocks/extract subfiles to smallest possible LBA-table-referenced unit -> patch -> recompress/reinsert subfiles -> insert game files into disc

Stacking theoretically dynamic/size altering patches though is mathematically tricky, even more so when you consider most such dynamic/relocation scans are computers doing something like a compression algorithm and not humans assigning meaning to sections of code.
I think the BPE's are the only ones where I need to stack patches, so I actually doubt I'll need to stack size-altering patches.

What I need is something that can potentially stack static patches or change file size, but does not have to be able to do both at the same time.

"to make sure they aren't going to modify the same bytes"
Which will work until you repoint something and thus render something else redundant. If you are dealing with pointers that is not even a rare case. If you know you are not going to do this and thus have a limited case then so be it.
I consider this scenario unlikely, if I'm understanding you correctly.
ROM wasn't hacked in a day.

STARWIN

  • Sr. Member
  • ****
  • Posts: 449
    • View Profile
Re: Something like xdelta that can stack patches?
« Reply #3 on: February 06, 2019, 04:12:18 pm »
how about a custom patcher that hides the complexity from the end users?

theflyingzamboni

  • Jr. Member
  • **
  • Posts: 91
    • View Profile
Re: Something like xdelta that can stack patches?
« Reply #4 on: February 06, 2019, 05:22:14 pm »
how about a custom patcher that hides the complexity from the end users?
I'm not quite sure what you mean?

If you're talking about a utility to handle the patch application process, I've already created one. Just requires the user to specify some directory paths, and handles all the file extraction/insertion/decompression/compression/patching behind the scenes.

I'm just hoping to find a patch format that fits my needs a little better than xdelta to use with it. I get that there are good reasons for file validation when patching, but I'm already running into situations where I've got bytes I need to change for different mods in close proximity.

In this specific instance, I have an inventory capacity mod that alters a couple values in a single 2048-byte decompressed block, but that same block also contains a couple pointers for my text mod. I'm hoping there's a patch format that lets a user stack these two sets of changes, which isn't possible if there's checksum validation, because once one is applied, the other will fail.

If you're suggesting that I design my own alternate patch format as an alternative to xdelta, that might be a little complex for me, and I'm trying to avoid that.
ROM wasn't hacked in a day.

PhOeNiX

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Re: Something like xdelta that can stack patches?
« Reply #5 on: February 07, 2019, 03:52:38 am »
If checksum validation is your only problem, xdelta allows you to disable it completely.

theflyingzamboni

  • Jr. Member
  • **
  • Posts: 91
    • View Profile
Re: Something like xdelta that can stack patches?
« Reply #6 on: February 07, 2019, 10:14:46 am »
If checksum validation is your only problem, xdelta allows you to disable it completely.
:banghead: :banghead: :banghead:

I dunno how many times I've looked at the option list for xdelta, and somehow I never noticed that was there. The patches stack fine with that switch set. Thank you, that's exactly what I needed to know.  :cookie:
ROM wasn't hacked in a day.

STARWIN

  • Sr. Member
  • ****
  • Posts: 449
    • View Profile
Re: [SOLVED] Something like xdelta that can stack patches?
« Reply #7 on: February 07, 2019, 12:25:01 pm »
i was basically suggesting you to create multiple patches for those cases, but let the utility hide that from the end user.

perhaps disabling validation will work for your use cases, but i believe there is always the risk of one of the patches referring to a region that another patch alters.

theflyingzamboni

  • Jr. Member
  • **
  • Posts: 91
    • View Profile
Re: [SOLVED] Something like xdelta that can stack patches?
« Reply #8 on: February 07, 2019, 01:34:43 pm »
i was basically suggesting you to create multiple patches for those cases, but let the utility hide that from the end user.

perhaps disabling validation will work for your use cases, but i believe there is always the risk of one of the patches referring to a region that another patch alters.

Ah, I see. Yeah, that was one thought I had, but I'm really trying to avoid the multiple patch version thing. That can easily get out of hand if you have to add, say, a third overlapping patch. Suddenly you need a whole lot of different versions.

And yes, I agree there are definite risks, but right now, I think they're worth it. My intent is to create a system for modding this game that other modders can use, which I think is feasible, considering there are only two of us actively developing mods for LoD. Getting new people to utilize a specific system is easier than asking vets to change their ways. As part of this, I can try to encourage modders to be fully aware of what they are changing and share that info, so other modders will be able to report any compatibility users in their documentation. I'll probably try to keep a compatibility masterlist available somewhere as well.

So right now, the patcher stops itself from running if it detects overlaps in the patch list. I'm going to change that to a warning about compatibility, instead. Hopefully, between that and good documentation, the pitfalls can be avoided, if they're ever encountered at all. Perhaps I can have the patcher run a simple version check on the clean disc images first, just to keeps some manner of safeguard in place.
ROM wasn't hacked in a day.