Utilities: Has the ROM hacking community finally found a patching format to call their own?

[RHDNBot]

Update By: Tater Bear

For some time now the ROM hacking community has been plagued with problems and limitations of the IPS format, which is the most popular patching format used by ROM hackers. There have been several attempts to resolve these problems, with the most notable attempts being Ninja and UPS.

Well, byuu, one of the co-developers of the UPS format (meant to be an enhancement to the IPS format), decided to start from scratch and create a patching format that would meet all the primary needs of the ROM hacking community. This format is called BPS and it is fairly robust and offers no file size limits, smaller patch sizes, non linear patching, embedded readme files, and much more.

byuu recently confirmed that the specification has been finalized, after the inclusion of folder/directory patching. byuu stated, "BPS is a done deal. Extremely well vetted, tested against thousands of files. It's not going anywhere". With the final addition of folder patching we can not only patch single files but entire CD directories or PC game folders.

August of this year he released beat which can apply and create BPS files and, as with most of byuu's projects, it is open source and highly portable. Visit the RHDN project page to read all that beat/BPS can offer and take the time to try it out and see if BPS is a good fit for your ROM hacking project.

RHDN Project Page

Relevant Link: (http://byuu.org/programming/beat/)

[Tater Bear]

Doing Delta patches for large files is slow, but byuu noted this as well and wrote "We just need someone who -really- understands delta encoding to help make a more effecient delta algorithm for files > 100MB".

Of note, there is already another separate implementation available. Screwtape has developed one in python and it can be found here. Screwtape has posted his test results as a spreadsheet here and as you can see BPS is capable of some very impressive small file sizes.

If you wish to try writing your own implementation, the spec for BPS can be found on the beat homepage.

[DankPanties]

This is nothing short of astounding.

[Azkadellia]

While it is a nice idea, I don't expect it will catch on. This has been tried before (see: NINJA) and resulted in nothing but drama.

[Garoth Moulinoski]

If I may ask, what kind of drama occurred when NINJA was released?

[Zoinkity]

The python implementation is a bit slower--it is interpretted code after all--but results in as good or frequently better filesizes most of the time.  That's great news if you're writing an editor in python, writing the patch directly from your program without requiring system calls, etc.
(though python could use a full implementation of gzip, since the zlib route doesn't compress as well, even at -9)

You know, there were a lot of problems with Ninja, and honestly one has to wonder why it could patch every other format under the sun including a few that were rather obscure.
The biggest difference is that there isn't a bunch of arcane (and sometimes erroneous) data type detection.  One format works on all the various different ROM formats, or for that matter any kind of file.

It's the only honestly viable delta option besides xdelta, but the difference there is that it isn't obscenely complicated.  That said I don't recommend this for anyone hacking disk consoles.  Stick with xdelta when doing delta patching if you want to use your computer for the next few days.

That more communities haven't picked up xdelta at this point is seriously bizare. 

[Next Gen Cowboy]

If I may ask, what kind of drama occurred when NINJA was released?

None that I remember, a few people myself included lauded it, and touted the praises but it just never seemed to catch on.  A few high profile projects did, and do use it though.

[DaMarsMan]

I liked NINJA but byuu's format looks promising. I like the approach he's taken with his higan/bsnes emulator and I think he has a good vision for this sort of thing.

[Tater Bear]

That said I don't recommend this for anyone hacking disk consoles.  Stick with xdelta when doing delta patching if you want to use your computer for the next few days.

If the CD is treated as one big file (iso or bin), then that is definitely going to be a problem. But if the CD contents are treated as a directory, then it can work well for some systems/games.

Hopefully, down the road a delta knowledgeable programmer can write faster delta code that will allow beat to be feasible for large files. For most ROM hacking projects, beat's implementation is fast enough.

I liked NINJA but byuu's format looks promising.

Good news is that the format is complete, so there are no worries about BPS files made now being incompatible down the road.

I also learned that the Snes9x team is going to add soft patching with BPS files to the emulator.

[Dwedit]

Xdelta is an excellent patching format, the only problems with it are:
*Horrible user interface (even the command line options are cryptic!)
*Requires the patched file to be an unaltered exact match of the original file.

[FAST6191]

I got turned to xdelta out of necessity and have used it ever since (BSDiff could kind of work but in practice has a few limits apparently), I could be persuaded out of it for size reasons but a proper use of options or just going down to file level (unpacking a filesystem with a dedicated tool) works for me. That said I do not have to play in the SNES world much so having to account for headers, interleaved ROM images and such is not something I have to do.

@Dwedit granted I still mess around with http://www.evanjones.ca/software/xdelta-win32.html and most are on http://code.google.com/p/xdelta/ but isn't there the -n command to disable such verifications? I do agree the command line stuff is a bit cryptic though.

Re: Drama and NINJA- I only saw it in passing/was lurking at the time, but I got the impression drama was down to people rather than any deeper philosophical reason.

[Trixar_za]

Anybody mind if I port this to Linux using nimrod? It converts code to C which it compiles. I'll probably add a quick GTK UI on top of it similar to the one in the screenshot. Been looking for something I could contribute to the community.

[Snatcher]

Looks like this would be great for CD based translations. making it easy to patch Playstation or Saturn games mite get more people interested in translating them. 

[KaioShin]

Looks like this would be great for CD based translations. making it easy to patch Playstation or Saturn games mite get more people interested in translating them. 

Check the other posts in the thread, it's horrible for CD based games currently.

[Zoinkity]

Only disc isos, but from what I've been hearing that only applies to earlier CD games.  I honestly assumed most games distroed that way in my earlier post. 
Any game with a file structure (which would include Wii WADs, wouldn't it?) would be okay.  Time to delta compare is dependant on individual filesize; the growth rate is O(n^2), but isn't noticable except when individual files top 100MB.

Ironically, it takes a few minutes to do delta on an N64 ROM, but because individual files are smaller (and files with no differences are skipped) it probably takes less time to do a disc game with filesystem.

[Tater Bear]

Anybody mind if I port this to Linux using nimrod? It converts code to C which it compiles. I'll probably add a quick GTK UI on top of it similar to the one in the screenshot. Been looking for something I could contribute to the community.

I never heard of nimrod before, I think that is an interesting idea. Please share the results with us, when you port it  .

Check the other posts in the thread, it's horrible for CD based games currently.

It is conditional on the CD/system. If the CD contents are treated like a folder structure and it has no files that are bigger than 100mb then it could work. I imagine the biggest problem, for certain systems, would be that the end user need to know how to rebuild the disc.

It is worth noting that beat is only slow for the creation of patches for large files. It can apply those same patches quickly with no problems. So if a ROM hacker wanted to use beat with a slightly large file it would only affect him, not the people that the patch would be distributed to. The slow patch creation on large files is not an issue with BPS, but just with the implementation found in beat.

Only disc isos, but from what I've been hearing that only applies to earlier CD games.  I honestly assumed most games distroed that way in my earlier post.

Earlier CD games could/might work, if the user is expected to rebuild the game disc themselves.

[Lilinda]

I don't really see a point to byuu's continued revisions of his patching format. They basically just suit his uses.

[Tater Bear]

I don't really see a point to byuu's continued revisions of his patching format. They basically just suit his uses.

My understanding is that BPS has only had one revision and that was the addition of folder patching. He has stated multiple times that the spec is finalized, so I don't follow why you are saying "continued revisions".

I have no use for folder patching, but all the other features of BPS suit me just fine. I think most ROM hackers could make use of the other features as well (The most important one being stored checksums).

I am sure there are many people that can make use of folder patching. I can see it streamlining the mod process for old PC games. I recently updated Thief 2 and Oni for play on my windows 7 OS. The mods included modifications to the resources and executables of the games (To allow increased resolutions, new HD graphics, proper function in newer versions of windows, and other neat features), which goes without saying that multiple things needed modification/patching in the games directories. I think it is fairly narrow-minded of you to dismiss this feature just because you personally don't have a use for it. Your shortsightedness makes me seriously want to make a new installation of Thief 2 and created a BPS patch of the directory with all the mods included... Honestly, saying that it just suit his uses, comes off as if though you have some personal issues with him and I would like to keep that unnecessary drama out of a serious discussion of the merits of this patching format for this community. Discuss BPS or beat, but not byuu.

[Gideon Zhi]

The issue with this isn't going to be one of perceived utility, it's going to be one of adoption, and that on three fronts: the users, the hackers, and the emulator authors. Given a lot of the issues with IPS I've been strongly considering distributing future patches in multiple formats, but once release time rolls around I'm usually completely exhausted and forget :/ But barring that, most of my users are going to be like "wtf is this? what do I do with this?" And as nice as bsnes/higan is, it's still the linux of emulation - great for power users (or purists, in this case) but not widely adopted for everyday use. 9x and ZSNES are both updated sporadically at best. Inclusion of the format into other emulators as an autopatch solution would be a major help for adoption, but I expect the vast majority of my users to want to stick with what they're comfortable with, and I'm not about to force them into a change they don't want. Options are always nice! It's when you're not given any choice that people start getting up in arms.

(Note for the slow of wit: link included as an example only! The comparison I'm drawing here would be one where I stop distributing IPS and migrate wholly to any given new standard.)

[Tater Bear]

I don't use soft patching on my stuff as I tend to play them on the systems directly. I don't think most of the emulators I use even allow soft patching (I have never even tried). Snes9x is adding soft patching with BPS, but that only covers the SNES.  Embedding a soft-patcher is possible with this format and that would be a universal solution for any system and wouldn't require that the end user change anything (I think). I know little about the subject of soft patching, as I don't use it. I would be interested in how many regular users soft patch their ROMs in an emulator instead of hard patching them. I think of soft patching as something only "purists" would use.

To be honest, I go with the latest/greatest thing and I think most people are like that when it comes to technology. Just look at how computer software and hardware gets away with constant updates even though there was nothing technically wrong with the previous stuff. The average user, I would think, would update their IPS patcher to a BPS patcher if it was simply a slightly newer technology with only minor improvements. In this case, BPS is far more than a slight improvement. From the end user's perspective, they would see smaller patches and would quickly know if the patch was incorrectly applied. Something that no IPS patch does for the average user.

I guess, what I am trying to say is that I don't see the average user caring about whether it is an IPS/BPS patch. They just want to play the game and if they have to download a IPS/BPS patcher they will. The people that should be caring about which patching format to use is us, the creators of the patches. The average user will use what ever we use, so we might as well give them the best quality patches we can offer.