Extended IPS?

[Tomato]

I'm not familiar with all of the many different patching formats that exist today, but I've always felt that IPS could be extended with minimal work, as a sort of EX-IPS or IPS2 maybe? It's probably been done before, but would either of these concepts potentially work for an updated format?

1. Do "bankswitching" to the next 16MB "bank" by specifically targeting address $000000, followed by the payload length of 0x0000

2. OR after the "EOF" string at the end of standard IPS patches, just begin a new run that will target the next "bank"

The idea is that either approach would still technically be compatible with the original IPS format. Of course, I'm assuming these probably already exist or have been considered before, so if this is an ancient idea that's been thrown aside, then please ignore this random post.

[Pennywise]

I can't really address any of your points, but I feel I should add another flaw to the IPS format for you to consider.

IPS can't do what I believe is called delta patching, which basically means if you move any data around, the IPS patch will treat it as one big change and your patch will contain copyrighted data. Hence defeating the purpose of the patch.

[mziab]

Your suggestions would likely conflict with the existing unofficial additions to the spec, as implemented by LunarIPS:

- RLE hunks declare their length as 0
- a ROM can be truncated by specifying the new length after the EOF marker

So the extended format you're suggesting would confuse existing patchers.

One of the problems with IPS is that there is no version byte in the header and there have been unofficial additions like this, leading to some ambiguity in parsing the format.

Rather than rejig IPS, IMO it makes more sense to just use a more modern format like BPS or xdelta, the latter being better if you move a lot of data around. BPS does support delta patches, but xdelta is faster and more proven in this capacity.

[Tomato]

I can't really address any of your points, but I feel I should add another flaw to the IPS format for you to consider.

IPS can't do what I believe is called delta patching, which basically means if you move any data around, the IPS patch will treat it as one big change and your patch will contain copyrighted data. Hence defeating the purpose of the patch.

Yeah, I agree. In my view, there are basically two types of patches, both with slightly different use cases: linear patches like IPS and delta patches like xdelta etc. I dunno if I consider that a flaw in the IPS format though, it's sort of like Cartesian and polar coordinates have their different usefulnesses.

I think what I'm getting at might not be clear though - I'm not proposing a new format (I've seen enough of that on RHDN etc. over the years) but more just of a mental exercise. Like "how feasible is this" and then "what would the solution look like maybe?". I think the thought was prompted by some of the comments I saw after my T-Edition patch went out, combined with my experiences after releasing the Mother 3 patch in UPS. I basically learned that 1. some people LOVE the IPS format, 2. some people will ONLY use the IPS format (they get confused and frustrated otherwise for some reason), and 3. some people expect the IPS format to just always work without understanding that it has technical limitations. Hopefully that makes sense.

Your suggestions would likely conflict with the existing unofficial additions to the spec, as implemented by LunarIPS:

- RLE hunks declare their length as 0
- a ROM can be truncated by specifying the new length after the EOF marker

So the extended format you're suggesting would confuse existing patchers.

One of the problems with IPS is that there is no version byte in the header and there have been unofficial additions like this, leading to some ambiguity in parsing the format.

Rather than rejig IPS, IMO it makes more sense to just use a more modern format like BPS or xdelta, the latter being better if you move a lot of data around. BPS does support delta patches, but xdelta is faster and more proven in this capacity.

That's interesting. So the format's already sort of been co-opted in a way, that's cool and makes it an even more fun challenge. What happens when an RLE chunk is defined as zero in a Lunar IPS patch?

The idea of having a header is interesting though - what if we just inserted our own? Like after the "PATCH" string at the start, we could put like $000001 (this could potentially be the extended IPS format ID number) followed by the RLE length of $0000? Would that still technically be compatible with the Lunar IPS method?

[mziab]

That's interesting. So the format's already sort of been co-opted in a way, that's cool and makes it an even more fun challenge. What happens when an RLE chunk is defined as zero in a Lunar IPS patch?

To expand upon what I wrote earlier: when an IPS hunk has a length of zero, what follows is the RLE data: the number of repetitions (2 bytes) and the data to repeat (1 byte). So if you mean the latter size a.k.a. RLE size, I guess that could be set to zero and the hunk would probably (not guaranteed) be silently ignored, or handled appropriately in your hypothetical patcher. By the way, here is a pretty handy unofficial spec.

The idea of having a header is interesting though - what if we just inserted our own? Like after the "PATCH" string at the start, we could put like $000001 (this could potentially be the extended IPS format ID number) followed by the RLE length of $0000? Would that still technically be compatible with the Lunar IPS method?

Like I mentioned earlier, this all depends on how the patchers actually handle it. But yeah, in theory RLE hunks could possibly be repurposed this way. While the code for Lunar IPS isn't available, it might be worth studying Floating IPS or RomPatcher.js to see the actual behavior or do some testing.

[MysticLord]

I think what I'm getting at might not be clear though - I'm not proposing a new format (I've seen enough of that on RHDN etc. over the years) but more just of a mental exercise. Like "how feasible is this" and then "what would the solution look like maybe?".

Well in my opinion as someone who is totally unqualified to comment on the subject, the ideal patching format would have a set of flags to operate on the rom (when moving existing data around) and to write to the rom (when overwriting) embedded in the patch with the data. Or maybe just a set of commands in one patch subfile, and the data to write in the other subfiles.

I'm not sure how you'd generate patches in a non-involved way though. It would be a programmers tool, the romhacking equivalent to esoteric bash utilities with their own built-in scripting languages. Obviously the tool to write the patch to a rom should be as non-technical-user friendly as possible.

List of commands might be:
* move(start address inclusive, number of bytes)
* move(start address inclusive, end address exclusive) //end address being exclusive seems to be the standard, but I don't like it.
* write(address, byte array identifier)
* delete(start address inclusive, number of bytes)
* delete(Start address inclusive, end address exclusive)

I think the thought was prompted by some of the comments I saw after my T-Edition patch went out, combined with my experiences after releasing the Mother 3 patch in UPS. I basically learned that 1. some people LOVE the IPS format, 2. some people will ONLY use the IPS format (they get confused and frustrated otherwise for some reason), and 3. some people expect the IPS format to just always work without understanding that it has technical limitations. Hopefully that makes sense.

Simplest solution is to find an open-source licensed GUI patching tool for the format you choose and distribute that with your patch.

[KingMike]

List of commands might be:
* move(start address inclusive, number of bytes)
* move(start address inclusive, end address exclusive) //end address being exclusive seems to be the standard, but I don't like it.
* write(address, byte array identifier)
* delete(start address inclusive, number of bytes)
* delete(Start address inclusive, end address exclusive)
Simplest solution is to find an open-source licensed GUI patching tool for the format you choose and distribute that with your patch.

I thought that idea is exactly what BPS was supposed to be (or whichever the last of byuu's patch formats introduced was).

[FAST6191]

This seems like some kind of lipstick on a pig or good money after bad scenario. I know some are desperately wedded to IPS but some of the proposed extensions (see also https://xkcd.com/927/ as you would not be the first, though might be a first in the last few years) here won't appease them either -- all the flash carts, weird combo ROM dumper-emulator setups and clone consoles will still have trouble, and I am not inclined to pay any real attention to the "I want to use the emulator inbuilt patching option" crowd either. If trying to also play nice with non standard implementations (so very few things respect/expect EOF when all is said and done) then going to be even worse.

A while back there was a discussion on getting IPS to support some kind of checksum while still being backwards compatible. Might be something in there to ponder. I was wondering if there are some kind of NOP command you could do, possibly even two of them and with that you can do some binary fun (that or you could note things at given locations and call that binary).

Continue to provide IPS where it is a viable patching option* (some seem in a desperate hurry to abandon it even when working within the 16 meg limit and not relocating data) but limitations mean embracing xdelta, bsdiff or something more file system aware if it is one of those things.

*some in the Wii hacking scene continued to use it for things where they got users to extract files themselves.

[tvtoon]

If the plan is extending IPS, there is a simple way that won't compromise the format: make the address relative instead of absolute (just add from beginning to end). I did it a long time with my patches, when you don't want to use anything else it is the way to go.
****
Notice: you have to give up RLE, so that you can just jump 16MB.

[Tomato]

I won't have time for a bit but I think it'll be fun to test my bankswitching idea with as many IPS patchers I can think of. If it's possible to do $BBBBBB $0000 where B is the bank number for many of them, that'd be neat. I think it'll boil down to how different developers have handled the RLE, I feel like a standard "for (i = 0; i < rleLength; i++)" style statement would still be compatible with this idea, but I can see it being riskier with any sort of "while" statements.

(Again, I'm not proposing a new format, this is just a fun thought exercise)

November 27, 2021, 04:34:15 PM - (Auto Merged - Double Posts are not allowed before 7 days.)

Simplest solution is to find an open-source licensed GUI patching tool for the format you choose and distribute that with your patch.

I vaguely recall that UPS was invented for the Mother 3 patch about halfway through, so we packaged the patch with a patching program for all platforms. We made it nice and clean and simple and reliable, but the number of technical issues + troubleshooting requests resulting from it was still surprising, as well as the amount of complaints about "just release it in IPS". I suddenly remembered how frustrating all that was back then so I was like "hmm, I wonder what it'd have taken to make it into an IPS patch of sorts after all".

[MysticLord]

If the plan is extending IPS, there is a simple way that won't compromise the format: make the address relative instead of absolute (just add from beginning to end). I did it a long time with my patches, when you don't want to use anything else it is the way to go.
****
Notice: you have to give up RLE, so that you can just jump 16MB.

But then someone will try to use a relative IPS patch with an existing IPS patcher.

Wasn't there are a guy posting here a few months ago that made the equivalent of a Traveler's Any-Tool for the SNES? Was that ever open sourced? I think it handled UPS and BPS.