[Disch]

We all know IPS sucks.  But it's kind of like a "it's been here so long it'll never go away" situation.

So rather than trying to get everyone to switch to a new format, why don't we fix arguably the biggest problem with IPS?  That is... the inability to tell whether or not the patch is being applied to the proper ROM?


NOTE:   I'VE REVISED THIS FORMAT -- IGNORE THE FORMAT IN THIS POST AND LOOK AT THE ONE HERE:

http://www.romhacking.net/forum/index.php/topic,21580.msg302018.html#msg302018



current format:

Code: [Select]

5 byte header =     'PATCH'

{
    3 bytes =       offset
    2 bytes =       length of block
    N bytes =       block data
}  <- repeat block X times

3 byte footer =     'EOF'


My proposal:

Code: [Select]

5 byte header =     'PATCH'

{
    3 bytes =       0
    2 bytes =       length of block
    N bytes =       meta info block  (see below)
}  <- repeat block Y times

{
    3 bytes =       offset
    2 bytes =       length of block
    N bytes =       block data
}  <- repeat block X times

3 byte footer =     'EOF'



;;;;;;;;;;;;;;;;;;;;;;;;;;
meta info blocks:

'PATCH_CRC4_****'       = **** is the 4-byte CRC32 value of the original target file (binary, big endian)
'PATCH_SIZE_****'       = **** is the size of the original target file  (binary, big endian)
'PATCH_NAME_****'       = **** is a non-null terminated UTF-8 string indicating a casual name for the target file.  IE:  "Legend of Zelda" ... or "Super Metroid (no header)"
'PATCH_FILE_****'       = **** is a non-null terminated UTF-8 string indicating a common 'good-list' file name.  IE:  "Legend of Zelda, The (U) (PRG1).nes"

   ...  possible future ones if needed, but above should cover most cases well enough

For CRC & SIZE fields, each meta block indicates a 'valid' file.  So if there are multiple CRC blocks, a ROM with any of the given CRCs is considered "good".  Ditto for SIZE.  If both CRC and SIZE blocks are provided, ROM must match at least 1 CRC and at least 1 SIZE for it to be "good".

NAME and FILE fields are more suggestions than requirements, and are more for a GUI patcher to relay information to the user so they're not asking themselves "wtf am I supposed to do with this patch?"

Note:  SIZE field can be used by a smart patcher to identify headered/headerless SNES ROMs and adjust patch offsets appropriately so patch makers/users don't have to worry so much about that.


------------

The big pro to this approach is that it is completely backwards compatible for all files greater than 16 or so bytes.  Since it's backwards compatible, all existing patchers and soft-patching emus will still work, and old IPS patches will not need to be updated

The big con is that it'll need some redundant data.  Since the 'PATCH_XXX' bit will overwrite the first few bytes of the file, a following block will need to restore it.

I won't have time to actually work on a patcher until Wednesday -- but it should be pretty easy to implement.


Thoughts?  Feedback?



EDIT:  added 'NAME'/'FILE' bits.  Elaborated a bit on CRC and SIZE

[Gemini]

If a patcher does effectively use EOF as an end marker (thus ignoring whatever comes after it), you can safely append the same metadata let whatever new patcher detect it and use it for additional checks. Older patches will ignore it and there should be no overwrite issues either at the beginning of a newly patched rom.

[STARWIN]

Backwards compability may be undesirable.

old ips + old patcher = nothing changes, but people may think it checks if the file is good because this patcher does
new ips + old patcher = nothing changes, but people may think it checks if the file is good because this patcher does
old ips + new patcher = people may think it checks if the file is good because this patcher does, even when it actually does not
new ips + new patcher = people need to get a new patcher to get a new feature

You can get around half of it by making the patcher tell if it checked if the file is ok, but it won't cover old patchers / emulators.

maybe people use ips just because the patches are in ips. not because they are incapable of trying a new format or patcher if the instructions said so. in that case there is no need to call the new patch .ips.

[FAST6191]

If a patcher does effectively use EOF as an end marker (thus ignoring whatever comes after it), you can safely append the same metadata let whatever new patcher detect it and use it for additional checks.

That may be a bigger if than you realise. It is a bit better now people have stopped using command line and visual basic but there are still enough awful legacy patchers and emulators with IPS support out there, done wrong it could be seeing us all begging to go back to today when we had the retron5 and android emulators as the main annoyances.
That said most of the issues I saw there were from patches that omitted the EOF rather than things not respecting it, not that anybody tried to append things to them.

What may be better in some regards would be to consider an anime/certain other warez scenes style sticking a crc in square brackets approach and having a suggested patcher scan for that. Might get a bit ugly if you have optional patches that work with each or or hardcoded cheats or something but as long as there is a "do it anyway" option then I guess it would work.
Even without the renaming issue I wonder is there a way to effectively make a dummy/NOP/do nothing command in IPS that even the legacy patchers will respect/ignore/not change things for?

[Disch]

If a patcher does effectively use EOF as an end marker (thus ignoring whatever comes after it)

That's a risk I'm not willing to take.  It'd be very easy to imagine a patcher which doesn't stop until the actual eof.

Backwards compability may be undesirable.

If we're not going to preserve backwards compatability, there's no point in sticking to IPS.  Any number of other patching formats already do this -- the problem is they're not as widely used because IPS is the de-facto standard (at least in NES/SNES/etc era ROM hacks)

old ips + old patcher = nothing changes, but people may think it checks if the file is good because this patcher does
new ips + old patcher = nothing changes, but people may think it checks if the file is good because this patcher does

Nothing can be done about old patchers.  But we can create new patchers to phase old ones out.  This is all about introducing a gradual change to make IPS better.  It's not about making an abrupt change to instantly solve all the problems.  If we want an abrupt change we'd just stop using IPS entirely.

old ips + new patcher = people may think it checks if the file is good because this patcher does, even when it actually does not

New patchers can give a notice or warning if they're applying an 'old' patch.  Not really a big deal.

new ips + new patcher = people need to get a new patcher to get a new feature

New IPS's will work on an old patcher -- that's the whole point with this being backwards compatible.  So there's no NEED to update... you don't gain or lose anything if you stick with an old patcher.  But if you update, you gain extra checking.

maybe people use ips just because the patches are in ips. not because they are incapable of trying a new format or patcher if the instructions said so. in that case there is no need to call the new patch .ips.

There have been superior alternatives to IPS for years.  IPS remains relevant.  It's harder than you think to replace a de-facto standard.

The NES scene has a similar problem with the iNES (.nes) format -- they experimented with a much more descriptive UNIF format but it never caught on because everybody had iNES ROMs.  They're now kicking around the idea of iNES 2.0 which is fully backwards compatible with the current iNES format, but is much more descriptive.  The problem is people aren't really "creating new" ROMs so 2.0 isn't getting much use.

What may be better in some regards would be to consider an anime/certain other warez scenes style sticking a crc in square brackets approach and having a suggested patcher scan for that.

I'm not a fan of storing meta data in filenames.  I suppose it could work, but it'd be harder to work into a patcher (especially a soft patching emulator where the filename is predetermined).

Also I'm not sure I see the benefit.  Is there any benefit of doing this over doing what I proposed above?

Even without the renaming issue I wonder is there a way to effectively make a dummy/NOP/do nothing command in IPS that even the legacy patchers will respect/ignore/not change things for?

Only way to MAYBE do a NOP would be to have a block of length zero, but then you can't embed any metadata because your length is zero  =/

Hence my approach of just overwriting data with the metadata -- then overwriting it again to restore it.

[FAST6191]

Yeah I am not a fan of using file names either. I have seen some truly creative ways of screwing things up done in the past (my favourite was deleting the ROMs after they were selected to regain space... from NES ROMs) and file names is often the first thing people change -- "it had all this weird punctuation at the end of the file name".

I considered the zero length commands but the lack of a payload would pose the main problem. Are there multiple types of zero/NOP do you reckon? If so then rig up some fancy decoder that uses the different zero/NOP as a kind of key. Even if there are only two options you could do a very expanded form of binary. About all that would cost is the capacity of the eventual payload and there are not too many 8 or 16 bit era consoles we deal in that hit the 16 meg range anyway.

Personally I am happy enough to see IPS die, about the only benefit to it right now for me is that it is considerably easier to implement in hardware than almost any other type of patcher. The only effort I would make is to make a truly bombproof implementation of a patcher that handles every edge case we can find and cook up, and in the most permissive license we can think of and possibly in enough languages that embedded hardware through to php users can implement it.

[Disch]

Are there multiple types of zero/NOP do you reckon?

A zero/nop consists of 8 bytes:

3 for the offset (theoretically you might be able to put a payload here)
2 for length (0 to indicate an RLE block)
2 for RLE length (0 to indicate NOP)
1 for RLE byte (theoretically you could have payload here)


With the offset and RLE byte you have 4 bytes... which is enough to hold a CRC and it would effectively be a NOP.  But there are problems with this:

1)  Behavior when the RLE length is zero is undefined.  I would assume patchers would treat this as a NOP since that's most logical, but there's no guarantee

2)  Putting payload in the offset portion is dangerous because a patcher may extend the file size to reach the offset before even realizing the block is a NOP -- this could result in an old patcher creating inflated files.

3)  'EOF' is a reserved value and can't be used in the offset -- and there's no way to ensure that value never appears as the CRC

4)  This is not very extensible anyway.  4 bytes of payload is very little, and the only way to expand this approach would be to string multiple of these NOP blocks together.

Personally I am happy enough to see IPS die,

Same.  But it should have died a decade ago and it still hasn't.  Might as well do what we can to improve it.

[snarfblam]

There is already an extension to the IPS format that appends data after EOF. No reason you can't further extend it. It's a simple 3-byte file length specifier to allow a patch to truncate the ROM. Just to be super-duper-extra-mega-safe, you should include a file truncation entry that specifies the correct file size. That way, a patcher checking for that info won't be thrown off when it encounters something else following EOF. It should be relatively safe to include additional metadata following the truncate entry, giving you backwards compatible patches.

[Disch]

There is already an extension to the IPS format that appends data after EOF. No reason you can't further extend it.

Hrm... that makes me feel a little better about appending data.  If that goes as far back as ancient SNEStool, then it's probably safe.

A downside to appending is you have to parse the entire file to get to the metadata -- and you probably will want the metadata before you care about the actual patch information.

One thing I definitely DON'T want to do is just tack-on a value like that truncation value.  That is not extensible at all... and in a way it kind of shoots this effort in the foot because now if we want to append something at the end of the file we have to include a target file size even if we don't want to.


I'm still leaning towards my original idea of putting all that stuff up front in a kind of block format ... and leaving EOF untouched.

What do you guys think would be better?

[STARWIN]

I think your method is better, assuming that all patchers patch in order (edit: ... and assuming they do a while loop over ips content and not something stupid like file offsets). Though my original post stands otherwise.

[snarfblam]

I think if you really wanted to get technical you could try to make the assertion that overlapping patches invoke undefined behavior. (Every spec I've looked at only explained file structure.) A patcher that is thrown off by unexpected data after EOF, on the other hand, is ill-behaved.

In reality, any patcher that is remotely worth using will tolerate either condition. Neither approach is any harder to implement, and neither has any significant drawback. (Who cares if you have to traverse the entire file to get the checksum? How many IPS patches have you run into that can't be applied virtually instantly?)

It's really a matter of personal preference, and since you're taking the initiative, I'd say it's really your call, Disch.

[AWJ]

I don't understand what you think is being gained by extending IPS instead of using one of the existing patch formats that does support verifying the base file (e.g. BPS).

No matter what format you use, you need new patches that contain the checksum information. Since the patches need to be updated one way or the other, you might as well just update them to BPS.

Likewise, patchers and emulators need to be updated to support the new checksum extension. Since they need to be updated anyway, you might as well update them by adding BPS support instead of yet another new format.

If anything, this idea is the worst of both worlds, because the users who are still stubbornly using ZSNES 1.42 in 2016 will be able to use these extended-IPS patches without seeing any of the actual benefits of the extension.

ETA: Posted before reading the entire thread. tl;dr: what STARWIN said. Also, AFAIK the problems with UNIF and the reason it failed went well beyond "not being iNES".

[Zynk]

What if you want that patch to work with multiple hacks? What should you use?

[STARWIN]

What if you want that patch to work with multiple hacks? What should you use?

It is the reason why I like IPS personally. That it doesn't ensure file identity. This property could be improved too but.. I don't know if this is the discussion for that.

Overall we got ips because the patches are in ips and RHDN doesn't want to alter the patches, so it doesn't seem that anything is going to change anytime soon.

edit: I changed my mind, let's post it: a nice way would be to count the checksum over the old bytes over the patched offsets. This way the patch applies only to files where those particular locations were unchanged, which in various cases rules out bad alignment (header vs no header) and obviously uncompatible other hacks (though some risk remains). Would still fail to apply in some desirable situations where ips would work.

[Disch]

Likewise, patchers and emulators need to be updated to support the new checksum extension. Since they need to be updated anyway, you might as well update them by adding BPS support instead of yet another new format.

That's the thing though, this isn't yet another format.  It's still IPS.  People who are too stubborn to update will still be able to use apply these new patches.

The whole point is that updating will grant you extra functionality -- but if you don't update you don't gain or lose anything.  So updating isn't necessary.  Nobody will actually have to download a new patcher.

The problem with newer patch formats like BPS is familiarity and incompatability.  Aside from there being several conflicting formats, everyone knows and is familiar with IPS -- and they have to be because 99% of NES/SNES/etc ROM hacks are in IPS format.  So everyone in ROM hacking needs an IPS patcher.  It's kind of a pain in the arse to download a hack that's in a format you don't have a patcher for, then you have to hunt down a new patching program.  And yeah there are programs that apply multiple formats -- and you might not say it's that big of a deal... but it kind of is.

(Your choice of words in describing "yet another patch format" is sort of a testament that there isn't any real successor to IPS -- there are just multiple competing formats.  That might be another reason why none of them have really caught on)


Ultimately I mostly agree with you guys.  IPS is stupid and nobody should really use it.  But people still do.  And at this point it's unavoidable.  IPS has too big of an existing library, user base, and software support to ignore.  Simply discarding it and using a different format might be the ideal solution -- but it is not practical -- which is why it hasn't happened and likely will not happen at any point in the foreseeable future.

If anything, this idea is the worst of both worlds, because the users who are still stubbornly using ZSNES 1.42 in 2016 will be able to use these extended-IPS patches without seeing any of the actual benefits of the extension.

That's not the worst of any world -- that's the entire point.

Upgrade to deluxe package?  Get extra bells and whistles!
Don't want to upgrade?  Don't worry, you don't have to, it'll still work.

IMO all software packages should take that approach instead of trying to cram updates down users throats.  Backwards compatability is a wonderful thing.

edit: I changed my mind, let's post it: a nice way would be to count the checksum over the old bytes over the patched offsets. This way the patch applies only to files where those particular locations were unchanged, which in various cases rules out bad alignment (header vs no header) and obviously uncompatible other hacks (though some risk remains). Would still fail to apply in some desirable situations where ips would work.

That's a FANTASTIC idea!  =)

Full-file CRC does block that.  Changed-byte CRC would be much more flexible.

This would mean you'd have to parse the entire IPS before calculating the CRC... so it might make sense to put this on the end of the file instead of the front.

Putting it at the front has a problem I didn't think of before... it hinders patch-layering for patches which change the first couple of bytes (which may be common in NES hacks).

Maybe they could be put at the front... but instead of the offset always being 0, it could be arbitrary -- that way you could put metadata blocks in a region of the file that the patch will be overwriting anyway.  I think I like that better.

It is the reason why I like IPS personally. That it doesn't ensure file identity. This property could be improved too but.. I don't know if this is the discussion for that.

This is absolutely the discussion for that.  I want to iron out all these details before I write up a patcher.  =)

[FAST6191]

Changed CRC would be nice but we do still see nested patches change other hacks, and indeed have encouraged others to nest things rather than use someone else's work. That said if it is the header (or possibly interleaved vs not) issue you are dodging then it could work.

Things other than IPS caught on where IPS falls short (those things with file systems and greater than 16 meg size ROMs/ISOs/whatever).

Anyway other than the reference + edge case thing I mentioned before I think I am in the "let them suffer" camp with regards to people persisting in using IPS.

[Disch]

So I'm seeing a lot of pushback with people saying "don't bother, this wouldn't be worth it"

Which is a legitimate point.  So I added a poll to this thread so see how many people actually think this idea is worth pursuing.

[snarfblam]

I think it would totally be worth it. It doesn't make the current patch format mess any more complicated. Worst case scenario, it doesn't catch on and we end up with a handful of patches that are slightly bigger than they need to be, best case scenario it supersedes LIPS and the world is a better place. Regardless of the final details, I'd definitely incorporate the functionality into my own IPS code.

[tryphon]

^this

[FAST6191]

Though I am not inclined to put in much effort beyond a discussion like this for the would be project I have no objections to a legacy friendly extension to the IPS format, and would applaud the effort at least. To what extent people would take it up I do not know, though I suspect not many for the same reasons we are all still suffering IPS to this day (technically superior formats to IPS having been around for how long now?).

Also relevant XKCD because why not
https://xkcd.com/927/
If this was the next next successor to IPS then I am sure would would all be pointing at the corpses of UPS, Ninja, BPS, DPS, fireflower and whatever else there is out there, the legacy friendly thing being about the only thing to prevent everybody from saying that (though I guess you knew that going in).