News: 11 March 2016 - Forum Rules, Mobile Version
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia

Author Topic: JPS: web-based IPS patcher - status: released + UPS beta + BPS prototype  (Read 9579 times)

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
JPS (stands for Javascript Patching System) is a web app that applies patches, currently IPS patches. It differs from previous patchers in that it is web based: you use it the same way regardless of your platform, and you don't so much install it as just navigate to an URL:

http://wanderingcoder.net/projects/JPS/ (EDIT: replaced beta URL with public release URL)

The benefits are obvious: someone new to using ROM hacks no longer needs to be told to also download a patcher (typically Lunar IPS)… with additional instructions in case he may be using a non-Windows platform, and with the potential problem of the program no longer running on current systems, especially in the case of the Mac: this happened for PowerPC-compiled patchers, it could happen again. Just point him at JPS, and he will be good to go (ideally, anyway; there are currently limitations, as we'll see). Furthermore, being web-based means you are, by default, using the latest version without any additional effort. And you do not lose the benefit of being able to use the software while offline: after having been loaded once, your browser will be able to reload it while cut off from the Net (of course, you should put it in a bookmark to make sure you can recall the URL without needing to go through this page); in fact, even if I were to let my domain expire and no one could access the URL anymore, your browser would keep the latest version it obtained and allow you to keep using JPS.

All this is made possible by recent advances in browsers, particularly in the domain of file manipulation and offline use support. So recent, in fact, that unfortunately not all browsers have the necessary support, and I am sad to say that, regardless of my efforts, Safari and Internet Explorer are not supported*. Any reasonably recent version of Firefox or Chrome (both of which update automatically anyway), as well as any version of Opera starting with Opera 15 is supported; Safari is supported starting with Technology Preview 19 or Safari 10.1, and Edge starting with version 13 normally.

Also, while I have subjected JPS to about 100 patches downloaded from this site to validate its behavior**, and fixed the most egregious issues (one of which crashed Firefox. Yes, taking down all your other Firefox tabs), there remains a few know issues. The most notable one is that, on Firefox, some patches (just one in the ~100 I tested), once applied, cannot be downloaded: that is, clicking the download link does absolutely nothing***. The only other known limitation is truncation support: it is not implemented yet.****

But even with the remaining issues, it is in good enough state that I now declare it in open beta. Use it, prod it, test it, abuse it, send feedback, or not, go ahead. Obviously, new beta versions will be deployed at the same URL and you will automatically get them.

And for the anecdote, yes, this is what I asked IPS test files for in the other topic. I intend to release its source code once it is out of beta.



*the reason being that neither browser supports downloading generated files (called blobs) with the download attribute. In the case of Safari, I requested the functionality in http://www.openradar.me/23550189; no answer so far the request was marked as a duplicate of a previous request, so at least they are not rejecting the idea outright it has now been implemented, and after additional issues were fixed, Safari can now run JPS starting with technology Preview 19.
**including checking the output is identical to what Lunar IPS gives for the same patch
***logged in the Firefox bug tracker at https://bugzilla.mozilla.org/show_bug.cgi?id=1224870
****what is not a limitation, however, is the lack of support for creating patches; creation of IPS patches will never be supported
« Last Edit: November 09, 2017, 04:38:48 pm by ZPedro »
Tired of IPS patching with a desktop app? Try JPS!

lockecole2

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: JPS: web-based IPS patcher - status: open beta
« Reply #1 on: November 18, 2015, 01:31:15 am »
Ah, this is cool.
I would note that your source code is already revealed, this being JavaScript and you've made no attempt to obfuscate it :)

I also wrote one of these things a few months back: http://sgvpm.org/jspatch
but I lost interest after getting the basic generic patcher working.
That also only does IPS as well though I was working on support for other patch formats.
You just drag and drop the ROM image and the patch and it autopatches for you.

One thing I did though was figure out how to embed a patch and the patcher inside a single HTML file (http://sgvpm.org/jspatch/standalone.html) so that a patch maker could distribute the HTML file and do other things, such as ROM expansion or hash verification before allowing the patch to progress, similar to some customized patching programs sometimes distributed.
With the standalone page, you simply drag and drop the ROM image and it does the rest.

Last I remember I was also working on getting Safari to work properly with it. I believe it worked and downloaded but it spit out a GUID filename rather than a nicely formatted one.

Another thing I was going to work on was using URLs to patches rather than having them be locally be available on disk and make it possible for sites to integrate my script and have the patcher right on the download page.

I also wanted to make it possible to load up popular ROM hash databases to verify ROM images before patching and perhaps allow some common modifications before patching SNES ROM images, such as snipping off the header if it already existed.

What made you want to develop such a thing?

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: open beta
« Reply #2 on: November 18, 2015, 03:55:01 am »
What made you want to develop such a thing?

Good question! Basically two things: Zelda Starring Zelda, at which point I realized the patcher I previously used no longer ran (was built for PowerPC), I used MultiPatch but asked myself "couldn't there be a better way?", and second Gatekeeper and Developer ID.

I'll take a look at your stuff and ideas, very interesting.

November 18, 2015, 06:47:06 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
So, I was able to take a longer look at your program. Glad to see I am not the only one trying to push these ideas of web-based apps for small file manipulation tools! In fact, jspatch feels very… tinkery. Mind you, this is not a knock on it, I very much support writing and distributing tinkerer programs, but it is clear JPS is better tested; for one, jspatch does not support overlapping directives.

Quote
What are you going on about, ZPedro? That's stupid. Why would a patch contain a directive putting data that another directive already specifies?

That is what I thought, too. When reading the IPS specification, I foresaw that possibility, and took it into account more than as a sense of anal-retentive completeness than anything else; especially after I asked for test files having such an oddity but no one pointed me to one, I ended up thinking that such a thing did not exist.

Until I saw one in the wild while testing JPS.

If you check the patch at http://www.romhacking.net/hacks/1501/, it contains a directive that starts at offset 0x0801FF in the output file, even though the previous directive starts at offset 0x0801F3 in the output file with length 0x000D, and so ends at 0x080200. As a result, usual IPS patchers have the first byte of the new directive overwrite the last byte of the previous one (because they simply do fseek() + fwrite() each time, or equivalent), while jspatch simply concatenates the two, so everything starting with that directive is offset one byte (and the file is one byte longer) from the expected result. JPS handles this correctly, that is the same as Lunar IPS and usual patchers do.

Unfortunately, jspatch does not appear to work on Safari at the moment: Safari complains that you try to use File as a constructor, so it does not appear to support that. If you recover how you managed to download blobs using Safari, even with limitations, I would like to see it.

Your planned improvements are interesting. While I do not intend to take JPS in these directions (they are not really in line with the self-contained web app approach), you are more than welcome to explore them, including as a fork of JPS.

And on that subject, while I won't try to stop you if you take and derive from the currently published code (under the BSD license as put in there), I strongly recommend waiting for the source code release, which will have:
- scripts to manage resource versioning, insert the versioned names in the output, generate the AppCache manifest and .htaccess, assemble the app entry point, and deploy the whole thing (it's all tied)
- test vectors for IPS and a test page that uses them to test the IPS engine part of the app.
« Last Edit: November 18, 2015, 06:47:06 pm by ZPedro »
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: open beta
« Reply #3 on: November 25, 2015, 06:13:06 pm »
Small update (same URL): Chrome running on Android is now supported, and I lifted the check that would make JPS refuse to proceed on it (JPS still refuses to proceed on other mobile browsers). You can now try it out, though it's not necessarily straightforward, because Chrome on Android will not offer the option to load a file from the filesystem by default (only a few sources such as captured images are offered); the workaround is to install a file manager, that will act as a file source as far as Chrome is concerned and allow you to pick files from the filesystem.

(Ironically, the built-in browser of my Android handset does support loading a file from the filesystem, but will not run JPS because it does not have the necessary JS API support; so I've only been able to test Chrome.)

EDIT: forgot to add: if you are loading JPS on a browser on which it was previously loaded, you will likely need to reload again in order for changes to be visible. This is because the version that was downloaded the last time is loaded (so that it loads faster), while the download of the new version is occurring in the background.
« Last Edit: November 26, 2015, 05:44:45 am by ZPedro »
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: final beta stage
« Reply #4 on: January 04, 2016, 04:12:52 pm »
Hello, and happy new year!
Big update (same URL; as usual, reload twice to make sure you are actually running it):
  • Support for truncation. I actually deployed that bit in December, but felt it was too small to specifically announce.
  • Progress indication. This was made necessary by the slowness of some supported browsers (namely, Google Chrome, even on a current desktop), where even the spinner was insufficient feedback given that particularly complex patches could take more than 30 seconds to apply (and a few minutes on mobile!); in my opinion, even a complex patch ought to only take a fraction of a second to execute, and not even require a spinner for feedback, and yet here I was, implementing a whole progress bar for that purpose… Anyway, it's done.
  • Support for cancellation. If a patching operation was started by mistake, you can stop it to avoid having to reload the page (or wait for the operation to complete) in order to be able to start the operation you actually wanted.
  • Removed "apply" button. Patching now starts automatically when both a base and patch file are provided; this was made reasonable to provide with the support for cancellation, given that it is easy to start patching by mistake when doing so occurs automatically.

This update was a long time coming not only because of these features, but because I went a few dead-ends as well; most notably I planned to integrate FileSaver.js, and was pretty far in the design, until I realized from the online demos that it would buy me exactly nothing with regard to Safari (it still wouldn't work on it), and not much elsewhere.

For a feature and interface standpoint at least, this is the version I plan to officially release; only fixes for functional issues (inaccessible functionality, wrong output, wrong user feedback, etc.) will happen prior to the release. In other words, we're in final beta testing stage. Thanks in advance for helping test it so that any such issue can be caught prior to the release!
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: final beta stage
« Reply #5 on: February 16, 2016, 04:17:39 pm »
JPS has now been publicly released at http://wanderingcoder.net/projects/JPS/ (the beta URL will keep working, but the version at this URL should be more stable). Its source code is now available on Bitucket, should you want to tinker with it.

I have submitted a news item for the front page.
Tired of IPS patching with a desktop app? Try JPS!

GHANMI

  • Hero Member
  • *****
  • Posts: 671
    • View Profile
Re: JPS: web-based IPS patcher - status: released
« Reply #6 on: February 16, 2016, 04:47:37 pm »
That's really nice to have :)

You may want to consider adding support for UPS patches - just like IPS patches but without the 16MB size restriction. I think some editors make some header to verify the game but it's not exactly needed.

The other main two formats would be xdelta and beat, but their implementation is very different than IPS.

BlackDog61

  • Hero Member
  • *****
  • Posts: 775
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: JPS: web-based IPS patcher - status: released
« Reply #7 on: February 16, 2016, 05:30:05 pm »
That's really nice to have :)

You may want to consider adding support for UPS patches - just like IPS patches but without the 16MB size restriction. I think some editors make some header to verify the game but it's not exactly needed.

The other main two formats would be xdelta and beat, but their implementation is very different than IPS.
I second the support of other formats. Err but I have my hands full, so I guess I'm just stupidly wishing for more presents. 8) Sorry about that! Here, have a cookie:  :cookie:

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released
« Reply #8 on: February 16, 2016, 05:55:47 pm »
That's really nice to have :)

You may want to consider adding support for UPS patches - just like IPS patches but without the 16MB size restriction. I think some editors make some header to verify the game but it's not exactly needed.

The other main two formats would be xdelta and beat, but their implementation is very different than IPS.

Thanks!

I will most definitely consider these formats (as well as BPS), though I can't promise I'll get to them right away, or in that order.

[…] Here, have a cookie:  :cookie:

Thank you!
Tired of IPS patching with a desktop app? Try JPS!

GHANMI

  • Hero Member
  • *****
  • Posts: 671
    • View Profile
Re: JPS: web-based IPS patcher - status: released
« Reply #9 on: February 16, 2016, 06:09:08 pm »
I will most definitely consider these formats (as well as BPS), though I can't promise I'll get to them right away, or in that order.

BPS is beat ;)

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released
« Reply #10 on: February 17, 2016, 06:24:34 pm »
BPS is beat ;)

Oh, OK. I didn't know it came under that name as well, thanks for the info.
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released
« Reply #11 on: April 25, 2016, 04:54:55 pm »
I just implemented application of UPS patches. For now this is not even a beta, merely a public prototype which lacks even basic safety checks (e.g. if the patch specifies that a 1TB file be created, this prototype will happily do so without even asking your input), though it correctly applied the one (1) patch I tried it with (samples are few and far between, the format clearly did not catch on):

http://wanderingcoder.net/projects/JPS-dev/ups-proto/ (warning: no IPS!)

A word of warning: the version of JPS at that URL cannot apply IPS patches, it will not even recognize them; keep using the official release URL to apply IPS patches. Of course, when UPS support is finalized I will update JPS at the release URL to support both formats.
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS prototype
« Reply #12 on: August 07, 2016, 02:19:15 pm »
Breakthrough!

Remember when back in January (a few posts above) I complained about Chrome needing more than 30 seconds to apply some patches? While working on UPS support, I found out why, and so I paused work on UPS to develop a solution, which I applied on the existing version (with only IPS support). I finally managed to work out all issues, and I deployed it in the beta channel, you can test it out (and compare with the current release, which remains slow while the fast version is being beta tested) at:

http://wanderingcoder.net/projects/JPS-dev/beta/

Results are impressive: with Google Chrome, I get a 10x (ten times, 1000%) improvement across the board: a patch which used to take 50 seconds to be applied now takes 5. This goes for Chrome on Android: the same patch which took 200 seconds (on a Galaxy S II) now takes 20. Improvements can be observed with Firefox as well, though given it was already fast they are harder to measure.

I pushed the code to BitBucket, you can check it out if you're interested. https://bitbucket.org/Pierre_Lebeaupin/jps

EDIT: As usual, after the initial load you will have to wait a few seconds then reload the page if you want to be sure you are using the latest version.
« Last Edit: August 08, 2016, 05:57:07 am by ZPedro »
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS prototype
« Reply #13 on: December 08, 2016, 07:10:57 pm »
Small news: I just updated the release version at http://wanderingcoder.net/projects/JPS/ to now allow it to run on (desktop) Safari, following the release of Safari Technology Preview 19, the first version to provide the support necessary for JPS to run (Technology Preview 15 was the first version to ostensibly provide the necessary support, but a serious limitation prevented JPS from working on it, limitation which was finally lifted in Technology Preview 19).

As usual, if nothing seems to have changed after the initial load, you will have to wait a few seconds then reload the page for the update to take effect.

(note that this is still the uncached, slow version; I haven't found the time yet to run the test plan on the version featuring the cache so that I can confidently push it to production… apologies)
Tired of IPS patching with a desktop app? Try JPS!

Ck

  • Jr. Member
  • **
  • Posts: 6
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS prototype
« Reply #14 on: January 23, 2017, 02:45:54 pm »
This is an amazing patching tool!

I was searching for a DOS-compatible BPS patcher and I came across your online development project.. all I can say is a BIG thank you for limited users. I use a Chromebook, so I have been patching NES ROMs using a simple DOS patcher using a DOS emulator within Chrome OS, which is quite a chore to say the least! Your online tool means I can say goodbye to my dinosaur IPS patcher  ;D

I encourage you to continue developing the tool for more formats so that next time, I and others can patch happily away!

Thanks again ZPedro!  :thumbsup:

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS prototype
« Reply #15 on: February 15, 2017, 11:50:06 am »
Thanks again ZPedro!  :thumbsup:

Glad to hear it! I must admit I did not consider this use case when writing JPS (I only recently became aware of web-based emulators that accept user-provided ROMs), but I am glad for that audience, too. Promise, I will make it to other patch formats, it will just require a bit of time.
Tired of IPS patching with a desktop app? Try JPS!

MathUser2929

  • Hero Member
  • *****
  • Posts: 1519
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS prototype
« Reply #16 on: February 15, 2017, 09:13:14 pm »
I've been using XDelta for bigger games. It's also useful for Sonic hacks cause alot of poeple shift data around. IPS patches would just copy all the moved code. XDelta recognizes if somethings been moved and don't copy it.

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS prototype
« Reply #17 on: February 16, 2017, 03:48:05 pm »
I've been using XDelta for bigger games. It's also useful for Sonic hacks cause alot of poeple shift data around. IPS patches would just copy all the moved code. XDelta recognizes if somethings been moved and don't copy it.

Rest assured I am most definitely aware of the importance of VCDiff/XDelta, in particular for Sonic the Hedgehog (GEN) hacks, where encoding the hack as IPS is completely inefficient. For now, I'm working my way up to decoding more progressively complex patch formats: UPS, then BPS/Beat, then VCDiff/XDelta.

February 16, 2017, 08:23:55 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
Important update: I released the accelerated version at the usual URL, http://wanderingcoder.net/projects/JPS/. Now even the most complex patches should apply reasonably fast (in a few seconds at most) in desktop Chrome, and it should be even faster in the other desktop browsers. Performance improved similarly for Chrome for Android.

To ensure you are using the latest version, load the page, wait a bit, then reload it, until you see the donation links (new to this version).

I put the version without the cache at http://wanderingcoder.net/projects/JPS-dev/nocache-demonstrator/, if you want too see for yourself how far JPS has come.

I pushed the code to BitBucket, you can check it out if you're interested. https://bitbucket.org/Pierre_Lebeaupin/jps
« Last Edit: February 16, 2017, 10:10:23 pm by ZPedro »
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS beta
« Reply #18 on: October 08, 2017, 07:15:01 pm »
Long time no see…

I was finally able to resume work on UPS support, and the main hurdle here is properly warning the user before the web app goes ahead and tries to generate a 8GB file (IPS does not have that issue, as this format by construction cannot patch beyond 16MB).

This system is finally implemented, with a user interface I am satisfied with, so I now declare UPS support to be in beta at its own URL:

http://wanderingcoder.net/projects/JPS-dev/ups-beta/

Go ahead, try it out with all your UPS patches, it should behave correctly. If you're curious how it handles UPS patches for large output files but don't have such a patch on hand, you can use http://wanderingcoder.net/projects/JPS-dev/bigoutput.ups (modified from an existing patch; it is not meant to give a meaningful ROM, though if you need one to apply it on, it was originally a patch over Pokemon Red).
Tired of IPS patching with a desktop app? Try JPS!

ZPedro

  • Jr. Member
  • **
  • Posts: 22
    • View Profile
Re: JPS: web-based IPS patcher - status: released + UPS beta
« Reply #19 on: October 20, 2017, 03:29:45 pm »
Good news: I implemented BPS support!

Bad news: with the BPS patch I have the processing ends up crashing the Chrome tab ( https://bugs.chromium.org/p/chromium/issues/detail?id=776572 ), and in Firefox it's not much better; the only way I know BPS patching works is that it works fine in Safari (I have not been able to try Edge yet).

So I am not releasing it yet, not even as a prototype or beta; I want BPS patching to work better in more browsers first. Sorry…
Tired of IPS patching with a desktop app? Try JPS!