News:

11 March 2016 - Forum Rules

Main Menu

Version control for ROMhacking?

Started by NaOH, August 09, 2017, 02:45:24 PM

Previous topic - Next topic

NaOH

I'm interested in how open-source collaboration works with ROMhacks. Obviously, the base ROM can't be distributed if it isn't in the public domain, so that presents some unique challenges for hosting a repository publicly.

Is there any kind of version control system for ROMhacking? I guess I could use Git but there are a couple problems with that. The most obvious one is that Git isn't (to my knowledge) capable of merging binary files (i.e. ROMs). The other of course is that Git would require the base ROM to be in the repo.

Does anybody know what the current solution is for collaborating on ROMhacks, or for open-sourcing a ROMhack? I imagine I could just post, for example, the FCEUX files that I have which document the Castlevania ROMhack I'm working on, but not actually post the ROM. But this seems inelegant.

Thanks!
~NaOH

FAST6191

"open-source collaboration works with ROMhacks"
It doesn't, several have tried but for the most part it does not work and hacking projects stay closed until they are done.

Within a group aiming at translating a game there will probably be milestones people aim for before continuing (dump the whole script, translate the whole script...) or for other things it might be a level by level approach or different aspects to different people (I'll do AI behaviour, you do level mods or graphics or something) and one hacker gets to merge it together after the fact.

Some have tried with git and patching formats, and if you are doing a system with individual files in a filesystem then patching those individually. If the hacking project uses a nice high level tool you can choose parts of the ROM and say don't use these sections as I want them for my stuff.

NaOH

#2
Quote from: FAST6191 on August 09, 2017, 04:32:04 PM
Some have tried with git and patching formats

This sounds like a headache to figure out but also potentially very lucrative. Could you link me to a project that does this? Is there any scheme doing this that is known to work, or any documentation for how to work with this?

Also, would there be interest in the community at large if I were to investigate a good way to do legal open source development / version control with ROMhacks?

snarfblam

On the last of my relatively few collaborative projects, one person maintained the actual ROM in progress, I provided ASM that could be applied via a patching assembler, and a third had his own "post-process" approach that was applied to each revision. Everyone has his own way of doing things and I think it's inevitably a messy process.

Kiyoshi Aman

Revision control on ROM hacking basically requires you to maintain your ASM hacks and whatnot separately from the ROM, since no revision control system I'm aware of handles binary diffs in any useful way.

FAST6191

It was a DS project. If you are thinking git in any useful way then don't, at least outside of the scripts/tools created to help with said same project. The patches were binary blobs, any space gains were merely the internal compression that it might have employed. Basically it was a site for project members to grab tools (always nice if you float between computers) and a copy of the current patch, as well as a bit of insurance if one of the hackers was not inclined to do their own versioning; for years most people here have copied a ROM they might be working on (zipped an old one...) and worked on that so as to avoid having to undo bad work to get back to good, and doubtless you can find a thread of three around here that see some new hacker learn the local equivalent of "backups, everybody forgets to do it once".
Speaking of the DS, and I suppose some other systems, there have been file system aware programs and patching formats. You could smash something like that and git together (it has some support for archive formats and famously started as a file system so eh).



Drenn

I use git for assembly hacking. Example: https://github.com/drenn1/kaizo-of-ages

For this project, I had two git repositories: one for the rom, and one for the assembly patches. I know, that using git on roms isn't ideal - I was concerned about the size ballooning, and merging just doesn't work - but it was still better than doing manual backups, imo.

It worked well for the asm patches, though. The workflow was like this: I edit the ROM with the level editor. My makefile creates an untracked copy of that ROM, and applies the assembly patches on top of that copy.

I just don't feel comfortable working without some kind of backup system, and git is the natural choice for me. Though, this only makes sense if you already use assemblers with romhacking (I find them to be indispensible tools).

(After this I started disassembling Oracle of Ages, which heavily involves git and will facilitate collaboration when complete, but that's another story.)


There are other options for collaboration, though. For my Wario Land 3 editor, I made a feature to import and export levels into files independant from the rom. In theory this would allow people to collaborate more easily, though they would need to keep track of which level uses which resources, to avoid situations like multiple people editing the same tileset.
Zeldahacking.net: The hub for Zelda: Oracle of Ages and Seasons hacking. Join our discord for help & more information!

FAST6191

Another example of git I came across since posting the previous stuff
https://github.com/enler/PokemonTypingDSHacking
Also used for assembly based work.

Gemini

I also use Github for some of my projects, sometimes just to upload scripts (Mizzurna Falls), some other times for the whole toolchain and IDA databases (Secret of Evermore, Spike McFang). It's quite handy and saves the hassle of having multiple people working on the same files, plus it's super effective when you update something and the other team members get a notification of changes.
I am the lord, you all know my name, now. I got it all: cash, money, and fame.

optiroc

I don't see any argument against using a version control system for ROM hacking projects. The issue of not storing a copyrighted binary in an open repository is easily solved by either requiring an environment variable set with the path to the ROM, or simply stating a path for it in your builds script that is .gitignored.

But aside from copyright issues that may become a problem (not only for the source ROM image, but for edited graphical assets, etc, too) of course version control could, and should, be used. If nothing else using a private repo.

NaOH

Today I tried making a solution to how to store a ROMhack in a repo without the actual ROM, without having to manually create patches a bunch.

Pretty much how it works is, when you download the repo, paste in an unmodified ROM, and run the setup script, it generates a modified ROM based on the patch that's stored in the repo. Thereafter, before committing, a githook forces you to make sure the patch is up-to-date with the modified ROM.

Implementation details: the hack requires three files: base.nes, which is the clean ROM file; working.nes which is the edited ROM file; and patch.ips which is generated from the diff between base.nes and working.nes. I also wrote a setup.sh script for people new to the project which checks for the existence of base.nes and generates working.nes from the patch.ips. It also sets up the githook to prevent you from committing without making sure the patch is up-to-date (which is as easy as running make-patch.sh).

I'd also like to write a githook that automatically patches the ROM when checking out a commit. Handling merges seems the trickiest. (Successively apply patches from both branches? Still not clear how to check for merge conflicts though.)

I'll let you all know how this works out. If you'd like to try this, you can grab my repo and just use your own patch.ips for your own project. (Also, delete the FCEUX ROM map for castlevania :P).

https://github.com/nstbayless/cv3-controls

optiroc

That looks like a working solution, if the goal is to have an up-to-date patch in the repository at all times. Which is great if that's what you want, but it adds quite much complexity.

It's much simpler to only store what's needed for building in repository and have the build script produce the binary (patch, in this case). To make pre-built patches available for download you can write a script that builds and uploads the current version using the releases API.

RedComet

I use git for all of my projects. Each one has a build script of some sort that expects the unaltered ROM in a certain place and then it will build the patches from the ground up. The output is the final ROM with any assembly mods injected and the scripts/graphics/etc inserted. Usually only takes a few seconds for my projects, so I haven't had a reason to optimize for more time consuming builds yet.
Twilight Translations - More than just Dragonball Z. :P

NaOH

Quote from: optiroc on August 29, 2017, 07:37:57 AM

It's much simpler to only store what's needed for building in repository and have the build script produce the binary (patch, in this case).

The tricky thing is, since I'm using FCEUX, I am editing the .nes file directly as part of my workflow. I imagine a lot of people will be editing their .nes file directly, too. So this introduces an unusual dependency; the patch affects the ROM and the ROM affects the patch. So there needs to be two scripts, as I have: apply and make-patch.