News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: MBuild - General Purpose ROM hacking data extraction and insertion tool  (Read 6969 times)

DackR

  • Full Member
  • ***
  • Posts: 130
  • Mo~
    • View Profile
    • Hackaday.io Page
Hey guys-- Ever since I started work on the Marvelous translation early last year, I've also been working on a tool that would help keep me organized. I call it MBuild (or Marvelous Build).

Here's a preview from the readme file of what I've got so far:

=========================
MBuild v1.28 - 2016/01/09
=========================
MBuild - A Marvelous Translation and Hacking Tool
=================================================
Written by DackR aka Daniel Martin Burgess
==========================================
!!!Requires .Net Framework 4.5
==============================
CREDITS:
Lunar Compress DLL v1.8 by FuSoYa.
Lunar Compress Interface Code Originally by giangurgolo and Omega (I've made some minor changes).
Super Bomberman 5 Decompression/Compression by Proton (Ported to C# by yours truely).
xDelta3 created by Joshua MacDonald (jmacd).
The SNES Checksum routine was ported to C# from the original SNES9x source.
=======================================================
ABOUT MBuild:
MBuild was originally created to assist in change management while
working on the translation of Marvelous: Another Treasure Island.
There are many features, with more features being added all the time.

Gone are the days of making permanant changes in a hex editor to the
original ROM file. With MBuild, the user is encouraged to extract data from
the original ROM and store it separately, in an organized way. When the time
comes to build the data, an XML file containing a list of data files, offsets,
and other information is used to neatly fold in the changes to the output file.

Features:
Ability to dump binary data from a file.
Ability to dump script text.
Ability to encode script files into binary format.
Build-time compression using a variety of methods.
Supports Lunar Compress DLL compression types.
Supports Super Bomberman 5 RLE-type compression.
Seperate command-line options for any process available during build-time.
Build-time SNES ROM file expansion support (pad option).
Build-time SNES checksum correction and header modification (pad option).
Build-time diff patch creation supporting output of xdelta and/or ips files.
Drag-and-drop XML files on the MBuild executable to use that XML file to build a project.

Notes:
Relative paths are accepted.
PC hexadecimal offsets only.
Previous build files of the same name will be overwritten.
Modify the MBuild.XML (can be named differently) file to customize the build process.
XML Comments are supported.

XML Structure example:
<build original="ROMFILENAME.SFC" name="NameUsedToGenerateOutput" path="..\">
   <lzr file="x121964_LZR_TITLE.bin" offset="121964" />
   <lzr file="x1DA5ED_LZR_SUBTITLE.bin" offset="1DA5ED" />
   <lzr file="x1D948F_LZR_NINTENDO.bin" offset="1D948F" />
   <lzr file="x123951_LZR_INTROFONT.bin" offset="123951" />
   <lzr file="x1257F6_LZR_INTROTEXT.bin" offset="1257F6" />
   <lzr file="x14B628_LZR_PUZZLE1.bin" offset="14B628" />
   <lzr file="x132FC4_LZR_ANTBUSTER_GINARANSOM.bin" offset="132FC4" />
   
   <rep file="x270000_REP_KANJI_FONT.bin" offset="270000" />
   <rep file="x2AC000_REP_MAIN_FONT.bin" offset="2AC000" />
   <rep file="x2B6000_REP_ACTION_MENU_GFX.bin" offset="2B6000" />
   <rep file="x2CE000_REP_LARGE_FONT.bin" offset="2CE000" />
   <rep file="x7C84_REP_ASM_RELOCATE.bin" offset="7C84" />
   <rep file="x268000_REP_MISC_MENU_AND_NUMBERS.bin" offset="268000" />
   <rep file="x7FDC_REP_Possible_PalleteFix.bin" offset="7FDC" />
   
   <ins file="x300000_INS_EnglishScript.bin" offset="300000" />
</build>

Explanation/Notes:
To simplify the management of changes to a ROM file,
you are able to keep separate binary files. This is for ease of backup and editing.

Supported Node Types:
build=    attributes under this node control the basic build parameters such
      as source file, name of output, and file path for all files.
rep=   Replace data at offset using RAW uncompressed data.
ins=   Insert data at offset using RAW uncompressed data. When inserting data,
        the output file will be padded to the next valid ROM size by default.
lzr=   Stands for LZ, Replace. Essentially compresses binary data before
      replacing data at a specified offset with the compressed data.
      All Lunar Compress compression types are supported.
lzi=   Stands for LZ, Insert. Compresses data and insert at specified offset.
bpr=   Bitplane conversion will be attempted. Currently, only two modes are supported.
bpi=   Same as above, but the data will be inserted into the outfile.
rlr=   RLE compression is performed before the data is overwritten at the specified offset.
rli=   Same as rlr, but the compressed data is inserted (file size changes).
sbr=   Script build and replace. When you specify a script text file and a table file,
      the data is build and then written to the output file at the specified offset.
sbi=   Same as sbr, but as an insert operation.

Additional Build Process Notes:
-Build process will follow node steps in order.
-Original ROM is not modified.
-Input files are not modified.
-Output file is generated fresh for each build.
-A unique file name is generated (for each date).XX- This is depreciated. Uses build info to generate the output file name.
-Multiple build nodes are supported, but nested builds are not.


Command-line arguments:

      "build"   -   Used to build a project using a specific XML file.
            -   If no argument is specified, this process runs against "MBuild.XML" by default.
            -   This behavior can be changed by modifying the "MBuild.exe.config" file's AutoBuild Setting.
            -   This argument only evaluates the /xmlfile property-- all others are ignored.
            -   Example: MBuild build /xmlfile:"c:\folder\file.xml"

  "dump-script"   -   Used to dump an unformatted script file. (Currently) Support is planned for pointer table traversal.

   "bin-script" -   Used to build the script file (unicode/ascii txt file) into a binary-formatted file. Support is planned for Pointer table generation.

      "comp" -   Uses one of any number of compression types to compress a file. Lunar Compress Types are supported as well as a few others.

      "fixsum" -   Use this argument to fix the checksum of SNES ROM files. Headered and Interleaved ROMs are unsupported.

        "ips" -   Use this argument to generate an IPS patch.

      "xdelta" -   Generates an xDelta3-compatible patch file.

  "bpp-convert" -   Convert a file between a few of the supported bpp formats.

--------------------------------------------------------

I'm working on several game-specific tools. Also general tools for bulk dumping of data utilizing pointer table data. (I need more examples of different pointer tables used in games.)

So far, I am able to dump data utilizing a pointer table in the case of Marvelous and Super Bomberman 5, but I want the tool to be flexible enough to be able to read a configuration file and be able to traverse different pointer types and dump data efficiently.

I've already made an older version of this tool available (comes with the Marvelous Localization Tools in the translation thread). Just wanted to see if there was any interest for me to maintain general releases. If not, I'll probably just keep it to myself because that really all I made it for...

EDIT:
I'm looking for some other ideas of features to implement as well. So, if you've got ideas, lets hear them. If you think this tool is poorly planned, let me know that too. ;)
« Last Edit: January 09, 2016, 05:59:15 pm by DackR »

BlackDog61

  • Hero Member
  • *****
  • Posts: 784
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #1 on: January 09, 2016, 06:57:18 pm »
This is kind of a tough space to discuss, and yet one where it would be so great to find something common. And so tough. :)
I'd like a text-supporting tool for dumping & inserting to handle quite a few things...
- a mix of hex data and text data. We generally talk about pointers tables, but it so happens my favourite example game also has headers that need updating (because sizes of sections containing text are changed, offsets to subsequent sections in the main header need to be updated.) On top of the pointers table, and the text encoded according to a table.
- some pure hex data, when I'm changing screen layouts and whatnuts.
- some pure hex data taken from the results of an assembly generating toolchain. Actually, for the PSP, I'm using armips and it is not obvious which toolchain should be called first. (Currently I have 1 specific tool of my making to generate a game-specific table (dependent on files sizes), and I run a 2-pass process: assemble, then generate the table, then assemble again.
- support custom encryptions / compressions because that's what these games do - they specialize a lot of things. (need a plugin or executable call interface)

- Cartographer + Atlas propose part this with quite some flexibility. (But it would have worked in my example of interest. I tried.)
- I don't see myself using the quickbms scripts to do anything beyond pure structure parsing. (I'm not even using it for that.)
- I haven't seen any updates on TextAngel for a while (but I may be missing stuff)

If versions are taken care of in that nice toolchain, that's great. (And I mean using a versioning system like git - since it's the hype system of the moment.)

Those would be my overall 3 cents.
« Last Edit: January 09, 2016, 07:02:44 pm by BlackDog61 »

DackR

  • Full Member
  • ***
  • Posts: 130
  • Mo~
    • View Profile
    • Hackaday.io Page
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #2 on: January 10, 2016, 01:27:40 am »
This is kind of a tough space to discuss, and yet one where it would be so great to find something common. And so tough. :)
I'd like a text-supporting tool for dumping & inserting to handle quite a few things...
- a mix of hex data and text data. We generally talk about pointers tables, but it so happens my favourite example game also has headers that need updating (because sizes of sections containing text are changed, offsets to subsequent sections in the main header need to be updated.) On top of the pointers table, and the text encoded according to a table.
- some pure hex data, when I'm changing screen layouts and whatnuts.
- some pure hex data taken from the results of an assembly generating toolchain. Actually, for the PSP, I'm using armips and it is not obvious which toolchain should be called first. (Currently I have 1 specific tool of my making to generate a game-specific table (dependent on files sizes), and I run a 2-pass process: assemble, then generate the table, then assemble again.
- support custom encryptions / compressions because that's what these games do - they specialize a lot of things. (need a plugin or executable call interface)

- Cartographer + Atlas propose part this with quite some flexibility. (But it would have worked in my example of interest. I tried.)
- I don't see myself using the quickbms scripts to do anything beyond pure structure parsing. (I'm not even using it for that.)
- I haven't seen any updates on TextAngel for a while (but I may be missing stuff)

If versions are taken care of in that nice toolchain, that's great. (And I mean using a versioning system like git - since it's the hype system of the moment.)

Those would be my overall 3 cents.

I'll need to update this post later to address everything you've talked about, however, I'd be interested to hear more about your current tool set. I will admit that right now this tool is very focused on SNES translation projects. I plan on adding onto the tool whenever someone sees a need. My plan was to post it on GitHub and allow people to customize the tool to their project and then I would possibly merge their changes into the main repo. I'm used to coding alone, so I'm hoping to get some experience with collaborative work...

I've posted an example of how this tool works over at GitHub. This includes all of the data that I used to build Version 1.05 of my Marvelous SFC translation patch. It also includes the MBuild tool and a few other tools. It's not completely documented nor is the latest source available yet. If you get a moment, give it a look and see how it would fit into your processes compared to what you currently do.

https://github.com/danielburgess/Marvelous-SFC-EN

BlackDog61

  • Hero Member
  • *****
  • Posts: 784
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #3 on: January 10, 2016, 03:26:29 pm »
I <b>think</b> I'm starting to understand what this tool does. Could you check if the following is correct?
As input to this tool, the user provides:
- the original rom
- binary files containing some modifications
As output, the program creates a new ROM file.
All operations you can specify (today) are binary insert, replace, remove operations, with the option of executing a compression on a binary file before its use for insertion/replacement.

Correct?

DackR

  • Full Member
  • ***
  • Posts: 130
  • Mo~
    • View Profile
    • Hackaday.io Page
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #4 on: January 10, 2016, 05:31:52 pm »
I <b>think</b> I'm starting to understand what this tool does. Could you check if the following is correct?
As input to this tool, the user provides:
- the original rom
- binary files containing some modifications
As output, the program creates a new ROM file.
All operations you can specify (today) are binary insert, replace, remove operations, with the option of executing a compression on a binary file before its use for insertion/replacement.

Correct?
This is more or less correct for the main "build" function, yes.
It has a script building function that is currently less feature-full than Atlas, but I plan on adding compatibility for Atlas-formatted scripts. I usually build the script into a binary separately from the main build process-- depending on the size of the script because this is something that is still a bit slow-- and when I'm making edits to other binary files and rebuilding fairly frequently to test. It's just faster that way. (It's the difference between a 300ms vs 5-8 second build.)

Don't forget that it can automatically create ips and xdelta patches following a build as well. (Not that there are any shortages of ways to do those functions in the first place...)

Anyway, does this seem like it could be potentially useful? Or are there already tools that I've overlooked the perform the same function? Maybe it's too complicated to be of any use to most people? Would it be better if it had some kind of GUI to support it in some way? Maybe I should rethink the design a bit.

RyanfaeScotland

  • Sr. Member
  • ****
  • Posts: 361
    • View Profile
    • My Brill Game Site
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #5 on: January 10, 2016, 06:24:00 pm »
I'm in the need of something like this with my current process being exactly what you described, making modifications via HxD directly on the ROM file.

I'm doing a bit of a toolchain review just now so can slot this in quite nicely and report back how it goes if you are interested?

Only issue I can foresee is I need it for the Megadrive (68K) but it's meant to be generic right, so could be interesting to see how I get on.

DackR

  • Full Member
  • ***
  • Posts: 130
  • Mo~
    • View Profile
    • Hackaday.io Page
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #6 on: January 10, 2016, 07:47:26 pm »
I'm in the need of something like this with my current process being exactly what you described, making modifications via HxD directly on the ROM file.

I'm doing a bit of a toolchain review just now so can slot this in quite nicely and report back how it goes if you are interested?

Only issue I can foresee is I need it for the Megadrive (68K) but it's meant to be generic right, so could be interesting to see how I get on.

Here's a current build. Give it a try and let me know how it works out for you. I've made some major changes to the code over the last 3 days, so it's probably a little buggy.
 
https://googledrive.com/host/0B7Lih2HxwoqqLS1nWmZXenFRbmkzcUtYcVpWMEN4dw/MBuild_1.28.zip

RyanfaeScotland

  • Sr. Member
  • ****
  • Posts: 361
    • View Profile
    • My Brill Game Site
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #7 on: January 10, 2016, 07:54:21 pm »
Here's a current build. Give it a try and let me know how it works out for you. I've made some major changes to the code over the last 3 days, so it's probably a little buggy...

Always reassuring to hear. :)

That's it download and it runs without any files displaying the 'Press any key to exit' message which is a start. Will get a project set up for it during the week and give it a whirl on something simple.

Had a quick look on the project on GitHub, is it just the binaries and not the actual source that is up there?

DackR

  • Full Member
  • ***
  • Posts: 130
  • Mo~
    • View Profile
    • Hackaday.io Page
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #8 on: January 10, 2016, 08:29:26 pm »
Always reassuring to hear. :)
:thumbsup: It's not anything that would mangle your files or murder your dog, but just in case it blows up with an unhandled exception or something...

That's it download and it runs without any files displaying the 'Press any key to exit' message which is a start. Will get a project set up for it during the week and give it a whirl on something simple.
Oh- also note that the XML file I packaged with the app should really be an "MBXML" file. If "MBuild.MBXML" exists in the same folder as the app and you run it without arguments, it will automatically try to build using that file.

Had a quick look on the project on GitHub, is it just the binaries and not the actual source that is up there?
The repo on GitHub is intended to be for the Marvelous translation project. I really only referenced it as a working example of the tool.

I'll try to have the source code for MBuild up in it's own repo sometime this week. I feel too lazy to set up the remote repository and push it at the moment-- unless you think that would help you get started. Then just let me know.  ;)

RyanfaeScotland

  • Sr. Member
  • ****
  • Posts: 361
    • View Profile
    • My Brill Game Site
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #9 on: January 11, 2016, 09:20:23 am »
No rush to get the source up, was just making sure I wasn't over looking it.

If I run into any problems and we can't solve them then I'll download the source (what's it written in btw?) and see if it helps but to start lets just go for it like a normal use case.

DackR

  • Full Member
  • ***
  • Posts: 130
  • Mo~
    • View Profile
    • Hackaday.io Page
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #10 on: January 11, 2016, 08:37:11 pm »
No rush to get the source up, was just making sure I wasn't over looking it.

If I run into any problems and we can't solve them then I'll download the source (what's it written in btw?) and see if it helps but to start lets just go for it like a normal use case.
I shall not rush then. :)
This is written in C#. There are a few portions of code that were originally written in C++.

Just let me know if you find anything broken, or if you have any other ideas for features.

ObiKKa

  • Full Member
  • ***
  • Posts: 106
  • Tome of retro gaming knowledge!
    • View Profile
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #11 on: January 13, 2016, 03:50:51 pm »
DackR, your new software seems like it could be a good starting place for beginner or amateur rom-hackers to learn about editing the inner's of a ROM and dumping script lots. Am I right? If so that would be a nice way to help hurry the fan-translation scene along and improve a user's experience with romhacking. Would a programming knowledge be necessary when inserting the scripts and data into the ROM and polishing it? I hope not. But perhaps scripting is a bit different from that in what it does like creating pointers and tables?

Some questions about compatibility. Would you make this software work well with the variable width font tools like Nightcrawler's nuVWF and text editors like TextAngel or Atlas? Sorry I don't know them much. That'd be the next nightmarish step in knowing how to make all text and their spacing elements in-game look polished in the end. NC's nuVWF is not yet updated. He's got a lot on his plate - he's got to finish a final patch for a new game and update two of his tools for public usage. I think we'd have to wait till he does it all in one go.
Seeing as your mBuild is SNES focused I was thinking if you could make it work with consoles of that era like the Sega CD and maybe TurboGrafx CD. They all had similar processing powers so it'd be ideal. Games on the Sega CD had technically better graphics than those 2 other consoles while it & TCD were able to output FMVs which the SNES couldn't do. Maybe even think about adding in GBC and GBA support? You only have to look at the Datacrystal lists. They're HUGE especially with many missing RPGs on the Sega CD!

RyanfaeScotland, your toolchain for the Genesis seems interesting, but according to your posts the 3 tools in them may require major programming knowledge, right? Are they only for the skilled programmers? Also could they be used or tweaked to work with the Sega CD games?

DackR can you clarify if your new build is intended for everybody from the newbies and amateurs to romhackers with little programming knowledge to the true programmers and any other roles like an independent translator?

January 13, 2016, 03:57:26 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
Also got two more questions. Many SNES games have a nasty habit of having compressed graphics. So those without the programming knowledge cannot decompress them. But your tool doesn't seem to include a decompressor of any sort yet. Or is this too much of a hassle for you?

Then there's the compression routines already included within your build. Is it your plan to have multiple routines of this sort so that an user can select one. And will you add more diverse routines in the future? How would one know which is the best pick for a specific game?
« Last Edit: January 13, 2016, 03:57:26 pm by ObiKKa »

RyanfaeScotland

  • Sr. Member
  • ****
  • Posts: 361
    • View Profile
    • My Brill Game Site
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #12 on: January 14, 2016, 07:58:20 am »
RyanfaeScotland, your toolchain for the Genesis seems interesting, but according to your posts the 3 tools in them may require major programming knowledge, right? Are they only for the skilled programmers? Also could they be used or tweaked to work with the Sega CD games?

Nothing in my toolchain requires programming knowledge to use but the output they create requires and understanding of 68K assembly. If you want to modify them then yes you'll need to know how to program.
« Last Edit: January 14, 2016, 05:42:05 pm by RyanfaeScotland »

ObiKKa

  • Full Member
  • ***
  • Posts: 106
  • Tome of retro gaming knowledge!
    • View Profile
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #13 on: January 14, 2016, 12:10:38 pm »
Nothing in my toolchain requires programming knowledge to use but the output the create requires and understanding of 68K assembly. If you want to modify them then yes you'll need to know how to program.

Oh, thanks so much for your reply! That bit is clear to me now.

r57shell

  • Jr. Member
  • **
  • Posts: 13
    • View Profile
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #14 on: January 14, 2016, 07:41:43 pm »
Only issue I can foresee is I need it for the Megadrive (68K) but it's meant to be generic right, so could be interesting to see how I get on.
For Sega Genesis hacking purposes I'm using asm68k. It's "just" assembler.
From meaning of word "assembler" - it assembles things up.
So, you would ask how do I make "toolchain" from just assembler? Easy!

You create let say "main.asm" file, and input there:
Code: [Select]
org 0
 incbin "original_rom.bin"

 org $12345

 include "first_thing.asm"
 include "second_thing.asm"

 include "first_thing_f.asm"
 include "second_thing_f.asm"

First two lines - copying of original rom.
Next line: setup place where start writing from.
Next asm files are those which assumed that data or code from them will be added one after another.
For example, in m68k you need to align some stuff, so for word align you may use "even" keyword, for other align - I don't remember.
In this files you define data via "incbin" keyword, or dc.b/dc.w/dc.l as normal data. Or you write code there, you can do it freely because it's assembler. Also, you can include other *.asm file, so use it as modules.

Important note: after all files that is assumed to be in continuous block (one after another), there are *_f.asm files, that in my usage has such suffix because they mean "fixes". They looks like following:
Code: [Select]
org $12323 ; where you need to fix pointer
 dc.l new_pointer

 org $32ABC ; where you need insert (by overwriting) jump
 jmp new_code

 org $9923 ; where you need to insert (by overwriting) call of subroutine
 jsr new_subroutine
 nop ; overwrite broken opcode affected by overwriting.

In the end whole hack consists of bunch of .asm files (one of our hack has ~2000 files, 200 directories)
and one "build.bat" file that consists of something like this:
Code: [Select]
asm68k /p /o ae- main.asm,new_rom.bin
fixheadr.exe new_rom.bin
move new_rom.bin "name_of_target_file.bin" >nul
pause
First line compiling whole stuff
fixheadr - utility that does fix checksum
move - to rename, because asm68k can't output file with name that we need :(
pause - to avoid closing of command line.

One note: we using asm68k that is runnable in Win7. There are several versions in internet, we using that one which runs OK on Win7.

That was "how it can be".

Other ways of doing "toolchain". If you developing your own game, you can use for example gcc, make, ld stuff. All you have to do is to write makefiles, ld configs, and maybe compile some of that tools. gcc can compile assembler and C/C++ code into *.o files (object files). I'm not sure about assemblers used in SNES, but gcc has m68k platform compiler. Also, for Sega Genesis, all that stuff already done, there are available SGDK for Win and Linux. It works fine.

I just wanna say that you may "configure" all that tools (gcc/make/ld) to use it as toolchain. And, of course you may run some other your own written tools like converters, compressors etc.
« Last Edit: January 14, 2016, 07:47:18 pm by r57shell »

ObiKKa

  • Full Member
  • ***
  • Posts: 106
  • Tome of retro gaming knowledge!
    • View Profile
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #15 on: January 15, 2016, 01:20:35 am »
For Sega Genesis hacking purposes I'm using asm68k. It's "just" assembler.
From meaning of word "assembler" - it assembles things up.
So, you would ask how do I make "toolchain" from just assembler? Easy!

...
In this files you define data via "incbin" keyword, or dc.b/dc.w/dc.l as normal data. Or you write code there, you can do it freely because it's assembler. Also, you can include other *.asm file, so use it as modules.

Important note: after all files that is assumed to be in continuous block (one after another), there are *_f.asm files, that in my usage has such suffix because they mean "fixes". They looks like following:
Code: [Select]
org $12323 ; where you need to fix pointer
 dc.l new_pointer

 org $32ABC ; where you need insert (by overwriting) jump
 jmp new_code

 org $9923 ; where you need to insert (by overwriting) call of subroutine
 jsr new_subroutine
 nop ; overwrite broken opcode affected by overwriting.

In the end whole hack consists of bunch of .asm files (one of our hack has ~2000 files, 200 directories)
and one "build.bat" file that consists of something like this:
Code: [Select]
asm68k /p /o ae- main.asm,new_rom.bin
fixheadr.exe new_rom.bin
move new_rom.bin "name_of_target_file.bin" >nul
pause

One note: we using asm68k that is runnable in Win7. There are several versions in internet, we using that one which runs OK on Win7.

That was "how it can be".
...

Cool. Oh, so each fix function/grouping in that code example shown does link to an .asm file containing the fix(es) codes written? Whew, that's a lot of work linking up everything! I guess from what you're saying that the assembling part is OK for a newbie to start learning from scratch. I know of one user who posts links to the how to get started guides and links to practise romhacking. That might be a good place to start off from. I'll try to track these down. That a good idea? And thanks everybody for all your helpful tips.

January 15, 2016, 01:36:26 am - (Auto Merged - Double Posts are not allowed before 7 days.)
I just realised something...

Hey guys-- Ever since I started work on the Marvelous translation early last year, I've also been working on a tool that would help keep me organized. I call it MBuild (or Marvelous Build).

...
 - A Marvelous Translation and Hacking Tool

ABOUT MBuild:
MBuild was originally created to assist in change management while
working on the translation of Marvelous: Another Treasure Island.
There are many features, with more features being added all the time.

Gone are the days of making permanant changes in a hex editor to the
original ROM file. With MBuild, the user is encouraged to extract data from
the original ROM and store it separately, in an organized way. When the time
comes to build the data, an XML file containing a list of data files, offsets,
and other information is used to neatly fold in the changes to the output file.

...

XML Structure example:
<build original="ROMFILENAME.SFC" name="NameUsedToGenerateOutput" path="..\">
   <lzr file="x121964_LZR_TITLE.bin" offset="121964" />
   <lzr file="x1DA5ED_LZR_SUBTITLE.bin" offset="1DA5ED" />
   <lzr file="x1D948F_LZR_NINTENDO.bin" offset="1D948F" />
   <lzr file="x123951_LZR_INTROFONT.bin" offset="123951" />
   <lzr file="x1257F6_LZR_INTROTEXT.bin" offset="1257F6" />
   <lzr file="x14B628_LZR_PUZZLE1.bin" offset="14B628" />
   <lzr file="x132FC4_LZR_ANTBUSTER_GINARANSOM.bin" offset="132FC4" />
   
   <rep file="x270000_REP_KANJI_FONT.bin" offset="270000" />
   <rep file="x2AC000_REP_MAIN_FONT.bin" offset="2AC000" />
   <rep file="x2B6000_REP_ACTION_MENU_GFX.bin" offset="2B6000" />
   <rep file="x2CE000_REP_LARGE_FONT.bin" offset="2CE000" />
   <rep file="x7C84_REP_ASM_RELOCATE.bin" offset="7C84" />
   <rep file="x268000_REP_MISC_MENU_AND_NUMBERS.bin" offset="268000" />
   <rep file="x7FDC_REP_Possible_PalleteFix.bin" offset="7FDC" />
   
   <ins file="x300000_INS_EnglishScript.bin" offset="300000" />
</build>

Explanation/Notes:
To simplify the management of changes to a ROM file,
you are able to keep separate binary files. This is for ease of backup and editing.

...

--------------------------------------------------------

I'm working on several game-specific tools. Also general tools for bulk dumping of data utilizing pointer table data. (I need more examples of different pointer tables used in games.)

So far, I am able to dump data utilizing a pointer table in the case of Marvelous and Super Bomberman 5, but I want the tool to be flexible enough to be able to read a configuration file and be able to traverse different pointer types and dump data efficiently.

I've already made an older version of this tool available (comes with the Marvelous Localization Tools in the translation thread). Just wanted to see if there was any interest for me to maintain general releases. If not, I'll probably just keep it to myself because that really all I made it for...


That's it! I can check the free tools he used to build the Marvelous translation patch. That's where we can started by studying how he did it there!
« Last Edit: January 15, 2016, 01:36:26 am by ObiKKa »

RyanfaeScotland

  • Sr. Member
  • ****
  • Posts: 361
    • View Profile
    • My Brill Game Site
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #16 on: January 15, 2016, 03:40:51 am »
For Sega Genesis hacking purposes I'm using asm68k. It's "just" assembler.
...

Great write up r57shell, thanks for sharing.

If I've read this right DackR it sounds like, on the Megadrive at least, asm68k is capable of writing out code/data sections to arbitrary places in the original ROM as well which might undermine the need for MBuild in this instance? Especially since (I'm assuming) MBuild require you to already have the source code compiled so it is just a binary to binary merge that is done?

I'm in the process of testing if MAME is a suitable replacement for my Gens triple at the moment and (as with most things) there is a lot to learn about getting MAME set up nicely for pain free debugging. Once I've got that done however I think I'll look at getting everything chained together so that I have a sleek way of reading code, making changes and relaunching the revised game. Here is where I think I'll end up:

MAME (for exploring/debugging) -> Gens Tracer (for assembly dump) -> Easy68K (for assembly re-writing / compiling) -> MBuild (for reinserting) -> MAME (for checking changes)

I also plan to check what Visual Studio has in the way of 68K editing facilities, the ability to hit F12 to jump to a function definition is really handy so would love to be able to do the same with a label in assembly.
« Last Edit: July 23, 2018, 04:44:36 pm by RyanfaeScotland »

mziab

  • Sr. Member
  • ****
  • Posts: 305
    • View Profile
    • mteam
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #17 on: January 15, 2016, 06:56:50 am »
For Sega Genesis hacking purposes I'm using asm68k. It's "just" assembler.

I can also recommend Macroassembler AS for this purpose. As an added bonus, it supports a plethora of architectures. One caveat is that what it outputs is not a raw file, but a .p file which contains all the sections and their offsets. This can be easily converted back to a ROM or patch if you so desire. I see it's also used to build lory1990's Phantasy Star IV disassembly.

I've also used vasm for a short while for my only Genesis project. It was more or less like AS, but less convenient. It didn't like overlapping sections caused by importing the original ROM. On the plus side, it performed some safe code optimizations for some code patterns, so that might be nice for some people. It cost me a fair bit of head-scratching when I saw different code being executed during debug the first time around, though.
« Last Edit: January 15, 2016, 08:33:36 am by mziab »

RyanfaeScotland

  • Sr. Member
  • ****
  • Posts: 361
    • View Profile
    • My Brill Game Site
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #18 on: February 04, 2016, 08:33:55 am »
About a week ago I noticed I was running a really outdate version of easy68K. Since updating I've found that it now has the ability to directly patch existing binaries with revised code so unfortunately DackR using this integrates the patching process into the tool I'm already using for coding / compiling meaning I think I've got my bases covered where MBuild would fit in.

Sorry!

DackR

  • Full Member
  • ***
  • Posts: 130
  • Mo~
    • View Profile
    • Hackaday.io Page
Re: MBuild - General Purpose ROM hacking data extraction and insertion tool
« Reply #19 on: February 04, 2016, 09:15:40 pm »
About a week ago I noticed I was running a really outdate version of easy68K. Since updating I've found that it now has the ability to directly patch existing binaries with revised code so unfortunately DackR using this integrates the patching process into the tool I'm already using for coding / compiling meaning I think I've got my bases covered where MBuild would fit in.

Sorry!
Hey, that's alright. Thanks for looking into it.

Especially if you work a lot in code rather than mostly modifying data, there are many tools that are better for that purpose

Mbuild is still a young project and I'll keep adding onto it to suit my purposes. :) I have a added Super Bomberman 5 and Super Famicom Wars compression algorithms recently. There are more on the way that Proton is working on...