Romhacking.net

Romhacking => ROM Hacking Discussion => Topic started by: Klarth on April 26, 2016, 08:16:30 pm

Title: What utility/document project should I work on?
Post by: Klarth on April 26, 2016, 08:16:30 pm
So I've had a few new romhacking-related ideas in my head over the past few years. I won't give details here, but each would be an upgrade significant enough to make the current utilities of choice obsolete (monkey-moore, CrystalTile2 / Tile Molester). I believe I can write a romhacking guide better than what's available, but not sure if it would be as transformative as the others. Let me know what you guys think.
Title: Re: What utility/document project should I work on?
Post by: hanhnn on April 26, 2016, 10:00:52 pm
A graphic editor that can support bitmap imported and produce the correct order of color palette
Title: Re: What utility/document project should I work on?
Post by: dougeff on April 26, 2016, 10:24:50 pm
And cut and paste from the clipboard. And import / export an indexed format like .gif , .tiff, or .png.

I don't care about .bmp which is RGB...it makes sense that RGB will have their color screwed up.

I suppose ypu could reduce any RGB image to its relative brightness, as if it were B+W image, and use brightness to create an index into 2bpp,4bpp indexing.

Title: Re: What utility/document project should I work on?
Post by: jink640 on April 26, 2016, 10:29:53 pm
It would be nice to have a monkey-moore type of program that would work with japanese characters. Every time I try with MMoore it screws stuff up  :'(
Title: Re: What utility/document project should I work on?
Post by: SleepyFist on April 26, 2016, 10:56:08 pm
Ive been using Tile Layer Pro with two roms open at a time, Zelda, and a "Blanked" rom for graphics storage and editing in place of the arranger, would be nice to have a graphics editor utility that has an saveable arranger that works the same way as storing in a blank rom,
Multiple switchable palettes would be cool too.

and this
A graphic editor that can support bitmap imported and produce the correct order of color palette
Title: Re: What utility/document project should I work on?
Post by: FAST6191 on April 27, 2016, 04:06:55 am
It would be nice to have a monkey-moore type of program that would work with japanese characters. Every time I try with MMoore it screws stuff up  :'(

Japanese does not exactly have a relative order to search with though. You kind of have something with kana but there are enough variations between cases, punctuation and such that they are hardly set in stone.
Monkey Moore has a kind of annoying to use hex relative search so I guess you could find the font and try a font order type search, though that is usually the opposite way people go. Then again it might be the opposite way because it is a pain to use.
Title: Re: What utility/document project should I work on?
Post by: Klarth on April 27, 2016, 06:12:11 am
Japanese does not exactly have a relative order to search with though.
Actually the method I have in mind builds upon relative search and is way more elegant for the end user. The end user would still need to manually identify the often out-of-order characters, control codes, and kanji, but finding a representation of the main alphabet is more than half the battle.

And import / export an indexed format like .gif , .tiff, or .png.

I suppose ypu could reduce any RGB image to its relative brightness, as if it were B+W image, and use brightness to create an index into 2bpp,4bpp indexing.
I don't see the point of specifically .gif, .tiff, or .png. BMP can be palette-indexed at a specific power-of-2 bpp's. Additionally, the program would likely generate a configuration file for reimporting the graphic back into the game that would take care of RGB->game palette, file location, and any other required info.
Title: Re: What utility/document project should I work on?
Post by: STARWIN on April 27, 2016, 08:43:38 am
more elegant

It doesn't matter what you have in mind, this alone sounds like fun.
Title: Re: What utility/document project should I work on?
Post by: BlackDog61 on April 27, 2016, 01:28:59 pm
Since we're asked for a wish-list ;D I'd go script analysis first.
Better guide to RH would come next in my mind.
Though they can be improved, graphics tools have done what I wanted from them so far (or I created my replacement scripts from edited BMP's when it was not the case).
Title: Re: What utility/document project should I work on?
Post by: VicVergil on April 27, 2016, 03:18:22 pm
Anything PS1 related, especially manipulating graphics and textures (to help fix the 12px fixed width fonts, or move stuff around the screen).
Title: Re: What utility/document project should I work on?
Post by: Gemini on April 27, 2016, 05:09:53 pm
A generic purpose library for hacking, maybe? Like a class to manipulate text and images. Anything task-specific usually falls behind pretty quickly, becoming obsolete as soon as your topic of hacking starts using complex formats or structures beyond logical blocks of continuous data.
Title: Re: What utility/document project should I work on?
Post by: shadow501 on April 27, 2016, 05:16:22 pm
Good idea to make graphics editor for PS1 because unitl now no tool for it.
Title: Re: What utility/document project should I work on?
Post by: Klarth on April 27, 2016, 05:43:31 pm
A generic purpose library for hacking, maybe? Like a class to manipulate text and images.
I already have TableLib that was released a long time ago. It's gotten some use, but maybe not as much as I had hoped. Mainly because it's in C++ and doesn't have a C# or python interface.

Image manipulation is interesting, but a library alone would defeat the purpose of why I would make a graphics editor -- for some specific features that nobody has made yet. It's more of a usability gripe than image decoding/encoding limitations. Though those certainly exist and I would try to address them, at least from the tile editing perspective. Linear modes may not see as much improvement.

Good idea to make graphics editor for PS1 because unitl now no tool for it.
PS1 graphics are going to be difficult to deal with regardless. TIM is more of a guideline than a standard in many games. A lot of graphics is behind compression. Browsing through large files looking for graphics in linear modes has limited success because you need to get the location and image width pretty spot on or image data will be completely unrecognizable. I'm really not sure how to significantly improve upon what's currently out there given these circumstances.
Title: Re: What utility/document project should I work on?
Post by: STARWIN on April 27, 2016, 06:18:03 pm
Browsing through large files looking for graphics in linear modes has limited success because you need to get the location and image width pretty spot on or image data will be completely unrecognizable. I'm really not sure how to significantly improve upon what's currently out there given these circumstances.

I'm not a graphics hacker so I have no idea what I'm talking about, but couldn't a program search for the exact location by using something from the emulator output (like taking a piece of a screenshot manually and using it as the search "string")?
Title: Re: What utility/document project should I work on?
Post by: Klarth on April 27, 2016, 07:20:25 pm
I'm not a graphics hacker so I have no idea what I'm talking about, but couldn't a program search for the exact location by using something from the emulator output (like taking a piece of a screenshot manually and using it as the search "string")?
That approach would have some success in specific settings, but you shouldn't use screenshots (even at native resolution and unfiltered) in a PSX game because textures are drawn on rotated/stretched triangles. You'd need to take the texture data out of VRAM to have a shot. The algorithm would need to allow for data gaps in between image lines to account for image strides and whatnot, but it could be reasonable in games without any sort of compression. Seems like a cumbersome method with not a lot of payoff. Reminds me of searching for SNES palettes, just less successful.
Title: Re: What utility/document project should I work on?
Post by: shadow501 on April 27, 2016, 09:20:10 pm
I don't mean TIM Viwer or else, I mean like yy-chr-chr..etc
Title: Re: What utility/document project should I work on?
Post by: Gemini on April 28, 2016, 05:41:25 am
I already have TableLib that was released a long time ago. It's gotten some use, but maybe not as much as I had hoped. Mainly because it's in C++ and doesn't have a C# or python interface.
I mean a more comprehensive library, not just to manage encoding and codepage. Something a bit more complex, like an xml parser for complete dumps and meta data required in more advanced projects. That would leave the user with a better dump format that can be integrated in a greater application with ease.

Quote
Image manipulation is interesting, but a library alone would defeat the purpose of why I would make a graphics editor -- for some specific features that nobody has made yet. It's more of a usability gripe than image decoding/encoding limitations. Though those certainly exist and I would try to address them, at least from the tile editing perspective. Linear modes may not see as much improvement.
A bitmap class for manipulation is the way to go here, something that can grab and impress pixels in an abstraction layer to keep all the math away from the final user. For example, you create a generic Image class wrapper that  can be used to replace data and saved to a custom format. Say, you start from a PNG/BMP loader and then the user can override a virtual method to save the new data required by a game. I've been using something like this for years, it's quite useful when you don't wanna use editors that follow either a weird standard or doesn't align with custom needs (say: Tilemolester easily messing up your palettes because it tries to match a 32 bit picture to indices that are easily missed or altered to make reinsertion fail).
Title: Re: What utility/document project should I work on?
Post by: Klarth on May 01, 2016, 03:25:39 pm
I have begun work on a graphics editor. This is the most ambitious project of the three (though the writing of a comprehensive romhacking document would take a long time). I will make a separate thread once substantial progress has been made. There are a lot of design considerations I still need to solve.

Additionally, the project will use C# / WinForms / Direct2D. This means Win7 (or Vista SP2) or above will be required. The decision to keep the codec core in C# or external in C DLLs has not been made yet. Performance will dictate that and I have not even started in that direction yet.
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 02, 2016, 05:51:31 am
A guide dedicated to explaining all of the hundreds of ways to go about extracting textures-with-text from games, how to edit them and with what software; how to convert them and back again into all of the different file types and finally how to re-insert them back into the game.

There really are NOT many good guides/FAQs that i've seen that really detail how to go about this and there are tons of ppl like me who have never edited an image in their entire life and more than that, have never extracted something from files that are not meant to be extracted from: bins and the like, and stuff that is compressed. Of course I know how to do this stuff now but I still know only maybe 10% of the subject.

I had to give up for the time being on the NDS rpg Wizardry: Asterisk because I just couldn't find any good guides on GBATemp that detailed how to deal with that games specific images. Sure, there are guides there on GBATemp about hacking and graphical hacking and the answer to everything is: LOL USE CT2. the problem is no one really explains how.
Title: Re: What utility/document project should I work on?
Post by: Klarth on May 02, 2016, 09:29:46 am
extracting textures-with-text from games
Fonts are one of the secondary areas where I'll add some features if I have time, but the feature set is already ambitious as-is. I have a few ideas, but they are not as concrete or as universally powerful as my core idea for the editor.

For your case with text-on-top-of-textures (I think this is what you're describing), you need artistic skill. If you can extract from the format into BMP and reimport, then software like GraphicsGale will be very useful. You will basically need to "redraw" the underlying image to some extent, then layer the new text on top.
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 02, 2016, 12:39:18 pm
Yeah I ran into my lack of knowledge of editing images with Elminage 2. I ended up editing the Elminage 1 (which was localized into english)-- the Elminage 1 menus and then replacing the japanese Elminage 2 menus with the english ones from the first game. It worked like a treat because there were no menu changes from E1 to E2. However for Elminage 3 they changed it a lot so I won't be able to do that again.

Screenshot:
(http://i7.photobucket.com/albums/y292/aweigh01/Screenshot%202016-04-30%2007.59.56_zpsker5it7i.png) (http://s7.photobucket.com/user/aweigh01/media/Screenshot%202016-04-30%2007.59.56_zpsker5it7i.png.html)
Title: Re: What utility/document project should I work on?
Post by: Gemini on May 02, 2016, 12:54:13 pm
That is the second smallest screenshot I've seen since last week. You need more resolution, it's like looking at a postage stamp.
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 02, 2016, 01:05:26 pm
It expands on the clickety-click? Postcount +1?

Title: Re: What utility/document project should I work on?
Post by: Gemini on May 02, 2016, 01:11:39 pm
Nope. Still 320x180.

EDIT: I had to click on the magnifying lens, but that produced a corrupted image which shut itself in a few moments.
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 02, 2016, 01:17:22 pm
(http://i7.photobucket.com/albums/y292/aweigh01/Screenshot%202016-04-30%2007.59.56_zpsker5it7i.png~original)
Title: Re: What utility/document project should I work on?
Post by: Klarth on May 02, 2016, 03:15:19 pm
Photoshop and other programs would make pretty easy work out of that. I don't think I'll address this issue specifically in my project. Though I'm hoping that import/export will be more successful than in other programs.
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 02, 2016, 04:23:19 pm
well, how would one go about translating a texture? like, open up MSPAINT and pick Letterbox tool and write letters on top of the tile? I honestly have no clue.
Title: Re: What utility/document project should I work on?
Post by: Klarth on May 02, 2016, 04:28:52 pm
You open up photoshop and create a template with the background graphic (or modify the existing one by removing the text). Then you pick a font and draw on it. Save each iteration separately. Then overwrite the original graphics.
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 02, 2016, 09:23:02 pm
Sounds like a lot of work. The reason I somewhat dislike editing images that function as menus and the like is that no matter what I do the end result will never look as good as the original game tile/texture.

It's frankly another facet of rom hacking where there is need of yet another skill set... i guess that just menas i'm going to have to learn. :)

BTW, i learned today how to implement a command, so to speak in the cheat engine debugger, to change a game's font to 8 pixels instead of 16! now i'm trying to do the same on a psp game and the referenced functions are all mostly the same, and infact they even use the same GDI32 but for whatever reason i ain't getting any joy.

Anyway i'll stop now since this thread isn't about this stuff. So yeah, a small section on basic step-by-step editing of common images, like rpg menus, how to do it / what to use / etc, and how to re-insert them on a per-console basis. That'd be sweet.

also it'd be amazing if you included as well in that section a detailed breakdown of the many things you can do with a game's font image once you've extracted it, all the way from stuff like shaving off the rough edges, so to speak, off each letter to stuff like replacing fonts and even maybe you could include a section on making your OWN fonts to serve a game's needs best and how to do it, ttf to per-console formats and the like and how to make it work.

now that would be awesome.

EDIT: oh, you should also make sure to include a detailed section on how to use debuggers, what they can do and what they are for and what you can accomplish with them and you CANNOT accomplish with them; debugging is quickly becoming one of the most important facets of rom hacking and almost no one knows how to use a debugger, myself included.

and you know? also a section on what tools are needed for what task, more specifically concentrating on whether you should use a rom hacking oriented hex editor or whether perhaps what you need to do at that moment could be better served with notepad++ and using its powerful macros and ability to set automatic line breaks. that's just a specific example but it's stuff that comes up. most people don't even know about macros for text/script editing.

also, i just thought... you really need a section all about tables and the crazy stuff you can do with them. and even more so what you can do with two tables at once. a lot of people don't know about DTE's and how you can stuff a table with them.
Title: Re: What utility/document project should I work on?
Post by: BlackDog61 on May 03, 2016, 03:40:21 pm
I find the way you tackle these topics both interesting and fresh, and fairly naive too.

Sounds like a lot of work. The reason I somewhat dislike editing images that function as menus and the like is that no matter what I do the end result will never look as good as the original game tile/texture.

It's frankly another facet of rom hacking where there is need of yet another skill set... i guess that just means I'm going to have to learn. :)
Yes, graphics editing can be a field of expertise in and by itself. If you think about the way game developers staff their teams, the graphics experts are often numerous, specialized and skilled, these days. It's an important part of what makes a game.
There is of course not "one answer to rule them all" (games). There are many ways to approach graphics editing for translation. But I believe that the "overall plan" is indeed what was described, or almost:
- Identify and extract graphics with textual parts that need editing. The extraction is often game-specific, all the more for older platforms.
- Choose if you want to go for a completely different graphical style (vs keep the existing one). Bear in mind that, while doing your own thing is likely to yield coherent looks, it may mean you need to edit graphics that don't contain text, for coherence's sake.
- Choose a graphics editor for the next steps. A lot of people liked Photoshop until it became cloud-based. Gimp is a good choice too, as it is free, but a lot of people need time to get used to its user interface.
- If you want to edit with existing style, then...
- Import the existing graphics as the bottommost layer.
- Create a layer on top to "delete" the text. Which actually means you paint the background on top of the old text, guessing what the pixels were. For this step, I recommend looking around the picture for similar areas. It is often best to copy paste from other areas, and to randomize a bit the pixels. (Yep, lots of manual pixel-by-pixel steps here, although people may know good techniques they could share here.)
- Add a layer with the new text. (This way, it is easier to change if you change translations... which happens often for chapter titles and such.)

BTW, i learned today how to implement a command, so to speak in the cheat engine debugger, to change a game's font to 8 pixels instead of 16! now I'm trying to do the same on a psp game and the referenced functions are all mostly the same, and in fact they even use the same GDI32 but for whatever reason i ain't getting any joy.
You do understand there is not one way to do it for all games, there, don't you? That's the bane of RH: lots of things are done uniquely per game, which means you have to understand how these things are done for each game. Sometimes you could be lucky. The concept seems universal, but the implementation is (generally) not.

Anyway i'll stop now since this thread isn't about this stuff. So yeah, a small section on basic step-by-step editing of common images, like rpg menus, how to do it / what to use / etc, and how to re-insert them on a per-console basis. That'd be sweet.
Again, how to reinsert is pretty much unique and specific to each game. Well, yuo can find some common formats (such as GIM for PSP) but there often are non-standard ones. (Supposedly, new platforms are supposed to help with this, since their system libraries include a lot of graphics-handling and text-handling routines. I'm not sure which generation of console really offers this (I'd bet NDS, PS3), so I'll let Fast and other better-knowing people explain it.

also it'd be amazing if you included as well in that section a detailed breakdown of the many things you can do with a game's font image once you've extracted it, all the way from stuff like shaving off the rough edges, so to speak, off each letter to stuff like replacing fonts and even maybe you could include a section on making your OWN fonts to serve a game's needs best and how to do it, ttf to per-console formats and the like and how to make it work.
Font edition and creation is an art by itself. I'm not sure I understand what you're expecting here. I guess there should be websites about this with artistic guidelines?

EDIT: oh, you should also make sure to include a detailed section on how to use debuggers, what they can do and what they are for and what you can accomplish with them and you CANNOT accomplish with them; debugging is quickly becoming one of the most important facets of rom hacking and almost no one knows how to use a debugger, myself included.
I guess that by "is quickly becoming", you mean that, on your learning path, you are bumping time and time again into suggestions that using a debugger could help with many aspects of translating a game?
We've had this request quite a few times and the thing is - you have to learn programming in order to learn debugging. (At least that's what I feel.) By learning how a program you write translates into assembly,and how a debugger interacts with assembly instructions, you can learn what a debugger can do. It's like trying to explain assembly. You can write books about it, but the fastest way for you to learn it and understand it, really, deeply, is through first-hand experience. Is it slow? Yes, it is. Is it worth it? Yes, most definitely. Just like hex editing is a raw tool that allows anything (but you want to complement it with other, faster,higher-level tools), you can also reverse-engineer most things with a debugger (but it usually takes a lot of time even for small pieces of code, so you want to favor other techniques and approaches where you can).
I'm guessing you'll answer I'm not helping. Yet, from my perspective,I'm actually proposing paths forward for you,rather than just saying "it's not possible to do a one-match-them-all method and tool and guide".

and you know? also a section on what tools are needed for what task, more specifically concentrating on whether you should use a rom hacking oriented hex editor or whether perhaps what you need to do at that moment could be better served with notepad++ and using its powerful macros and ability to set automatic line breaks. that's just a specific example but it's stuff that comes up. most people don't even know about macros for text/script editing.
Good luck with that. It's a full life's worth of work.

also, i just thought... you really need a section all about tables and the crazy stuff you can do with them. and even more so what you can do with two tables at once. a lot of people don't know about DTE's and how you can stuff a table with them.
I agree tables would need a little revisiting.
DTE's are not multiple tables, for the record. They are about having several letters encoded as a single symbol. (Like representing "an" as a single byte, to spare 1 byte of space in every place they occur.)
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 04, 2016, 05:37:18 pm
I don't think that was "unhelpful" at all! That was a great reply. Thanks. And you're right about everything you said.

Just for the sake of keeping my post short I'll give one example: after my success at stumbling across Wizardry Empire 1's SetPixel / GetPixel functions I thought it would allow me to do the same with Wizardry Empire 3, which is a PSP game and not a PC game like the 1st gmae is; however it's been about a week so far and I've yet to successfully replicate my success on WE 3 on PSP.

HOWEVER! I have slowly, day by day, been learning more and more about WE 3's code and I understand what I'm seeing outputted in the PPSSPP's debugging log more and more. I firmly believe that EVENTUALLY I'll change the Pixel size on WE 3's font. In the 1st WE game, which as I said is on PC all you need to do is change a single value: "08" -> "10" and voila, the game's font is now 8 pixels instead of 16. Of course the real work is debugging the game and coming through its ASM routines until you find WHERE that value is.

On the PSP it is much, much, much more complicated. For starters the game has a buffer where it stores a lot of values, such as I suspect the pixel size one, and that buffer is continually read from and therefore it is not enough to change the pixel value(s) in one place; I think I have to do something like creating some sort of routine that makes the buffer redirect to newly created rom data containing the pixel values I want.

To be honest I've come to find out I enjoy trying to achieve these things much more than I do playing the actual game. I don't think I even like Wizardry Empire 3 that much; it is not as good as the 1st or 2nd games in the series. But I absolutely love coming home and spending a few hours trying to learn what it is I have to do to translate the game. I don't think I'll even play it after I translate it; I've already got 2-3 other japanese Wizardry-clones (dungeon crawlers) that I want to translate.

I started work on Wizardry XTH: Unlimited Students on the PS2 and while hex editing the executable is no problem (i.e. my changes reflect in-game without any problems), AND the game features latin-character support (no immediate need to figure out how to change the font)-- I've hit a complete roadblock with the game's only other game file, a file called DATA.DAT which is 800mb's and contains basically everything else that isn't in the PS2 executable, i.e. dialog and "lists of stuff" like items/spells/etc.

However unlike with the executable where I've already translated all of the menus and the like without any problems whatsoever when I translate something from the DATA.DAT file, say the spell "Cure"... IT DOESN'T SHOW UP IN GAME. The spell is still selectable and it works but THE WORDS ARE INVISIBLE.

So yeah, that's the first time this specific problem has happened to me as I've had a lot of stuff "go weird" in game when I edit a string but so far I've never had the in-game text go invisible but still functional LOL.

And like I said, I've come to find I enjoy this puzzle-solving. I have _zero_ clue why this is happening with this PS2 game but by god I swear I will I will find out. Unfortunately I can't find a good debugger for PS2 as the PCSX2 doesn't feature any... so I've been having to use Cheat Engine and while it "works" It isn't as intuitive as using the emulator-specific one, such as the PPSSPP debugger which has taught me a LOT about what I need to do and what I was doing wrong.

EDIT: I'm thinking the invisible text in the PS2 Wizardry game probably has something to do with pointer tables / or perhaps I have written over some invisible control character, or "pointer" character, maybe an invisible @ or B or whatever. You know what I mean. The problem is I've compared the original japanese DATA.DAT with my edited one using 010 Hex Editor and the only changes I've made are to simple japanese characters. I'm thinking I'll end up learning more than I ever wanted to know about pointers by the time I solve this puzzle.
Title: Re: What utility/document project should I work on?
Post by: Gemini on May 04, 2016, 06:57:03 pm
Unfortunately I can't find a good debugger for PS2 as the PCSX2 doesn't feature any... so I've been having to use Cheat Engine and while it "works" It isn't as intuitive as using the emulator-specific one, such as the PPSSPP debugger which has taught me a LOT about what I need to do and what I was doing wrong.
Actually:
(http://i.imgur.com/1WBWzIf.png)
Again, written by Kingcom. You're probably using an outdated version, the screenshot comes from PCSX2 1.5.0.20160424221415 (git).
Title: Re: What utility/document project should I work on?
Post by: aweigh on May 04, 2016, 08:45:18 pm
THANKS. Yeah i am, i was/am using 1.3.x. downloading now.