Romhacking.net

Romhacking => ROM Hacking Discussion => Topic started by: Spindaboy on May 10, 2016, 05:05:49 pm

Title: Do Game Developing Companies rely upon Hex Editors?
Post by: Spindaboy on May 10, 2016, 05:05:49 pm
As I have found for myself, hacking certain aspects of a ROM using a Hex Editor are impracticable and sometimes borderline impossible (especially if you don't know the location of what you want to change). So my question is, when making a game are the programmers forced to have every offset and every possible value for that offset written down, or do they have their own utilities that enable quick game editing? Some examples would be a graphics, map, or text editor. All of those make editing a game so much easier. It just doesn't seem realistic for anyone to be able to code an entire game solely in Hex.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: dougeff on May 10, 2016, 05:14:51 pm
The only reason you need to use a hex editor, is because you don't have access to the source code. If you had access to it, you wouldn't have to keep track of any file/object/graphics location... it would just be included in the project, and the compiler would keep track of everything.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Bregalad on May 10, 2016, 05:15:16 pm
No. Games were made using assembly, and custom tools. An hex editor is not required for making a game - and building up a game is different than hacking an existing ROM.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Klarth on May 10, 2016, 05:21:38 pm
Programmers/designers are not using hex editors for any reason while developing videogames. Having the source code and being able to actively debug the program is much more powerful. File formats have their offsets "hidden" to the programmer, unless they specifically want to make code that is hard to understand. You program routines to load/save your data and don't keep track of offsets unless there's a system performance reason. The compiler/linker will dynamically assign offsets as the game is being built in all modern game workflows.

A hex editor is a crude tool as you know. They're only used as a last resort in data modification because recreating a full level editor is a monumental task.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Spindaboy on May 10, 2016, 05:29:14 pm
So the source code that is used, is that similar to a language like Java or HTML that is easily readable? Are all NES games being built off the same base code that is unique to the NES? Or... would that be like an OS or something....?
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Bregalad on May 10, 2016, 05:34:42 pm
So the source code that is used, is that similar to a language like Java or HTML that is easily readable? Are all NES games being built off the same base code that is unique to the NES? Or... would that be like an OS or something....?
HTML is not a programming language.

NES games were programmed mostly in 6502 assembly. Modern games are programmed in high level programming languages, such as Java (if there is GB of memory used for nothing, long loading times and occasonial random crashes, it's usually a sign Java is there ;) ).

The is no Operating System.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Chronosplit on May 10, 2016, 05:40:41 pm
Hex Editors are used as last resorts.  Even in cases where a port is made without the source code.

For example, this seems to be a persistent thing said about Square-Enix.  Final Fantasy VI's PC port is more-or-less a GBA ROM, but all of it's new assets are on the outside.  One could say this is an extremely intensive hack, but they went out of their way to not touch the cart itself.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Spindaboy on May 10, 2016, 05:41:56 pm
HTML is not a programming language.

What? O.O

This is kind of off topic, but how do 3D models work? I understand that sprites are just an assembly of bits on the screen, but how does a game read a model of something? How is that data even stored?

May 10, 2016, 05:43:42 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
Hex Editors are used as last resorts.  Even in cases where a port is made without the source code.

For example, this seems to be a persistent thing said about Square-Enix.  Final Fantasy VI's PC port is more-or-less a GBA ROM, but all of it's new assets are on the outside.  One could say this is an extremely intensive hack, but they went out of their way to not touch the cart itself.
But if two systems run off different engines, how could a port work correctly without being touched? How could the PC port even interpret the original code of the GBA ROM?
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: VicVergil on May 10, 2016, 05:57:16 pm
Yes. It happens a lot, actually.

The official localization for Magical Knight Rayearth (Saturn) for example had the staff resort to hex editors to insert some of the text. Nintendo and Capcom use hex editors a lot too for their retro re-releases (this means lots of cases of just replacing stuff with spaces). In the early nineties, lots of developers would actually insert messages meant for people opening the ROM images with hex editors.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Gemini on May 10, 2016, 06:00:09 pm
But if two systems run off different engines, how could a port work correctly without being touched? How could the PC port even interpret the original code of the GBA ROM?
FF6 on Android and iOS uses the original SNES code converted as macros, where each opcode corresponds to a C++ function. The ROM is used to drag whatever data the original code is still addressing.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: freem on May 10, 2016, 06:00:57 pm
I was going to say that I thought Martin Hollis did it on Goldeneye, but he wrote a tool to extract the GZipped data instead (http://www.destructoid.com/a-last-minute-hack-helped-goldeneye-64-s-release-233115.phtml).
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: dougeff on May 10, 2016, 06:02:45 pm
Quote
Are all NES games being built off the same base code that is unique to the NES?

Atari (computers and console), Apple II, NES, Commodore 64, and a few others, were programmed in 6502 assembly.

I could be wrong, but i think officially licensed NES developers were given DEV KITs, which included detailed explanation of how to program NES games (poorly translated from Japanese).
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: jonk on May 10, 2016, 06:10:12 pm
So the source code that is used, is that similar to a language like Java or HTML that is easily readable? Are all NES games being built off the same base code that is unique to the NES? Or... would that be like an OS or something....?
First off, I'm not a game-writing expert. Though one game company did make me an offer for a job, the day following a day-long interview there. My expertise is in designing hardware and writing embedded software for scientific and commercial instrumentation. That said, there is a great deal of similarity. What I do is sometimes called "bare metal programming." Which is to say, you really need to apprehend hardware design very well and your programming deals directly with the hardware in very intimate ways. You must understand how things work to do that. And your programming skills are often quite different from those required for, say, Windows programming under .NET.

The work I do is going to depend upon the microcontroller and the other limitations imposed on the design. If I have very little RAM and ROM to work with (for competitive reasons I won't dwell on right now) and I won't be using libraries of code developed by other folks and forced onto my application for other (schedule) reasons, then I will probably be writing the application entirely in assembly. I can use semantics at that level which simply don't exist with any other language. And it may make sense to do that. I would tend to expect NES games to be written mostly in assembly code, as a slightly-educated guess. However, if there is more room to work with or if there are software libraries written in some other language (like C), then I will use a mixture of C and assembly... using assembly where needed and C everywhere else. I almost never have a project that can be entirely done in C and have never been exposed to a project using Java, for example. But if I were doing an embedded project that required the use of video and audio codecs and specialized flash file system designs where I wasn't permitted to write them all myself from scratch (likely, I would not be allowed to duplicate those "wheels" if they already are available for a reasonable cost), then I'd use the languages appropriate to those libraries and appropriate to the project requirements.

I am finding C being used with SNES development. But I've just started looking over the code, too. But what little I've learned so far suggests that at least some significant portion of SNES games were written as a C+assembly mixture. If you get a chance, go look on the web for some examples of C code. That will give you a good idea about what that kind of coding looks like.

Texture maps are often developed through the use of specialized tools to help artists. The same is true for sprites, and pretty much anything where you are having digital artists do their work. Sometimes, digital artists will use commercial off-the-shelf (COTS) software to do that job, producing standarized file formats that the game software writers will then either use directly, or else after some other conversion/compression steps. The tool chain can be quite long, in fact. Some of the tools (and steps) may be the result of captive programming employees working as part of the game team. Some may be COTS software. The exact details will vary by project, project size, team experiences, and a host of other factors all of which enter into the early planning stages. (I haven't even mentioned the musicians, yet.)

The game software itself will usually be running stand-alone. This means, without an operating system environment. Sometimes, a limited operating system will be written, designed to the game requirements. Sometimes, a COTS operating system will be used. Sometimes, no operating system at all, instead carefully crafting the interrupt system and timers and other related hardware so that the software can operate "as if" lots of stuff happens in parallel even though everything really is happening one instruction at a time. Some game systems will have two CPUs, though. I suppose some will have even more than that. And of course, they may also be "multi-core." Those tend to require operating system support of some kind.

HTML coding is not at all similar. In fact, I look at HTML more like a kind of "set of word processor commands" than I do a programming language.

Hex editing is a tool I'd probably still have on hand. But I probably wouldn't use it much. It's one of those things you might use if you know exactly what you are doing and need to do a "quick test" without having to re-run your entire compiler tool chain just to find out. So I'd want one present if something like that came up. But I wouldn't expect it to be part of a regular routine. It's an exception-based tool, I guess. But really handy when you have to have it.

Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: FAST6191 on May 10, 2016, 06:36:32 pm
[HTML is not a programming language]
What? O.O

This is kind of off topic, but how do 3D models work? I understand that sprites are just an assembly of bits on the screen, but how does a game read a model of something? How is that data even stored?

May 10, 2016, 05:43:42 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
But if two systems run off different engines, how could a port work correctly without being touched? How could the PC port even interpret the original code of the GBA ROM?

To be a programming language most would argue you have to be what is known as Turing complete. HTML traditionally is not this and was not intended to be, I am not sure what might go with some things for HTML5 though. When web devs needed this they would look to other things like php, ASP.net or on the client side with Javascript, flash, java, activex or that sort of thing. Learning HTML and expecting to be able to program is a bit like reading a dictionary and expecting to know a language -- it will probably help with several concepts but it is going to leave massive gaps.

3d models work in various ways. I am not sure how I want to describe them, and also where I would want to bring in textures and animations and cameras and maths relating to it all. There are probably three main types, those being point cloud (mainly for medical and 3d scanning), vertices (you have points, draw lines between them and you have your model) and bones (like vertices but instead of points on the nominal outside then you have things running along the inside and define how far out it goes from them. Textures also play a role here as you can improve low polygon count meshes with a texture by various means, modern textures are insanely complicated as well in what they can do. Animations in 3d then actually move the models (be it point cloud, vertices or bones) rather than just swap things out very quickly or slide/rotate/scale them around the screen like in a lot of 2d animation. As you might imagine this can get very hard to do on a simple CPU which is why things tend to have hardware to handle the 3d, and why your computer will play games far better when you have a graphics card in there.
On how they are stored then that gets fun, it is kind of like music in that earlier/weaker devices will tend to use formats very close to the hardware and later/more powerful devices will use higher level formats that are more easily manipulated.
It is a massive subject that many people spend whole careers on so I am not going to be able to cover much in a forum post.

How a GBA ROM could work on a PC. The GBA hardware is known ( http://problemkaputt.de/gbatek.htm ), though even if it was not there are still things you can do, and thus you can then know when it says this then how you do it on a normal PC is that. Do that for enough of the game and you have an emulator, however you can also tell it that when the game tries to load this piece of data then ignore it and instead load this thing off the hard drive and thus you have your PC port.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: BlackDog61 on May 10, 2016, 07:13:45 pm
I could see use of a hex viewer for verifications that things happen at low level like you're programming, but not often and just a viewer.
The editing part would go for very specific cases. Like you have a unit test and it uses vector datain binary, for which you don't have time to writea full editor, so you edit in hex just a few bytes.(Even that doesn't sound like stellar good practice...)
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Spindaboy on May 10, 2016, 07:17:13 pm
So if most NES games were written in 6502 assembly then compiled into Hex, shouldn't there be a way to reverse the process and see the original code?
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: FAST6191 on May 10, 2016, 08:08:50 pm
Yes and no. A disassembler will take the ROM and interpret it at though each byte was an assembler instruction. Equally when writing assembly as a production language you have all sorts of nice perks -- your disassembler will tell you where the value is read from or written to but when writing the code you can assign that location a nice name and say write this to this nice named location. There is no reason for the assembler to include this nice name so it is lost when assembling.

Not everything in a game is assembly code (graphics, text, stats, levels, music...) and not so much in the NES but in other systems you can vary how things are interpreted (you were discussing the GBA earlier -- it quite notably has two very different instruction sets known as ARM and Thumb that it can switch very quickly between basically whenever it likes).

Obviously you can make things slightly more intelligent, and if you can emulate the game then you can also learn a lot from watching it run (if you look in a disassembly window for a GBA emulator it will have said ARM and Thumb but also auto as the emulator knows what mode the game is running in at any given time). That said you are going to be lucky to make something you can assemble again even with a very fancy disassembler. There are a vanishing small number of games that have had their disassembly worked on to the point where you can assemble it again.

In later systems where code was not assembled as much, if at all, and compiled instead then there is a process called decompilation. Right now there are some practical disassembly tools for some machines but not many. I mentioned Turing complete earlier, the same guy (there is a reason he is considered the father of modern computing) also came up with another concept called the halting problem which you might want to look up.
In even later systems there are methods of coding that allow for very good decompilation or code reversal -- things like .NET (used in XNA game studio which Microsoft pushed) can have a lot figured out and things like Java and Python and Lua are often not even compiled at all and instead interpreted.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: jonk on May 10, 2016, 08:39:41 pm
I could see use of a hex viewer for verifications that things happen at low level like you're programming, but not often and just a viewer.
The editing part would go for very specific cases. Like you have a unit test and it uses vector datain binary, for which you don't have time to writea full editor, so you edit in hex just a few bytes.(Even that doesn't sound like stellar good practice...)
I've used a hex editor to just look quickly at something (as a viewer.) I've used a hex editor to patch in specific values where I knew the location and didn't want to waste time editing the source and recompiling it and then running it through a locate tool to create the final binary image. And then to have to re-edit the source code backwards. Or else being earlier forced to copy the entire production directory so you could toss it later. Sometimes, you just don't want to go do all that stuff and the hex editor is the quickest way around the barn. I don't know what you'd consider "stellar practice" in such a circumstance. In my mind, "stellar == expedient" in such a case. You are free to imagine your own path, of course.

May 10, 2016, 08:48:38 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
So if most NES games were written in 6502 assembly then compiled into Hex, shouldn't there be a way to reverse the process and see the original code?
FAST6191 pretty much said what I'd have wanted to say. In fact, he captured an astonishing range with incredible parsimony. I'm still trying to find a novel nook he might have missed! What a vista! I wish I could have written so well.

Read what he said and if you have questions about it, ask them. But the very best way to get a glimmer is to roll up your sleeves and try writing a small program of your own. Just go do it. Things will clear up soon enough.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: BlackDog61 on May 11, 2016, 02:41:24 am
Just one addition to FAST's, taking a step back if you were considering this beyond the NES: several high-level language statements can lead to the same ASM code. There's actually a loss of info when compiling. (I guess philosophically it means machines aren't able to comprehend all the subtleties we say. Or maybe it's the reverse. ;D)
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: STARWIN on May 11, 2016, 09:23:11 am
In later systems where code was not assembled as much, if at all, and compiled instead then there is a process called decompilation. Right now there are some practical disassembly tools for some machines but not many. I mentioned Turing complete earlier, the same guy (there is a reason he is considered the father of modern computing) also came up with another concept called the halting problem which you might want to look up.

I think the halting problem is irrelevant. If there is a problem it is about brute force being slow more than something being impossible.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: jonk on May 11, 2016, 01:17:27 pm
I think the halting problem is irrelevant. If there is a problem it is about brute force being slow more than something being impossible.
Yeah, that was the one thing he said where I wasn't sure why he said it.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: FAST6191 on May 11, 2016, 02:09:41 pm
That paragraph was more of a curio, and natural follow on from all the rest if we were still talking about systems using 3d, and that in particular was meant more as a "if you are curious then..." rather than I am too lazy to type it out, though I take your points.
If you find a good explanation (the predictive branching the core2 stuff brought to the fore does well here)  it covers code flow/execution paths pretty well, or indeed does fairly well at bridging high level loops/branches with low level concepts. Save for recursion it is not as bad as pointers for messing with heads of new programmers but I have seen people struggle, and equally if HTML is to be considered a large part of programming experience of the OP then it is not like such endeavours would have covered loops all that much.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: jonk on May 11, 2016, 02:52:51 pm
That paragraph was more of a curio, and natural follow on from all the rest if we were still talking about systems using 3d, and that in particular was meant more as a "if you are curious then..." rather than I am too lazy to type it out, though I take your points.
I was (and am) impressed by the sheer scope of your response. I couldn't really add anything that you hadn't at least started to say something about. I could expand. But what's the point? But the "halting problem" was the only "which part isn't like the others?" to me. Doesn't make it wrong. I just wasn't sure why it was there.

If you find a good explanation (the predictive branching the core2 stuff brought to the fore does well here)  it covers code flow/execution paths pretty well, or indeed does fairly well at bridging high level loops/branches with low level concepts. Save for recursion it is not as bad as pointers for messing with heads of new programmers but I have seen people struggle, and equally if HTML is to be considered a large part of programming experience of the OP then it is not like such endeavours would have covered loops all that much.
I'm working heavily on writing scalable software. This means (in broad strokes) that using a 4-core CPU will provide twice the performance as using a 2-core one would, and that using a 16-core CPU will provide 8 times the performance. It sounds "obvious," to those without much experience here. But almost everyone using .NET and threads really have no clue, here. They imagine that using separate threads does the trick. But it doesn't. Especially if they have any important data structures for which they use any kind of lock. All that really happens with a very expensive server board sporting four 16-core CPUs is that they have 64 cores all tied up at the same serializing bottle-neck waiting for the same resource and the whole thing performs not so much better than a single core. Look up software transactional memory to see a part of what I'm doing with x86 servers. There are other important theoretical aspects (data structure design figures extremely important here.) But if you design things for scalability, then you actually can achieve quite a lot. I know of only a handful of people doing this, though. Most just use the usual means provided by the usual .NET facilities and honestly have no clues at all. And as a result, their products don't really perform when throwing lots of cores at it. It really takes a new mindset to get that to work well.

[I worked on chipset testing and CPU design at Intel, so I'm pretty well aware of branch prediction and most of the other details operating in the x86 family. Up to the P2, anyway. Those things did single-cycle parallel decode of up to three instructions, converting them to RISC instructions, the use of a ROB or re-order buffer, out-of-order execution, branch prediction, multiple functional units that could be used in parallel (two integer ALUs, two floating point ALUs, etc), and how these things were "retired" as well. (And some of the "approaches" used to deal with sad, earlier design limitations -- the push pop floating point stack, which was a serious bottle-neck itself, for example.) There's a lot more to all this. But probably no point writing about it here.]

Anyway, things are interesting today. I'm enjoying all this.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Spindaboy on May 11, 2016, 05:26:55 pm
Wow. I understood almost none of that so I hope it's not too important for NES hacking!  :laugh:
So do modern systems such as the Wii U run on common code like Javascript?
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: Klarth on May 11, 2016, 05:34:18 pm
But if you design things for scalability, then you actually can achieve quite a lot. I know of only a handful of people doing this, though. Most just use the usual means provided by the usual .NET facilities and honestly have no clues at all. And as a result, their products don't really perform when throwing lots of cores at it. It really takes a new mindset to get that to work well.
Scalability is huge in scientific computing where most programs are made in FORTRAN or C/C++. Unfortunately, a lot of the algorithms in the field I published in (computational quantum chemistry) are not able to perform efficiently in parallel past 64 or 128 cores, depending on what your definition of "efficient" is. So institutions that do this work typically buy a supercomputer with 1000+ cores and split up the server into multiple jobs across multiple research groups. This is a bit frustrating because it's a much more expensive approach than what would be possible with GPU or even Intel Xeon Phi if the algorithms could work efficiently in parallel. Additionally, calculations can take days or weeks even on a large number of CPU cores.
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: jonk on May 11, 2016, 05:41:19 pm
Wow. I understood almost none of that so I hope it's not too important for NES hacking!  :laugh:
If you are referring to my writing, no.... software transactional memory, threads, locks, and so on have almost nothing at all to do with NES hacking!  :)

So do modern systems such as the Wii U run on common code like Javascript?
I honestly don't know. I have a Wii U here. But I've not considered what game developers (isn't the Wii U now cancelled for the US?) use to write code for it. This link, Develop games for Wii U with Unity or HTML5 (https://wiiu-developers.nintendo.com/), suggests that they use Unity for their engine. (They pay the license fees.) That means C++ to me as the language you need to know about. Not JS. But what do I know? I've not attempted any Wii U code. Looks interesting, because Nintendo doesn't charge a fee. (Though they say you need to buy a development system, which costs money, and that you need to prove to them that you can keep what you get from them confidential.)

May 11, 2016, 05:46:51 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
Scalability is huge in scientific computing where most programs are made in FORTRAN or C/C++. Unfortunately, a lot of the algorithms in the field I published in (computational quantum chemistry) are not able to perform efficiently in parallel past 64 or 128 cores, depending on what your definition of "efficient" is. So institutions that do this work typically buy a supercomputer with 1000+ cores and split up the server into multiple jobs across multiple research groups. This is a bit frustrating because it's a much more expensive approach than what would be possible with GPU or even Intel Xeon Phi if the algorithms could work efficiently in parallel. Additionally, calculations can take days or weeks even on a large number of CPU cores.
Fortran would be the go-to language. C and C++ don't offer the the same "guarantees" that FORTRAN offers, so optimizers for C/C++ are hampered in what they can do. (Used to be, anyway. See Why Physicsts Still Use Fortran (2015) (http://www.moreisdifferent.com/2015/07/16/why-physicsts-still-use-fortran/). Plus, a lot of early work went into FORTRAN. (For example, the Bulldog compiler for VLIW architectures investigated a host of interesting and useful ideas using Fortran.)
Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: dougeff on May 11, 2016, 06:34:40 pm
Re:Unity

I thought unity games  were programmed in C# and/or JavaScript.

I'm still learning C#, with hopes that one day I can program a game in unity. (C# is very similar to C++).

Title: Re: Do Game Developing Companies rely upon Hex Editors?
Post by: jonk on May 11, 2016, 06:44:50 pm
Re:Unity

I thought unity games  were programmed in C# and/or JavaScript.

I'm still learning C#, with hopes that one day I can program a game in unity. (C# is very similar to C++).
You may be right about that. Unity is new to me. So thanks for the clue! I only looked at a line or two from a screen shot. So I'm sure it's my mistake and I should have looked further.