News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Do Game Developing Companies rely upon Hex Editors?  (Read 4301 times)

linkncb16

  • Restricted Access
  • Full Member
  • *
  • Posts: 159
  • They actually updated this site. Hallelujah.
    • View Profile
    • Patreon
Do Game Developing Companies rely upon Hex Editors?
« on: May 10, 2016, 05:05:49 pm »
As I have found for myself, hacking certain aspects of a ROM using a Hex Editor are impracticable and sometimes borderline impossible (especially if you don't know the location of what you want to change). So my question is, when making a game are the programmers forced to have every offset and every possible value for that offset written down, or do they have their own utilities that enable quick game editing? Some examples would be a graphics, map, or text editor. All of those make editing a game so much easier. It just doesn't seem realistic for anyone to be able to code an entire game solely in Hex.
Final Fantasy Redux is complete! Download

dougeff

  • Sr. Member
  • ****
  • Posts: 359
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #1 on: May 10, 2016, 05:14:51 pm »
The only reason you need to use a hex editor, is because you don't have access to the source code. If you had access to it, you wouldn't have to keep track of any file/object/graphics location... it would just be included in the project, and the compiler would keep track of everything.
nesdoug.com -- blog/tutorial on programming for the NES

Bregalad

  • Hero Member
  • *****
  • Posts: 2636
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #2 on: May 10, 2016, 05:15:16 pm »
No. Games were made using assembly, and custom tools. An hex editor is not required for making a game - and building up a game is different than hacking an existing ROM.

Klarth

  • Sr. Member
  • ****
  • Posts: 484
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #3 on: May 10, 2016, 05:21:38 pm »
Programmers/designers are not using hex editors for any reason while developing videogames. Having the source code and being able to actively debug the program is much more powerful. File formats have their offsets "hidden" to the programmer, unless they specifically want to make code that is hard to understand. You program routines to load/save your data and don't keep track of offsets unless there's a system performance reason. The compiler/linker will dynamically assign offsets as the game is being built in all modern game workflows.

A hex editor is a crude tool as you know. They're only used as a last resort in data modification because recreating a full level editor is a monumental task.

linkncb16

  • Restricted Access
  • Full Member
  • *
  • Posts: 159
  • They actually updated this site. Hallelujah.
    • View Profile
    • Patreon
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #4 on: May 10, 2016, 05:29:14 pm »
So the source code that is used, is that similar to a language like Java or HTML that is easily readable? Are all NES games being built off the same base code that is unique to the NES? Or... would that be like an OS or something....?
Final Fantasy Redux is complete! Download

Bregalad

  • Hero Member
  • *****
  • Posts: 2636
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #5 on: May 10, 2016, 05:34:42 pm »
So the source code that is used, is that similar to a language like Java or HTML that is easily readable? Are all NES games being built off the same base code that is unique to the NES? Or... would that be like an OS or something....?
HTML is not a programming language.

NES games were programmed mostly in 6502 assembly. Modern games are programmed in high level programming languages, such as Java (if there is GB of memory used for nothing, long loading times and occasonial random crashes, it's usually a sign Java is there ;) ).

The is no Operating System.

Chronosplit

  • Hero Member
  • *****
  • Posts: 1329
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #6 on: May 10, 2016, 05:40:41 pm »
Hex Editors are used as last resorts.  Even in cases where a port is made without the source code.

For example, this seems to be a persistent thing said about Square-Enix.  Final Fantasy VI's PC port is more-or-less a GBA ROM, but all of it's new assets are on the outside.  One could say this is an extremely intensive hack, but they went out of their way to not touch the cart itself.

linkncb16

  • Restricted Access
  • Full Member
  • *
  • Posts: 159
  • They actually updated this site. Hallelujah.
    • View Profile
    • Patreon
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #7 on: May 10, 2016, 05:41:56 pm »
HTML is not a programming language.

What? O.O

This is kind of off topic, but how do 3D models work? I understand that sprites are just an assembly of bits on the screen, but how does a game read a model of something? How is that data even stored?

May 10, 2016, 05:43:42 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
Hex Editors are used as last resorts.  Even in cases where a port is made without the source code.

For example, this seems to be a persistent thing said about Square-Enix.  Final Fantasy VI's PC port is more-or-less a GBA ROM, but all of it's new assets are on the outside.  One could say this is an extremely intensive hack, but they went out of their way to not touch the cart itself.
But if two systems run off different engines, how could a port work correctly without being touched? How could the PC port even interpret the original code of the GBA ROM?
Final Fantasy Redux is complete! Download

VicVergil

  • Hero Member
  • *****
  • Posts: 712
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #8 on: May 10, 2016, 05:57:16 pm »
Yes. It happens a lot, actually.

The official localization for Magical Knight Rayearth (Saturn) for example had the staff resort to hex editors to insert some of the text. Nintendo and Capcom use hex editors a lot too for their retro re-releases (this means lots of cases of just replacing stuff with spaces). In the early nineties, lots of developers would actually insert messages meant for people opening the ROM images with hex editors.

Gemini

  • Hero Member
  • *****
  • Posts: 2011
  • 時を越えよう、そして彼女の元に戻ろう
    • View Profile
    • Apple of Eden
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #9 on: May 10, 2016, 06:00:09 pm »
But if two systems run off different engines, how could a port work correctly without being touched? How could the PC port even interpret the original code of the GBA ROM?
FF6 on Android and iOS uses the original SNES code converted as macros, where each opcode corresponds to a C++ function. The ROM is used to drag whatever data the original code is still addressing.
I am the lord, you all know my name, now. I got it all: cash, money, and fame.

freem

  • Jr. Member
  • **
  • Posts: 19
  • ɯǝǝɹɟ
    • View Profile
    • AJWorld
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #10 on: May 10, 2016, 06:00:57 pm »
I was going to say that I thought Martin Hollis did it on Goldeneye, but he wrote a tool to extract the GZipped data instead.

dougeff

  • Sr. Member
  • ****
  • Posts: 359
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #11 on: May 10, 2016, 06:02:45 pm »
Quote
Are all NES games being built off the same base code that is unique to the NES?

Atari (computers and console), Apple II, NES, Commodore 64, and a few others, were programmed in 6502 assembly.

I could be wrong, but i think officially licensed NES developers were given DEV KITs, which included detailed explanation of how to program NES games (poorly translated from Japanese).
nesdoug.com -- blog/tutorial on programming for the NES

jonk

  • Sr. Member
  • ****
  • Posts: 273
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #12 on: May 10, 2016, 06:10:12 pm »
So the source code that is used, is that similar to a language like Java or HTML that is easily readable? Are all NES games being built off the same base code that is unique to the NES? Or... would that be like an OS or something....?
First off, I'm not a game-writing expert. Though one game company did make me an offer for a job, the day following a day-long interview there. My expertise is in designing hardware and writing embedded software for scientific and commercial instrumentation. That said, there is a great deal of similarity. What I do is sometimes called "bare metal programming." Which is to say, you really need to apprehend hardware design very well and your programming deals directly with the hardware in very intimate ways. You must understand how things work to do that. And your programming skills are often quite different from those required for, say, Windows programming under .NET.

The work I do is going to depend upon the microcontroller and the other limitations imposed on the design. If I have very little RAM and ROM to work with (for competitive reasons I won't dwell on right now) and I won't be using libraries of code developed by other folks and forced onto my application for other (schedule) reasons, then I will probably be writing the application entirely in assembly. I can use semantics at that level which simply don't exist with any other language. And it may make sense to do that. I would tend to expect NES games to be written mostly in assembly code, as a slightly-educated guess. However, if there is more room to work with or if there are software libraries written in some other language (like C), then I will use a mixture of C and assembly... using assembly where needed and C everywhere else. I almost never have a project that can be entirely done in C and have never been exposed to a project using Java, for example. But if I were doing an embedded project that required the use of video and audio codecs and specialized flash file system designs where I wasn't permitted to write them all myself from scratch (likely, I would not be allowed to duplicate those "wheels" if they already are available for a reasonable cost), then I'd use the languages appropriate to those libraries and appropriate to the project requirements.

I am finding C being used with SNES development. But I've just started looking over the code, too. But what little I've learned so far suggests that at least some significant portion of SNES games were written as a C+assembly mixture. If you get a chance, go look on the web for some examples of C code. That will give you a good idea about what that kind of coding looks like.

Texture maps are often developed through the use of specialized tools to help artists. The same is true for sprites, and pretty much anything where you are having digital artists do their work. Sometimes, digital artists will use commercial off-the-shelf (COTS) software to do that job, producing standarized file formats that the game software writers will then either use directly, or else after some other conversion/compression steps. The tool chain can be quite long, in fact. Some of the tools (and steps) may be the result of captive programming employees working as part of the game team. Some may be COTS software. The exact details will vary by project, project size, team experiences, and a host of other factors all of which enter into the early planning stages. (I haven't even mentioned the musicians, yet.)

The game software itself will usually be running stand-alone. This means, without an operating system environment. Sometimes, a limited operating system will be written, designed to the game requirements. Sometimes, a COTS operating system will be used. Sometimes, no operating system at all, instead carefully crafting the interrupt system and timers and other related hardware so that the software can operate "as if" lots of stuff happens in parallel even though everything really is happening one instruction at a time. Some game systems will have two CPUs, though. I suppose some will have even more than that. And of course, they may also be "multi-core." Those tend to require operating system support of some kind.

HTML coding is not at all similar. In fact, I look at HTML more like a kind of "set of word processor commands" than I do a programming language.

Hex editing is a tool I'd probably still have on hand. But I probably wouldn't use it much. It's one of those things you might use if you know exactly what you are doing and need to do a "quick test" without having to re-run your entire compiler tool chain just to find out. So I'd want one present if something like that came up. But I wouldn't expect it to be part of a regular routine. It's an exception-based tool, I guess. But really handy when you have to have it.

An equal right to an opinion isn't a right to an equal opinion. -- 1995, me
Saying religion is the source of morality is like saying a squirrel is the source of acorns.  -- 2002, me

FAST6191

  • Hero Member
  • *****
  • Posts: 2560
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #13 on: May 10, 2016, 06:36:32 pm »
[HTML is not a programming language]
What? O.O

This is kind of off topic, but how do 3D models work? I understand that sprites are just an assembly of bits on the screen, but how does a game read a model of something? How is that data even stored?

May 10, 2016, 05:43:42 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
But if two systems run off different engines, how could a port work correctly without being touched? How could the PC port even interpret the original code of the GBA ROM?

To be a programming language most would argue you have to be what is known as Turing complete. HTML traditionally is not this and was not intended to be, I am not sure what might go with some things for HTML5 though. When web devs needed this they would look to other things like php, ASP.net or on the client side with Javascript, flash, java, activex or that sort of thing. Learning HTML and expecting to be able to program is a bit like reading a dictionary and expecting to know a language -- it will probably help with several concepts but it is going to leave massive gaps.

3d models work in various ways. I am not sure how I want to describe them, and also where I would want to bring in textures and animations and cameras and maths relating to it all. There are probably three main types, those being point cloud (mainly for medical and 3d scanning), vertices (you have points, draw lines between them and you have your model) and bones (like vertices but instead of points on the nominal outside then you have things running along the inside and define how far out it goes from them. Textures also play a role here as you can improve low polygon count meshes with a texture by various means, modern textures are insanely complicated as well in what they can do. Animations in 3d then actually move the models (be it point cloud, vertices or bones) rather than just swap things out very quickly or slide/rotate/scale them around the screen like in a lot of 2d animation. As you might imagine this can get very hard to do on a simple CPU which is why things tend to have hardware to handle the 3d, and why your computer will play games far better when you have a graphics card in there.
On how they are stored then that gets fun, it is kind of like music in that earlier/weaker devices will tend to use formats very close to the hardware and later/more powerful devices will use higher level formats that are more easily manipulated.
It is a massive subject that many people spend whole careers on so I am not going to be able to cover much in a forum post.

How a GBA ROM could work on a PC. The GBA hardware is known ( http://problemkaputt.de/gbatek.htm ), though even if it was not there are still things you can do, and thus you can then know when it says this then how you do it on a normal PC is that. Do that for enough of the game and you have an emulator, however you can also tell it that when the game tries to load this piece of data then ignore it and instead load this thing off the hard drive and thus you have your PC port.

BlackDog61

  • Hero Member
  • *****
  • Posts: 784
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #14 on: May 10, 2016, 07:13:45 pm »
I could see use of a hex viewer for verifications that things happen at low level like you're programming, but not often and just a viewer.
The editing part would go for very specific cases. Like you have a unit test and it uses vector datain binary, for which you don't have time to writea full editor, so you edit in hex just a few bytes.(Even that doesn't sound like stellar good practice...)

linkncb16

  • Restricted Access
  • Full Member
  • *
  • Posts: 159
  • They actually updated this site. Hallelujah.
    • View Profile
    • Patreon
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #15 on: May 10, 2016, 07:17:13 pm »
So if most NES games were written in 6502 assembly then compiled into Hex, shouldn't there be a way to reverse the process and see the original code?
Final Fantasy Redux is complete! Download

FAST6191

  • Hero Member
  • *****
  • Posts: 2560
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #16 on: May 10, 2016, 08:08:50 pm »
Yes and no. A disassembler will take the ROM and interpret it at though each byte was an assembler instruction. Equally when writing assembly as a production language you have all sorts of nice perks -- your disassembler will tell you where the value is read from or written to but when writing the code you can assign that location a nice name and say write this to this nice named location. There is no reason for the assembler to include this nice name so it is lost when assembling.

Not everything in a game is assembly code (graphics, text, stats, levels, music...) and not so much in the NES but in other systems you can vary how things are interpreted (you were discussing the GBA earlier -- it quite notably has two very different instruction sets known as ARM and Thumb that it can switch very quickly between basically whenever it likes).

Obviously you can make things slightly more intelligent, and if you can emulate the game then you can also learn a lot from watching it run (if you look in a disassembly window for a GBA emulator it will have said ARM and Thumb but also auto as the emulator knows what mode the game is running in at any given time). That said you are going to be lucky to make something you can assemble again even with a very fancy disassembler. There are a vanishing small number of games that have had their disassembly worked on to the point where you can assemble it again.

In later systems where code was not assembled as much, if at all, and compiled instead then there is a process called decompilation. Right now there are some practical disassembly tools for some machines but not many. I mentioned Turing complete earlier, the same guy (there is a reason he is considered the father of modern computing) also came up with another concept called the halting problem which you might want to look up.
In even later systems there are methods of coding that allow for very good decompilation or code reversal -- things like .NET (used in XNA game studio which Microsoft pushed) can have a lot figured out and things like Java and Python and Lua are often not even compiled at all and instead interpreted.

jonk

  • Sr. Member
  • ****
  • Posts: 273
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #17 on: May 10, 2016, 08:39:41 pm »
I could see use of a hex viewer for verifications that things happen at low level like you're programming, but not often and just a viewer.
The editing part would go for very specific cases. Like you have a unit test and it uses vector datain binary, for which you don't have time to writea full editor, so you edit in hex just a few bytes.(Even that doesn't sound like stellar good practice...)
I've used a hex editor to just look quickly at something (as a viewer.) I've used a hex editor to patch in specific values where I knew the location and didn't want to waste time editing the source and recompiling it and then running it through a locate tool to create the final binary image. And then to have to re-edit the source code backwards. Or else being earlier forced to copy the entire production directory so you could toss it later. Sometimes, you just don't want to go do all that stuff and the hex editor is the quickest way around the barn. I don't know what you'd consider "stellar practice" in such a circumstance. In my mind, "stellar == expedient" in such a case. You are free to imagine your own path, of course.

May 10, 2016, 08:48:38 pm - (Auto Merged - Double Posts are not allowed before 7 days.)
So if most NES games were written in 6502 assembly then compiled into Hex, shouldn't there be a way to reverse the process and see the original code?
FAST6191 pretty much said what I'd have wanted to say. In fact, he captured an astonishing range with incredible parsimony. I'm still trying to find a novel nook he might have missed! What a vista! I wish I could have written so well.

Read what he said and if you have questions about it, ask them. But the very best way to get a glimmer is to roll up your sleeves and try writing a small program of your own. Just go do it. Things will clear up soon enough.
« Last Edit: May 10, 2016, 08:48:38 pm by jonk »
An equal right to an opinion isn't a right to an equal opinion. -- 1995, me
Saying religion is the source of morality is like saying a squirrel is the source of acorns.  -- 2002, me

BlackDog61

  • Hero Member
  • *****
  • Posts: 784
    • View Profile
    • Super Robot Wars A Portable translation thread
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #18 on: May 11, 2016, 02:41:24 am »
Just one addition to FAST's, taking a step back if you were considering this beyond the NES: several high-level language statements can lead to the same ASM code. There's actually a loss of info when compiling. (I guess philosophically it means machines aren't able to comprehend all the subtleties we say. Or maybe it's the reverse. ;D)

STARWIN

  • Sr. Member
  • ****
  • Posts: 449
    • View Profile
Re: Do Game Developing Companies rely upon Hex Editors?
« Reply #19 on: May 11, 2016, 09:23:11 am »
In later systems where code was not assembled as much, if at all, and compiled instead then there is a process called decompilation. Right now there are some practical disassembly tools for some machines but not many. I mentioned Turing complete earlier, the same guy (there is a reason he is considered the father of modern computing) also came up with another concept called the halting problem which you might want to look up.

I think the halting problem is irrelevant. If there is a problem it is about brute force being slow more than something being impossible.