[M-Tee]

Introduction

         This post begins a low-learning curve workshop in which I will walk through the steps of creating fairly extensive graphical modifications of an NES game, from setting up a workspace through publishing your completed work through RHDN. Aside from beginners, this should reveal some tips & tricks for more experienced folk, as well as provide a solid foundation for those interested in producing NES-era graphics for other purposes such as indie game design or animation.
   This workshop will reflect my process, which has modified over time through trial and error. It is not how everyone works; therefore, I fully encourage feedback on the effectiveness of my process and explanations.
     Also, keep this thread in mind anytime someone comes around asking those über-newb questions or obviously trying to bite off more than they can chew for their first project. (I certainly would have benefited from the same advice when I first came around.)

Table of Contents
(will be updated with each new addition)
Getting Started
Setting Up Your Software
Customizing Your Photoshop Workspace

Getting Started

     To begin with, you will need to acquire the programs and files with which we will work. Download and install the following files, and organize them into an easily accessible directory. (Unfortunately, most of the tools of the trade are Windows-only, so we will be working in a Windows environment. For others, I apologize. )

• FCEUX - The emulator of choice for  ROM hacking due to its list of features.
• yy-chr - A graphics viewer and editor. It is not the most recent version. However, the manner in which this version handles pasting from outside programs makes it better suited for our purposes.
• Lunar IPS - Although the legality of ROM hacking is in question, the distribution of ROMs which contain copyrighted data is not. Therefore, to distribute your work without distributing ROMS, you will need to produce patches, files which contain only the original data created by yourself and the instructions on how to apply these changes to the original ROM file. This program is one of many that can create and apply these patch files. You will not only need it to distribute your work, but also to play any work distributed by others.
• HashTab - File identification software. Used to ensure use of correct rom versions. More info here.

     Next, if you don't have access to web hosting, I suggest signing up for an account through TinyPic to be able to upload and post your screenshots of work-in-progress.

Selecting a Game

     The purpose of this workshop is to modify a game. Therefore, you need to select one and come up with an idea for your hack. I'm sure you've already got some totally awesome hack in mind. Now is not the time for it. Now is the time to choose a simple game so that we may produce a thorough and complete hack in a reasonable amount of time. The confidence and experience you will gain by doing so will enable you to handle larger projects better.

Therefore, browse this list of mapperless NES titles (they are the smallest and best-suited for instruction). Youtube the titles to watch gameplay and brainstorm some ideas for hacks. Keep in mind the scope of the changes you will be making will be complete graphics and text. (This workshop will not cover music and programming behaviors, primarily because they are skills not in my repertoire.)  Once selected, use your own ingenuity and the worldwide web to acquire the rom.

Next Up: Setting Up Your Software

[VicVergil]

Actually, I do have some degree of knowledge in the matter, since FCEUX does all the work (Discovered the map viewer recently only recently I'd prepare a pic like this

 and paste it with a tile editor in place of that particular 16x16 pattern table loaded (or part of it) then write down all of it manually
Anyways, later it's the same, I just look the PPU hex code sequence for that particular tilemap in a ROM search and I modify manually with hex editors. That part is tedious though, do you know of any tool that could automate it ?

Besides, I'm totally in the black when it comes to altering the palette for a given part of the title screen.
I'd be interested in learning how to alter palettes.

That aside, do you have a trusty tool/emulator that has similar options for SNES games?

[naxis]

Are you planning to do something similar for the SNES? I sure could use a workshop on that.

[M-Tee]

Setting Up Your Software

     Now that everything's downloaded and installed or organized, you'll need to adjust the settings for your programs across the board. Although you're certainly not obligated to follow the same setup I have, by understanding why I've chosen it, you can wisely choose your own.

FCEUX

(NOTE: I generally use NEStopia (a different emulator) to play games for pleasure. That way, I am able to test hacks on two emulators, and can set up FCEUX settings such as zoom and directories primarily for these purposes with different settings in NEStopia for entertainment.)

Open FCEUX and address the following settings:

• Config>Video>Windowed Settings>Size Multiplier
  
  By default, FCEUX displays video at 100% which I find uncomfortably small. You'll also want to keep the screen at a reasonable size because we will often have three or more windows open at the same time. Therefore, I choose 2 and 2 for X and Y respectively for 200% zoom.
  
• Config>Directories
  Set up convenient directories for screenshots, cheats, savestates, and roms. I like to keep all of my in-progress work separate from my normal rom collection, so I ensure that none of these directories are shared with my NEStopia settings.
• Config>Input>Port 1 & Port 2
  Since you'll need to test two-player mode, you'll need to configure the controls for both. I use keyboard input for FCEUX, but when I play for entertainment, I use a joystick in NEStopia. I find picking up and putting down my controller each time I want to do something in FCEUX to be a waste of time. Use what's comfortable for you. I like to put player one on arrow keys with a and s for select and start, z for A, and x for B (some people prefer to invert this to mimic the orientation of the NES pad. I prefer it this way so that the index finger controls attack in most games, like in arcades). I put player two on numpad (5 instead of 2 for down to mimic the spacing of the arrow keys), d and f for select and start, c for A, and v for B. This allows me to physically control both players at once without having to move my hands.
  
• Display>Input Display>Two Player
  
  This displays the controller icons located on the bottom-left which light up with input. Useful for playtesting.
  Note the red vertical bars in the bottom right which indicate that emulation is paused (controlled with the Pause/Break key). It actually pauses the emulation, not like pausing the game with the controller. Other useful keys to know are + and - for increase and decrease speed respectively. The former is useful for quickly moving through the game to find and check changes, the latter is useful for previewing and analyzing animation.
  
  In the image above, you can also see the Display options Graphics: BG and GRAPHICS: OBJ. These can be toggled to display backgrounds and sprites respectively. See the below images:
  
     
• Config>Palette>Use Custom Palette
  Due to its non-RGB nature, the NES palette is a highly debated topic. You may certainly use FCEUX's default. However, I use the nestopia_yuv.pal file located in FCEUX's palette folder (also available as nestopia.pal in the Graphics Toolkit download.) You'll notice in the options that FCEUX also supports an NTSC emulation mode, which some also prefer.

Palette Controversy

     Above is a series of screenshots taken of commonly used palettes. I have added the hex codes for each color above it. (The NES has a palette consisting of 64 colors, each represented by a two-digit hex value. More on that later.)

Points of Contention

• Value 23: The reason I was initially so opposed to FCEUX's palette was that older versions of FCEU and FCEUX display this as blue. However, that has since been corrected as of FCEUX v. 2.2.0.
  
• Values 08–38: The -8 range of colors is probably the most hotly fought over area of the palette. Some palettes display them as shades of tan, others tinted green. It is a fact that your audience will be using both. Therefore, I choose to err on the green side. My reasoning is that anything that I intend to depict as greenish tan, such as dirt, soil, etc. would not be unusual displayed as orangish tan. However, the more orangish hues could very well be used to depict shades of tan flesh or other things which would look unusual if green.
• Values  06–36 and 07–37: Differing palettes display these ranges of colors in varying hues between orange and red, producing differing warmths of brown.
• Value 05: In FCEUX's palette, is nearly indistinguishable from 06 with a very large jump in hue between the -4 and -5 ranges of colors.
• Value 0D: Pure black. Word has it that Nintendo warned developers against utilizing this color due to potential damage to older model televisions. 0F is used as industry-standard black.
• Shades of Gray: If you choose to read about the NES palette, you'll find posts online claiming that the NES has up to six shades of gray (00, 10, 20, 1D, 2D, & 3D). 00 and 10 are perfectly fine. If you see gray in a licensed NES title, it is almost certainly one of these (I have not encountered anything in the -D range outside of pirated games and hacks, however, I haven't necessarily sought them out.) Value 20 is often described as  "a very light shade of gray," but it is visually indistinct from white. I tend to use 20 instead of 30 out of habit for white, many licensed games do as well. Either 20 or 30 are fine. Just don't expect to use either as gray or any difference between them. 1D is depicted as the darkest gray by NTSC emulation, but in every other palette, it is without a doubt black. Therefore, it is unusable. 2D and 3D are commonly used by other artists to add levels of gray. However 2D's relative value to 00 is inconsistent, therefore it cannot be reliably utilized with 00 on the same screen, meaning there is no reason to use it instead of 00. The same is true for 3D in relation to 10.
• Saturation: These colors are all designed to approximate CRT color, of which there was never an industry standard. Much more so than now, each model required various adjustments to meet not only an acceptable range of accuracy, but also to adhere to the aesthetic of its owner. I, for one, have never known anyone to keep their color/intensity/saturation at maximum level on their television. NEStopia has the most reasonable level of saturation in regards to my personal analog television preferences.
• Darkness: Some palettes display the -0 range as significantly darker than others.

YY-CHR

     The version of YY-CHR you use affects color assignment when pasting from outside sources. The colors in the hex chart I've included have been pulled from the photoshop swatches file that accompanies. Those colors paste accurately into the version of YY-CHR linked in the OP as colors 0, 1, 2, and 3 respectively (more on that in the later). Older versions have required more drastic contrast such as pure 255 red, green, blue, and black.
     Also, if your system language is not English, YY-CHR will attempt to display its interface in your system language. This is corrected with a filename change, explained in a text file included with the program.

Next Up: Customizing Your Photoshop Workspace

Actually, I do have some degree of knowledge in the matter, since FCEUX does all the work (Discovered the map viewer recently only recently I'd prepare a pic like this

...

 and paste it with a tile editor in place of that particular 16x16 pattern table loaded (or part of it) then write down all of it manually
Anyways, later it's the same, I just look the PPU hex code sequence for that particular tilemap in a ROM search and I modify manually with hex editors. That part is tedious though, do you know of any tool that could automate it ?

Unfortunately, I do not have a method to automate the typing of the hex. However, I do have some pointers for identifying the addresses. The hex chart which I use (and is included in the Graphics toolkit link) is as follows:



Although not easy to read now, when pasted from Photoshop into the version of YY-CHR available in the OP, the white acts as color 0 (transparent), first digit displays as color 1, the underline as color 2, and the second digit as color 3. Meaning that the chart not only provides you information of what tile is being used, but what palette is assigned where.  Essential information when palettes may share a color or more, which is very common.

Besides, I'm totally in the black when it comes to altering the palette for a given part of the title screen.
I'd be interested in learning how to alter palettes.

As for background palettes, you'll want to find information on the PPU and on Attributes. You can search and track that info down (there is good documentation available), or wait until I cover the topic further down the road (it is on my agenda).

Are you planning to do something similar for the SNES? I sure could use a workshop on that.

That aside, do you have a trusty tool/emulator that has similar options for SNES games?

I haven't tried any SNES romhacking since high school, more than decade ago. Quite frankly, it doesn't interest me. The limitations of the NES visuals are what attract me so strongly to the hobby.

[VicVergil]

I heard that Nintendo did have arcade machines that supported RGB palette output, so they do have a fixed color value for each. Wouldn't that one be the "official" palette?

[M-Tee]

Customizing Your Photoshop Workspace

     Some may oppose the heavy reliance on Photoshop this workshop will have (I suspect roughly 50%). However, I feel that it is essential for efficiency to produce graphics in an external program with a customizable palette, layers, and basic animation features. I build and preview everything in Photoshop and then import it through YY-CHR. I suggest version CS4 or newer (CS4 introduced an optional pixel grid at 500% or more zoom, very useful for our purposes.)
     I understand that GIMP and Paint.net both have large user bases, are legally free, and I'm sure for most purposes, will suffice. However, I'm 15+ years into Photoshop for school, personal, and work purposes though, so I will be unable to provide assistance specific to free alternatives.

Customizing Your Workspace

(Click for full size)
TIP: The above settings for text will produce characters that take exactly one tile each. Useful for judging how many characters can fit within an area.

   We will build a custom workspace (arrangement of windows, keyboard shortcuts, and settings) for NES graphic work. First, we need to decide which windows (aka panes) are useful for us and how to organize them on the screen. Click and drag them to combine or dock where necessary. The following is my layout:

• Left Side:
  
  • Tools - Docked
  • Swatches - Minimized and Docked with small icon previews at minimum width. This allows the colors to be displayed in multiples of four. To load the custom NES palette, alt+click each swatch until the pane is empty. Then select Load Swatches... from the dropdown menu on the top left. Select nesPalette.aco. I made this series of swatches, and have labeled them accordingly with their hex addresses, viewable by hovering.
    
• Right Side: Layers, Histories, and Actions combined. Paragraph and Character combined. Both minimized.
  
• Bottom: Timeline. (Used for previewing animations and making animated gifs to display work in progress.) Minimized.
  

Preferences, Options & Shortcuts

   The following are preferences and tool options to optimize Photoshop for tile-based pixel work. Many of which counteract anti-aliasing, the blurring that occurs when scaling or transforming images, which can prove to be majorly frustrating when unwanted.

Change the following settings through Edit>Preferences:

• General>Image Interpolation:Nearest Neighbor (clean scaling)
  
• Performance>:
  
  • History States: 99 (or higher. This controls how many steps you can undo. For larger print work, 99 would put a seriousdrain on resources. But working a-pixel-at-a-time, the amount of memory required per step is miniscule and the number of steps you'll want to undo when working will be large.
  • Cache Levels: 2 (suggested lower for smaller resolution files with many levels, which is what we're doing)
  • Use Graphics Processor:Checked. (Enables pixel-grid)
• Units & Rulers>Rulers: Pixels
  
• Guides, Grid & Slices:
  
  • Gridline Every: 16 pixels (Creates gridlines for common arrangement of 2x2 tile squares.)
  • Subdivisions: 2 (Produces lighter gridlines for individual 8x8 pixel tiles.)
  • Style: Lines

Select the following tools to change their options:

• Move Tool: Check Auto Select Layer (very useful for selecting layers, only negatively effected by unlocked transparent layers, which you will probably have no need for.
  
• Rectangular & Elliptical Marquee: One is nested under the other. Ensure that Feather is at 0 and anti-alias is NOT checked.
  
• Lasso & Polygonal Lasso: Again, one nested under the other. Uncheck anti-alias.
  
• Magic Wand & Paintbucket: (Paintbucket nested under Gradient) Set Tolerance at 1 (for selecting exact colors) and uncheck anti-alias. Gradient will not be used.
  
• Pencil: Nested under Brush. Select Pencil, decrease size to 1px. Brush will not be of use.
  
• Eraser: Set Mode to Pencil. Decrease size to 1px.
  
• Transform: Not on the toolbar, but under Edit>Free Transform, set Interpolation to Nearest Neighbor
  

Through Edit>Keyboard Shortcuts, set up the following keyboard shortcuts to replace nonexistent or inconvenient defaults:

• File>Save for Web: Ctrl + Shift + s (definitely optional as the default is ctrl + alt + s, but I find Save As to be practically useless 90% of the time, I either want to save my PSD or save a jpg, png, or gif for web view. It is rare that I need to save as a different type, so I swap the more common Save As shortcut for the more useful Save for Web function.)
  
• Edit:
  Undo/Redo: Alt + Ctrl + z (Only toggles last action. Is not useful when working in pixels)
  
  • Step Backward: Ctrl + z (Steps backward through history. Much more useful.)
  • Step Forward: Ctrl + Shift+ z (or Ctrl + y if you're accustomed to ctrl + y for redo in programs like msword)
  • Flip Horizontal: Ctrl + f (replaces Apply Filter, which will not be used for pixel work. X and Y flipping and rotation definitely will, though.)
  • Flip Vertical: Ctrl + Shift + f
  • Rotate 90° Clockwise: Alt + Ctrl + f
  • Rotate 90° Counterclockwise: Alt + Ctrl + Shift + f
• View:
  
  • Grid: Ctrl + h (will be the most commonly used of the show/hide commands, so should have the simplest shortcut)
  • Pixel Grid: Shift + Ctrl + h (only applicable for 500% or more zoom. Also, can be disabled by lack of system resources and other causes.)
  • Guides: Alt + Ctrl + h (Not so commonly used, since our scale is usually small enough for grid lines to serve as visual guides.)
  • Extras: Alt + Shift + Ctrl + h (toggles Grid, Pixel Grid, Guide, and Lassos)
  • Snap: Ctrl + Q (snaps selection to grid when grid is visible. Intense timesaver because you must paste exact size of tiles into YY-CHR. Q was chosen due to left edge proximity and because Alt + F4 performs the same role as File>Quit, the default function for Ctrl+Q)
    
    

Other Useful Shortcuts to Know
(Press shift + the letter to cycle through nested tools.)

• Tools:
  
  • Move: Hold Ctrl
    
  • Hand tool: Hold Spacebar
    
  • Marquees: M
    
  • Magic Wand: W
    
  • Marquees: M
    
  • Pencil: B
    
  • Eraser: E
    
  • Paintbucket: G
    
  • Zoom: Hold Ctrl and scroll mousewheel
  • Increase and Decrease Cursor Size, such as for Pencil and Eraser: [ and ]

• Edit:
  
  • Copy Merged:Ctrl + Shift + c (copies all visible layers instead of only current layer, can be useful for copying sprites to import to YY-CHR if you maintain a white background layer to represent color 0)
• Layer:
  
  • Layer via Copy: Ctrl + j (creates new layer from selection)
  • Merge Layers:Ctrl + e
• Select:
  
  • Select all: Ctrl + a
  • Deselect: Ctrl + d
  • Inverse selection: Ctrl+Shift+i

Once you have everything to your liking, minimize all panes and in the Workspace dropdown near the top-right (by default, I believe it says Essentials), and select New Workspace..., check both options, then give it a suitable name.

Next Up: Viewing and Understanding NES Graphics

I heard that Nintendo did have arcade machines that supported RGB palette output, so they do have a fixed color value for each. Wouldn't that one be the "official" palette?

That would be the PlayChoice-10. Thanks for bringing it to my attention, I've updated the palette images in the post to represent it. It can be seen in NEStopia's RGB palette as well as Nintendulator's PC-10 palette pictured. If the goal were to emulate that system, then it would be the "official" palette. However, there are known discrepancies between it and what would would typically be seen from a normal home console hooked up to a television. 

Nevertheless, defending a specific palette was not my intention. Know that when you release your patch, it will inevitably get a couple of hundred downloads. Those people will be using an assortment of palettes. My goal is to bring the differences between these palettes to attention, so that you can work with all of these limitations and strengths in mind, knowing that if you use color x, it could appear as x, y, or z to different users.

EDIT: I also noticed while double-checking these palettes that blue 23 color in FCEUX was corrected with v. 2.2.0 in Nov. of 2012. Therefore, it should not be an issue anymore, making FCEUX's default palette an acceptable choice. Previous Post updated.