So I'm curious: how can I disable this tile reusing, so that I can have the freedom of allowing sprites to be more uniquely drawn?
This is handled differently in each game. Somewhere in the ROM is some kind of data structure that indicates which 8x8 tiles are used to construct the full sprite (as well as the coordinates to draw them relative to the sprite position). You'll have to find this data to modify it, though again, the exact format of it can vary so I don't really know what to tell you to look for.
Some possible ways of finding it:
- Check a wiki or something (like DataCrystal). LoZ is a pretty popular game so this information may have been found already.
- Use a file corrupter: You can use the PPU viewer in FCEUX to see which tile number represents the 8x8 portion of a sprite. Corrupt the ROM to change occurances of that byte to something else and narrow down which byte is the byte you're looking for.
- Do a back-trace and examine the assembly to see where it's getting the data from. LoZ uses the $02xx page in memory to store its OAM, so watch [interesting] writes to that range to see where it's getting the values from.
Though remember, even if you do this, you still are limited to 256 sprites tiles at any one time.
How can I increase the number of possible tiles
Do you mean 8x8 pixel tiles? Or "map tiles" (floor, sand, statue, block, etc)
The latter would involve completely redesigning the map format and writing all new code to load your new maps. Needless to say this is quite a big job and would require lots of asm work. Plus it would render any map editors completely useless as they wouldn't recognize your new format.
As for the former... it doesn't look like all the CHR is used for dungeons (there are a LOT of blank tiles)... so you might be able to the CHR to put more graphics in, and modify the TSA for the map tiles to make use of them.
(and possibly increase variation of palettes used for tiles within one room)?
You can't have more than 4 BG palettes on-screen at once. <- This is a hard limit of the NES. No way around it*.
In dungeons, LoZ seems to use them like so:
2 palettes for the HUD at the top of the screen (red/pink/orange for the life meter and coin/key images ... and white/blue for everything else)
2 palettes for the actual map
So you can't add more palettes for the map without removing them from the HUD. OR you could have the HUD and map share a palette, but this would have the side-effect of some tiles not "fading out" in dark rooms (or, if you change it so they DO fade out... part of the HUD will also fade out)
I'm thinking one part of the solution might require ROM expansion
ROM expansion is not a magic cure-all. It's not exactly trivial on the NES, and making use of the extra space requires a good amount of asm work anyway, as you need to swap PRG banks around -- which can lead to game-crashing bugs if you don't know what you're doing.
My word of advice here is: if you don't know enough asm to know how to expand the ROM and make use of the expanded space -- you probably don't know enough asm to know whether or not you need to expand the ROM in the first place.
(*you can technically get around it by swapping palettes mid-frame but this is extraordinarily difficult to do properly and is generally not something to be considered)