[firedropdl]

Hey everybody,

So I've learned quite a bit about NES headers in the last few days.  Most importantly would be that they're just about absolutely necessary these days for playablility on emulators, and that they're somewhat unique in that way since many other systems don't have a need for them.


So that brings me to the FDS games.  What's the nature of headers for that system?  I've noticed a few things while looking at instructions and trying to find correct versions for the games.  Some specifically say that the rom MUST be unheadered for the patch.  Most don't mention this.  Some show a pre-patched value of a headerless rom on the page.  Most don't.

Should I be stripping all of the headers before patching these games, or just the ones that specifically say no header?

Are the headers necessary for the FDS?

Will I have to add my own headers somehow after patching non-headered games?

Thanks again!

[Bregalad]

Most importantly would be that they're just about absolutely necessary these days for playablility on emulators, and that they're somewhat unique in that way since many other systems don't have a need for them.

NES headers are crutial in order to know what cartridge hardware to emulate. This is hardly unique. SNES have several carts with different hardware. Emulators can (and usually does) use heuristics on the ROM to determine whether they have a LoROM or HiROM cartridge, and Nintendo had an imposed header in the ROM so emus uses that. But there's no guarantee the header in the ROM is correct, actually you could even have something completely wrong in the ROM header or remove it entierely and have a game that works on hardware - but it wouldn't work in emulators. How hironic.

Many NES ROMs have also a header in the ROM but the format is less standard, and info is largely incomplete and often plain wrong, or missing entierely (about half of games). So this header cannot normally be used to emulate the system.

So that brings me to the FDS games.  What's the nature of headers for that system?

FDS headers are mostly useless and contain no information. Basically their only purpose is to verify that the data following it is effectively a FDS dump. Info about disk sides amount, etc... is found on the disk itself in it's internal header. Which is itself used by the FDS BIOS.

I've noticed a few things while looking at instructions and trying to find correct versions for the games.  Some specifically say that the rom MUST be unheadered for the patch.  Most don't mention this.  Some show a pre-patched value of a headerless rom on the page.  Most don't.

So basically it's the same hell as for the SNES

Should I be stripping all of the headers before patching these games, or just the ones that specifically say no header?

I think if they don't say anything it's with header ? In all cases you should try both and see which one works (like with the SNES), unfortunately there's no real alternative.

Are the headers necessary for the FDS?

No.

Will I have to add my own headers somehow after patching non-headered games?

If an emulator you wish to use only supports headered FDS dumps then yes. I have no idea which emu supports or don't support headerless FDS dumps, but it seems with header is the "standard".

[KingMike]

Literally the only information in an FDS header is the disk size, which can already be inferred from the (.fds) file size.
So they're useless.

[firedropdl]

Cool.  Thanks for the info guys.

So... should I delete the headers before patching the FDS games, KingMike?

Or are the patches written in a way where they expect the header on the file unless they explicitly tell you that you are to use a rom without a header?

Nevermind.  It's going to be a case-by-case basis.  I'll try to document this as I go to take some of the trial and error out of it.

I was just reading the readme for BodyConQuest since I'm having an especially hard time finding the correct two-disk image and that readme has the following text inside:

1) A FDS file.  The file needs to include the standard 16 byte iNES header
   followed by the program disk image data.  With header, the FDS file is
   262016 bytes in size.

   The header should be as follows:

       46 44 53 1A  04 00 00 00  00 00 00 00  00 00 00 00

So I'll just patch the images with the CRC values on the pages and only change anything if I'm directed to.
August 18, 2017, 02:19:21 pm - (Auto Merged - Double Posts are not allowed before 7 days.)EDIT:

Has anybody ever patched BodyConQuest before?

http://www.romhacking.net/translations/1758/

I guess it was super rare to have a 2 disk game for the FDS, so this is kind of awkward to patch. 

The patch page says there are two disks:

ROM / ISO Information:

    Bodycon Quest I - Abakareshi Musume Tachi (Japan) (Disk 1) (Unl).fds
    CRC32: DEC6333D
    MD5: 9C858C062F518534247703A8C0717451
    SHA-1: E2F52D2D34E88C7B02B13E015AC135D19A792F63
    SHA-256: D63C795F6EC1B672D5C4CB69A2DD1DA9A302292C59CBAD2BDB332D958B0CE7CD
    Bodycon Quest I - Abakareshi Musume Tachi (Japan) (Disk 2) (Unl).fds
    CRC32: 6D1F3544
    MD5: 4FA209CD662DAC1A1BF3A51147FED49D
    SHA-1: 3583E29BD8FB8C54B9ED34737595C23EF2FF229D
    SHA-256: 27FC6D67CE0EDD525BC21448ECDF1AE26C77E9A387B9B4C33A93A74A43565DCD


I haven't been able to find a two disk image of this game anywhere and every set seems to treat it as if it were only one disk image.  Any google search attempts that I've tried referencing a second disk come up empty, and this site is pretty much the only hit I get for two disks.

[Disch]

Don't shake the boat.

FDS files have headers.  This is a uniform standard across all emulators.  Keep them.  Use them.  Don't try to make a new standard -- that will only lead to problems.


Other than that, I can't help you with BodyConQuest.


EDIT:  wowzers, 2500 posts!

[firedropdl]

Hey Disch.

The problem comes in when about 5 of the 35 or so FDS translations have specific instructions to use headerless roms.  No instructions about adding them back in that I can find.  So it looks like it's just going to be done according to how the author of the translation designed it on a case by case basis.


I think I have some ideas about BodyConQuest.  It's possible that both images were merged into one image and the header is pointing to that somehow.  I could be completely wrong about that, but I'm going to look into it deeper later.  It will be the last one I work on since it's no doubt going to be the hardest to figure out.

[Disch]

Well this is kind of my point -- the people making patches that work with headerless ROMs are being stupid.  These ROMs have headers -- always.  Don't complicate things.  It's bad enough we have this problem with SNES ROMs.

[Psyklax]

Well this is kind of my point -- the people making patches that work with headerless ROMs are being stupid.  These ROMs have headers -- always.  Don't complicate things.  It's bad enough we have this problem with SNES ROMs.

But with FDS, why?

There are two types of headers in FDS files: the pointless 16 byte ones, and the ones from the original disks which are definitely not pointless. Emulators don't care about the 16 byte ones these days. And as I said in another thread you really shouldn't get worked up over CRCs in FDS files due to both header differences and save games. It'd be frankly ridiculous to say that a file has a bad CRC because it has a different write date and different save games to the "right" file.

Honestly, the focus on CRCs is a bit much for me. You wanna go real hardcore? Examine the IPS files for which bytes are changed (has someone done a program to analyse IPS files? The format is in the docs database) and compare bad ROMs to the verified good dump in GoodTools (not in FDS of course). If the IPS doesn't even touch bytes that are different between ROM versions, this whole CRC business is moot. I recommend something like HxD, my hex editor of choice, to do file comparisons. In the past I've changed bytes inside patches files so it's the same as the verified good dump, in cases where the hacker used an overdump or bad dump.

[firedropdl]

Well this is kind of my point -- the people making patches that work with headerless ROMs are being stupid.  These ROMs have headers -- always.  Don't complicate things.  It's bad enough we have this problem with SNES ROMs.

It certainly makes things more difficult, but I can't say anything about it being stupid.  I have no idea what these guys actually do when they're coding and the reasons why things are done. 

Honestly, the focus on CRCs is a bit much for me. You wanna go real hardcore? Examine the IPS files for which bytes are changed (has someone done a program to analyse IPS files? The format is in the docs database) and compare bad ROMs to the verified good dump in GoodTools (not in FDS of course). If the IPS doesn't even touch bytes that are different between ROM versions, this whole CRC business is moot. I recommend something like HxD, my hex editor of choice, to do file comparisons. In the past I've changed bytes inside patches files so it's the same as the verified good dump, in cases where the hacker used an overdump or bad dump.

Yeah, I know you said the CRCs don't matter much, but I just want to do whatever I can to ensure I'm using the exact rom the author of the patches used.  In most cases I've been able to do that, and even some tricky ones I was able to figure out and put in the other thread the reason why I couldn't initially find them.  I'd say that out of around 347 translations on the NES a miss rate of only 10 is not bad at all(9 including your patch that we figured out).

FDS actually isn't as bad as I thought it would be.... that is, assuming that the patches work correctly with the CRC values that the authors put on the pages and the read-mes.  I'm just assuming they are right now if I've found a headered or headerless match.  I will actually have to test the games after they're patched to see if they work right with or without a header.  I'll document this and make another thread that says exactly which version needs to be used for each FDS game since I'm assuming that there are going to be some patching problems that I won't know about until I actually test the games. 

On the NES we knew that the header is necessary, so when a headerless CRC value was given it was a mistake.  That's not the case on the FDS necessarily since headers don't matter as much and some roms have specific instructions to NOT use a headered rom.  It's really just a crap shoot.  I'm glad there are only 37 of them. 

[Psyklax]

On the NES we knew that the header is necessary, so when a headerless CRC value was given it was a mistake.  That's not the case on the FDS necessarily since headers don't matter as much and some roms have specific instructions to NOT use a headered rom.  It's really just a crap shoot.  I'm glad there are only 37 of them. 

The whole FDS file format was messed up from the start: first they made a header which said "this is an FDS file with two sides" and an emulator says "yes, I can see that already". But even worse, the format removes stuff that is deemed unimportant for emulation, but just replaced it with empty bytes. Why?! Don't know who came up with that bright idea, but it means that no FDS files are truly 100% accurate rips.

Anyway, I've said enough. Godspeed.

[Disch]

On the NES we knew that the header is necessary, so when a headerless CRC value was given it was a mistake.  That's not the case on the FDS necessarily since headers don't matter as much

No No No

It matters just as much because the header is part of the FDS file format!  Headerless CRC values for FDS isos are just as confusing and erroneous as headerless CRC values for NES roms.

I have many complains about the FDS file format -- and it could stand to be improved in a lot of ways.... but again... don't rock the boat.  There is a consistent and uniform standard that literally every single emulator under the sun uses and agrees with.  If you're going to do something that goes against an established standard, you better have a good reason for doing so.  "We don't really need these headers" is not a good enough reason.


If absolutely nothing else, the header identifies the file as an FDS iso in the first 4 bytes.  Which already makes it useful.  I could get into a lengthy debate about why relying on file extensions to determine file type is a bad idea -- and how virtually no program identifies files that way if they can avoid it.... but I'll spare you.

[firedropdl]

Interesting...

Is the "this is an FDS file with two sides" in the 5th byte?

Here's the reason I'm asking...

It looks as if a "standard" (lol) FDS header is as follows:

46 44 53 1A 02 00 00 00 00 00 00 00 00 00 00 00

(I was able to actually "create" a "good" image for Gall Force that matched the CRC on the page by adding this header to a rom I found.  I wasn't able to find this CRC otherwise.)


In the case of BodyConQuest I though, the header reads as follows:

46 44 53 1A  04 00 00 00  00 00 00 00  00 00 00 00


I'm just wondering if that's telling the emulator that the image has four sides, two for each disk image.

As I posted earlier, the readme for this game says specifically that you need to add this header to the image if it doesn't exist (although it doesn't explain why).  Most games don't go out of their way to do this.

[Disch]

Yes -- that is exactly what that is.  "4" is two disks with two sides each.

[firedropdl]

Sweet!  I'm on the right track then. 

I'm still going to work on that game last, but knowing that info I know that the .FDS files out there are combinations of both of the disks.  I'll try to break a few down and see if I can get the matching CRC values from the patch page and then I can tell people for sure which exact CRC they'll need when trying to patch this game. 



EDIT:

So here's a question I've got about something just below the header in an FDS file.

I added a header to a rom I found for Samurai Sword, but this didn't give me the CRC I was looking for.  I was then able to find the CRC elsewhere.  I decided to compare that "good" file to the one I added a CRC to and found the difference almost directly below the header. Here's the comparison...

"GOOD" ROM with header:

46 44 53 1A 02 00 00 00 00 00 00 00 00 00 00 00
01 2A 4E 49 4E 54 45 4E 44 4F 2D 48 56 43 2A 08
53 4D 55 20 00 00 00 00 00 0F FF FF FF FF FF 63
11 01 49 61 00 00 02 06 24 06 65 02 63 11 01 FF
FF FF FF FF 00 00 00 00 02 0E 03 00 00 4B 59 4F

"BAD" ROM that I inserted a header on:

46 44 53 1A 02 00 00 00 00 00 00 00 00 00 00 00
01 2A 4E 49 4E 54 45 4E 44 4F 2D 48 56 43 2A 08
53 4D 55 20 00 00 00 00 00 0F FF FF FF FF FF 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 02 0E 03 00 00 4B 59 4F



It doesn't really matter I'm sure, but I was just wondering if anybody here could tell me what that might be since my curiosity is peaked right now. 

[Psyklax]

http://wiki.nesdev.com/w/index.php/Family_Computer_Disk_System

Read all of that first then continue.

[firedropdl]

Sweet.

Manufacturing Date: November 01, 1988 (63 11 01)
Country Code: 49 (Japan)
Region Code?: 61 (Unknown)
Unknown Bytes: 00
Unknown Bytes: 00 02
Unknown Bytes: 06 24 06 65 02
Manufacturing Date: November 01, 1988 (Repeat of 63 11 01)

But then it gets tricky.....

The next 5 bytes I have in the "good" file are: FF FF FF FF FF (Bytes 47-51)

47: Unknown
48: Unknown (Raw Byte $80)
49-50: Disk Writer serial number
51: Unknown (Raw Byte $07)

That probably shouldn't be FF FF FF FF FF if the file was actually "good", huh?


Also, why do you think this data would have been stripped out of the "bad" file when otherwise it was identical?


I'm assuming that this is a great example illustrating why CRC values don't matter much.  I can't imagine that any of this data matters at all to the gameplay since it just seems like internal manufacturing code information.





EDIT:

Yeah... the FDS is going to be tricky.

As I suspected, you can't rely on the given CRC values to tell you if you need to use the headered or headerless versions.  The only thing it's good for is having one of 2 possible values for the rom, and you have to figure it out from there. 

For instance, the value given for Eggerland is B149EF10.  This is the headerless version of the rom.  If you try to apply the IPS patch to that rom it will say it was successful, but nothing actually happens.  The CRC remains unchanged. 

I added the "standard" FDS header to it and got the value 01D63062.  When you apply the patch to that version of the rom it works fine.  This is a very easy test because according to the page the only thing that this "translation" actually does is remove the Japanese subtitle at the top of the title screen. 

I'll actually be testing the FDS games as I go along instead of doing them all later since it seems that this will be a 50/50 chance of working thing.  I'm preparing a document that will state what the CRC should be before and after and if you're using a headered or headerless rom for each version.  I realize that some readme files actually go out of their way to state this, but most of them don't.

[KingMike]

Sorry, I disagree. FDS ROM headers are useless garbage and I'm not going to support them.
I made by disk ripping/merging program include them if they already existed but that's it.
There is literally nothing of value in the header. All it has is "FDS" and the number of sides.
I could make a text file with "FDS" as the first three letters and apparently that means it's a valid FDS ROM.

You can already tell the sides by the number of sides by the ROM size.
And the actual (Nintendo-created) disk headers already have an identifying mark, the 01,"*NINTENDO-HVC*" string.
(it seems even the unlicensed porn games retain that string. I didn't check every single one but the handful I did all had it)

Just like SNES copiers headers are garbage and give no benefit, only potential for ROM hacking errors. I'll honestly support higan's foltainers before I'll support headers for copiers that nobody even owns anymore, and even people that do own them, let alone working floppy disks/drives to use with them, have to rewrite them anyways because they're unreliable.
(higan's extra metadata actually adds info about the cart layout that can't be inferred from the Nintendo header for nearly all games aside from a few prototypes with incomplete headers)
I'm not preserving worthless garbage because of "standards".

[Disch]

Well... your stubbornness and refusal to conform to the established standard will only lead to the ambiguities we see with SNES roms where hackers have an extra hoop to jump through to determine whether or not the patch requires a header.  I don't know why anyone would want to introduce that problem into FDS images, but I guess I can't stop them if that's what they really want.


Also, putting 'FDS' in a text file to make a "valid" fds file is equivalent to putting 'MZ' in a text file to create a "valid" windows executable.  Or 'PK' to make a "valid" zip file.

These kinds of markers in the header are, again, another widely accepted standard way to identify binary files.  And they're the way in which virtually all programs identify them beyond basic filters in file managers.  But you don't have to take my word for that... try it yourself!  Change that .zip file to a ".fuckmeintheass" file and open it in the archive manager of your choice.  It'll still be able to read it!  Now change the 'PK' marker to something else and try to open it.  It will likely fail!  Even if the extension is .zip

Having those markers in ROMs is absolutely a benefit.

But hey, I'm sure you know better than decades of computer scientists that came before you.  And those pesky 16 extra bytes are just unnecessary bloat, right?


EDIT:

Sorry for all the snark.  But standards make everything better for everyone.  So wanting to throw out a working and widespread standard just to shave 16 bytes off your filesize is pretty fucking ridiculous.


Remember the DISKDUDE header problem with NES ROMs?  That was caused by some jackass who ignored the standard and wanted to do his own thing.



EDIT 2:

So I went back to look at my FDS set, and it looks like there are a lot more headerless images than I thought.  I guess they already have the SNES ROM problem.  Siiiigh.

My Apologies, KingMike.

[firedropdl]

My question really isn't at all about the value of headers on FDS roms so much as what standard was used when making translations. 

The answer, no matter how frustrating it may be, is that there is NO standard....

I find myself in a bad position now with The Legend of Zelda translation.  The readme lists "added header" to one of the changes made to the game. http://www.romhacking.net/translations/fds/patches/2958readme.txt

I've run the exact same rom through the patch with a header (3FBDDDCD) and without a header (EE11AA63).  I was hoping when it was done that I'd get the same CRC.  That didn't happen.

Both patched roms have the same header, and patching the headered version didn't add a second header.  So what happened?

Who the hell knows.  Doing a compare of both files in HxD shows that 126 random bytes are different between the two games now throughout the entire file!!!!!!!


I'm just going to go with the non-headered version of the rom for patching since the instructions specifically state that adding a header was one of the changes to the game. 


If anybody has any ideas about what the changes to the 126 random bytes would be all about I'd love to hear your theories.

FINAL VERDICT FOR ZELDA PATCH:  Using the patched headerless version that the patch adds a header to seems to be the way to go, although I didn't test the other one.  I played through the first dungeon and went into a few shops without any issues and the text was all translated into English. 



EDIT:

Man......  I'm glad there are only 37 FDS translations.  Did I say that before?   



Monty on the Run - http://www.romhacking.net/translations/1545/

ROM / ISO Information:

    Monty on the Run - Monty no Doki Doki Dai Dassou (Japan) [b.].fds
    CRC32: F37893B9
    MD5: bad dump D485DC135BE2DA04F38CCA6EA592C146
    SHA-1: 97A1B69A5ADA499C1067AF1DB0E6A13736A75083

Headerless CRC:  F37893B9
With Header CRC: 43E74CCB

Beat throws an error with both roms.

Beat patch applied to Headerless CRC: 43415f7e
Beat patch applied to Header CRC:     f329655c

IPS patch applied to Headerless CRC:  af4611bb
IPS patch applied to Header CRC:      2d0ce3ce

lol.... which to choose, huh?



TESTING ON MONTY OUTPUT ROMS:

Beat Patch, With Header:  Plays well.  Text in title and beginning screens before action is in english. 

Beat Patch, No Header: Game won't even start.  Emulator says "Format not recognized".  I looked at this in HxD and it's ALL "0"'s!!!!

IPS Patch, With Header: Plays well. Text in title and beginning screens before action is in english. 

IPS Patch, No Header: FDS BIOS starts, but when it loads the disk it says "Disk Trouble Err.25".


I guess I'll just go with the IPS patch.  There's no scientific reasoning for choosing that one, but Beat threw an actual error.  I know Lunar won't do that, but what I don't know won't hurt me, right?

Whoops.... I forgot that if using IPS that you needed to use the included diskexpand.exe.... Hopefully I get the same final CRC results for both patch types when that's done.

EDIT:

Nevermind.... It doesn't look like diskexpand does anything to the rom.  Both headered (43e74ccb) and non headered (f37893b9) versions of the rom are the same size and have the same CRC after running the program with the Monty_expand.txt.  It doesn't even add a header to the non-headered rom, so I don't know why it's included since it doesn't do anything.

Monty_expand.txt
;0
C,2716
;9

Same CRC after the program is run on the headered rom.  Running the Beat patch on these doesn't actually change the size of the rom either...  but something is different because there is a different CRC in the end.


So... I'm going to go with the BPS patch instead.  Even though Beat throws an error, it's doing something that the IPS patch doesn't do because of the different CRC value in the end.  It might be doing what the diskexpand.exe program is attempting to do.   I dunno...

[KingMike]

Removing a redundant header completely is different than leaving an existing one filled with garbage. It's easy to detect if the useless header is present or not. If the header is there, it is still left in operational state.

I don't think it's wrong to ignore standards if we don't agree.

iNES 1.0 is the accepted "standard" despite that we know it's very flawed. There have been attempts to make better standards but AFAIK none have been widely adopted so they quickly fade out.
I'm not sure how much iNES 2.0 would've improved (though it sounds like certainly some amount) but I haven't seen it personally since nobody has wanted to update their ROM sets (I'm used to having to redownload stuff for MAME regularly, I'd be willing to it once for NES) and I hear the 2.0 isn't widely supported by emulators anyways.

I get that iNES 1.0 came about during the infancy of emulation and it feels very haphazardly put together, and probably should've been replaced but it became the de-facto standard, again despite that people have tried to improve it (not to suggest I myself am better than those who tried). (basically anyone dumping could arbitrarily "claim" a mapper number) So we have abiguities such as, I think it was, mapper 34 (combining two unrelated mappers, one Nintendo and one Color Dreams, oddly enough). And I mean... they considered TRAINERS (I had to look up what a "trainer" even is, since I've never seen a trained NES ROM in my life), but not a proper PAL flag? (I know that even people who grew up with PAL despise it, but still to ignore it was a huge omission, resulting in emu authors doing hacky things, like look for "(E)" in the ROM filename, to detect it)
I know Nestopia tried to obsolete mapper 23 (my guess is it was some attempt to clean up the mess of nearly-identical Konami mappers, though that's not the only mapper issue that could use attention but just one of the most popular. The Bandai mappers sound kind of icky but those games aren't nearly as popular.)