News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: The nature of NES and FDS headers  (Read 8243 times)

Psyklax

  • Hero Member
  • *****
  • Posts: 1109
    • View Profile
    • Psyklax Translations
Re: The nature of NES and FDS headers
« Reply #20 on: August 19, 2017, 12:18:08 pm »
This is getting out of hand... ::)

Look, the FDS 16-byte headers should all be cut out. Every emulator that can pay FDS files doesn't give a rat's ass about them, and it's irrelevant what any other program in the universe thinks of the files.

Compared to the amount of bad dumps, overdumps, hacks, trainers and God knows what else that you see with ROMs, there are relatively few dumps of FDS games, so the difference is 99% of the time just header nonsense and save games.

If some poor soul patches a file and it doesn't work, tell 'em to add or delete this pointless header. And if they can't figure out how to do that, write a program that does this trivial task for them.

Sorry, but playing ROM hacks is already the nichest of niche hobbies, if some people can't just RTFM then I have no sympathy. >:D

firedropdl

  • Full Member
  • ***
  • Posts: 111
    • View Profile
Re: The nature of NES and FDS headers
« Reply #21 on: August 19, 2017, 12:36:49 pm »
I could see most rom hacks being a niche.  Except for a rare few, I don't really care for them (no offense, but they're just not my thing).

Translations are different though.  Even if they're not perfect word for word translations, it opens up an entire new library of games to an English speaking audience.  Not including the adult games and the games that were translated by two separate groups here, you guys have made right around 375 NES and FDS games available to an English speaking audience that weren't before. 

Granted, not all of them are great games, but neither were all the US/EU games.

Some of them are amazing though, and considering all the crappy games that were officially translated the work you guys have done here just proves how many mistakes were made by the suits that chose which games to port and which ones not to bother with.

I think it's unfortunate that this is a niche hobby thing, at least when it comes to translations.  I also think one of the reasons for that is because of how difficult the patching can be for novices.  Everyone is free to have any opinion about those novices that they want and say that they don't deserve to play them unless they learn... I'm not going to get into that argument.

I'm just going to say that I think, from experience, that one of the major obstacles is that it is unnecessarily difficult because there is no standard when it comes to making the patches.   What works for one game will absolutely not work for the next game, and vice versa.  This problem isn't nearly as bad on the NES translations as the FDS translations, and I think that's DIRECTLY a result of the necessity of the header on the NES files, regardless of how good or bad the iNES system is. 

The fact that the FDS header is superfluous causes A LOT of problems when it comes to patching the translated games.  Unlike the NES games that were mostly fairly easy to figure out what was wrong when you learned a little bit about the process, the FDS becomes a complete game of trial and error. 

Sometimes the patch page says the CRC for a headerless rom.  There's a 50/50 chance that you're going to need the headerless rom.  In most cases, there is no readme file or if there is it doesn't say if you do or don't. 

None of the information is reliable for FDS games, save for the few authors who wrote VERY specific instructions in their readme files.  Just seeing the replies in this thread, I'm going to say it's safe to attribute that to the fact that everybody has their own idea about what to do with headers on the FDS and they did their own thing with them.  I'm thinking it probably doesn't even get discussed much so the authors that don't put specific instructions didn't even think they were necessary because they thought the way they were doing it was the way that everybody else was doing it.



Anyways.....

Not judging nothing.  Just talking.

I'm going to have a thread with some more details about the FDS translations when I'm done.  I'm already writing it while I'm doing the games.  It will let you know which ones need a header, which ones don't and notes about specific problems you may run into on any of the patches.  It will also have info on the proper rom the author put on their page and the CRC value for both headerless and roms with headers so if they can find one version but not the other all they will need to do is add or remove the header before patching it depending on what the author made the patch do.



Didn't mean to cause any drama here guys.  :)








EDIT:

Here's an example that I just patched.....   

Boxxle - Warehouse Evening of Tears - http://www.romhacking.net/translations/1/

(September 29, 2001, by KingMike) Just about 16 years ago.....

ROM / ISO Information:

    Namida no Soukoban Special.fds
    CRC32: 24A387F0
    MD5: 3DFAB68E8C33035829A168BE7C2E0805
    SHA-1: 4650F8574A10F74319A93AA7E1905D7065E06D96
    SHA-256: 38792E51B2AAB05AC278C99553C11412E519F00046AAF755D8DF43F7E900DBF0


24A387F0 is the rom with the header.

Strip the header and you get 943C5882.

Don't apply the patch to the headerless rom though.  It will apply without an error like all other IPS patches do, but the title screen and all text in the game is extremly garbled and just as unreadable as it was pre-translation.  In fact, if you just stumble around and press buttons, the game goes to a black screen and freezes on you before gameplay.


You have to apply it to the 24A387F0 headered rom that was put on the rom page.  That works fine.




Again... not trying to call anybody out.  Just pointing out the problem when there is no standard and nobody agrees on one.  16 years was a long time ago.  Things change.


« Last Edit: August 19, 2017, 12:54:19 pm by firedropdl »

Psyklax

  • Hero Member
  • *****
  • Posts: 1109
    • View Profile
    • Psyklax Translations
Re: The nature of NES and FDS headers
« Reply #22 on: August 19, 2017, 12:56:53 pm »
What works for one game will absolutely not work for the next game, and vice versa.

But you've been through hundreds of games and encountered only a handful of problems, so I don't think that's a fair assessment. The vast majority of the time you unzip an IPS, double click it, select your ROM and it works. It really can't get simpler than that.

Again, because the FDS is a special case, maybe someone (not me, as I can't program) can write a very simple program like Lunar IPS that says "choose your IPS, choose your FDS" then spits out two files: one which patches as Lunar IPS would, and one which adds/removes 16 bytes to each address listed in the IPS. The program would just check if the 16-byte header exists, and removes 16 bytes for file 2, otherwise it adds 16 bytes. This would be very simple to program but I have no programming skills. The IPS file format is very simple, too, so it'd be easy to implement.

Then you can try both files and keep the one that works. Simple.

firedropdl

  • Full Member
  • ***
  • Posts: 111
    • View Profile
Re: The nature of NES and FDS headers
« Reply #23 on: August 19, 2017, 01:10:56 pm »
But you've been through hundreds of games and encountered only a handful of problems, so I don't think that's a fair assessment. The vast majority of the time you unzip an IPS, double click it, select your ROM and it works. It really can't get simpler than that.

Yeah... that sentence alone taken out of context is unfair.  I do say that on the NES overall it's not a huge problem.  (Although there are quite a few pages that list the headerless rom info instead of the headered rom which is very confusing until you learn what you're doing here)

Quote
Again, because the FDS is a special case, maybe someone (not me, as I can't program) can write a very simple program like Lunar IPS that says "choose your IPS, choose your FDS" then spits out two files: one which patches as Lunar IPS would, and one which adds/removes 16 bytes to each address listed in the IPS. The program would just check if the 16-byte header exists, and removes 16 bytes for file 2, otherwise it adds 16 bytes. This would be very simple to program but I have no programming skills. The IPS file format is very simple, too, so it'd be easy to implement.

Then you can try both files and keep the one that works. Simple.

I can't program either.  I'm documenting all of this stuff though.  All "fully playable" translations as of 8/19/2017 on the FDS will have info saying whether or not you need header when patching them.  If anybody wants to update the pages or write a program with that information is out of my hands.  ;D

goldenband

  • Sr. Member
  • ****
  • Posts: 333
    • View Profile
Re: The nature of NES and FDS headers
« Reply #24 on: August 19, 2017, 01:23:53 pm »
I think the moral of the story for me is that, since headers aren't copyrighted, I'm going to add them to the README of any translation I do in the future. :D That way, anyone who has a headerless ROM or one with a different header can reconstruct the version I used, as long as the dumps themselves otherwise match.

From an end-user's perspective, all you want is to be able to find or generate a patchable ROM with a minimum of headaches -- or without having to troll random sketchy sites to find a ROM with the right CRC, only to discover that a single byte in the header is the only difference between that ROM and the version that 99.99% of ROM sets (and people) have.

firedropdl

  • Full Member
  • ***
  • Posts: 111
    • View Profile
Re: The nature of NES and FDS headers
« Reply #25 on: August 19, 2017, 02:29:23 pm »
I think the moral of the story for me is that, since headers aren't copyrighted, I'm going to add them to the README of any translation I do in the future. :D That way, anyone who has a headerless ROM or one with a different header can reconstruct the version I used, as long as the dumps themselves otherwise match.

Sweet.  I might also suggest that you put the before and after CRC values on the patch page as well.  In the nearly 400 translations I've patched, less than 5 had this information in them.  :)

Quote
From an end-user's perspective, all you want is to be able to find or generate a patchable ROM with a minimum of headaches -- or without having to troll random sketchy sites to find a ROM with the right CRC, only to discover that a single byte in the header is the only difference between that ROM and the version that 99.99% of ROM sets (and people) have.

Had to go to a couple of sketchy places for a handful of NES games that were more obscure like the Chinese Unlicensed games.  For the most part GoodNES and No-Intro had the rest (Once I figured out that the 30 or so pages were giving you the CRC value without a header).

As long as you're not patching the FDS I don't think you have to go too far out of your way to make that easy.  The FDS is a freaking mess though.



.......................

Deep Dungeon I (2006): Requires header to patch, diskexpand.exe will add the header. (Needs header, but if you run diskexpand.exe on a headered rom it just keeps making a bigger and bigger output file until you stop the program, so you need to use a non-headered rom and have diskexpand.exe add it before patching).  Final result is a patched rom with a header.

Deep Dungeon II (2010): Requres non-headered rom to patch, use one or diskexpand.exe can remove it.  Final result is a patched rom without a header.



EDIT:

Freakin sweet!

This stuff is really starting to make sense to me now.  In just a matter of minutes I was able to pull apart the BodyConQuest rom and get the two disks that are specified on the patch page with matching CRC values.  All I had to do was copy the two halves into new documents and then put the standard single disk header on them and they matched.  :)

Got one more that I have to go back and try to get working, then I'm going to retest everything to make sure that I kept the right versions of all of the FDS translations and didn't accidentally keep any bad patch jobs, then I'm going to get to work on that FDS thread I promised.
« Last Edit: August 19, 2017, 08:46:46 pm by firedropdl »

KingMike

  • Forum Moderator
  • Hero Member
  • *****
  • Posts: 7060
  • *sigh* A changed avatar. Big deal.
    • View Profile
Re: The nature of NES and FDS headers
« Reply #26 on: August 19, 2017, 11:05:31 pm »
Note you could PROBABLY use the DD2 expander on DD1.
I can't say for certain since it's been so long since I wrote the program, but it was designed to be a "general" program.
"My watch says 30 chickens" Google, 2018

firedropdl

  • Full Member
  • ***
  • Posts: 111
    • View Profile
Re: The nature of NES and FDS headers
« Reply #27 on: August 19, 2017, 11:47:18 pm »
Don't need to.  The game works fine with the tools you put in there if you use the correct rom version.

I was just using that to illustrate what not having a standard does.  Two translations done by you that were 4 years apart and one needed a header and the other one won't work with one.  I can't pinpoint exactly when you stopped using FDS headers, but for quite a while there back in the day you used them in all your FDS translations, and then all of your later ones won't work if you have them.

Totally understandable when somebody changes their mind or learns something new over the years, so I'm not giving you any crap about it.  It's probably something nobody ever noticed before and you might not have even realized all these years later.

I don't imagine it's everyday that somebody starts going through 16 years of your history here in the span of 2 days.   :laugh:



BTW... did you see my request to check the values for your newest NES translation "Light of Indra" in the other thread, KingMike?  I can't find any rom with the values you say to use, so I was wondering if you could check them. 

I'm not pressuring you on it.  Just never saw anything by you that you saw the post and wanted to make sure you were aware of it.

Thanks :)



EDIT:

I'm all set to post the thread about patching FDS games... even used the preview feature here to make sure everything is good.

Problem is that it's way more than 15,000 characters and I'm still on a provisional status so I can't double post for a few more days.  I've got it saved in notepad and ready to post whenever that restriction is lifted.
« Last Edit: August 20, 2017, 01:19:46 am by firedropdl »

KingMike

  • Forum Moderator
  • Hero Member
  • *****
  • Posts: 7060
  • *sigh* A changed avatar. Big deal.
    • View Profile
Re: The nature of NES and FDS headers
« Reply #28 on: August 20, 2017, 11:27:26 am »
Original, unexpanded, 128KB Japanese ROM (with header):
Header: 4E 45 53 1A 08 00 12 00 00 00 00 00 00 00 00 00
CRC32: FF03672E
MD5: EFEDDAF077ECFA95E82C635BE042D58C
SHA-1: 55055548C7AFDF41980493CF49F1592DE0A4ED87
(without the header, the ROM matches the No-Intro DAT)

->

Expanded to 256KB but not yet translated:
(again, with header)
CRC32: E4EBB15F
MD5: 627D5B02F20789CD8250710AE083DE3B
SHA-1: F84F587EB88470553A50E094CBF229F4624626BA

->

Translated:
CRC32: 3B5CCB05
MD5: 8DAB0AFEFD940DA8B0D96FBDE66A8A79
SHA-1: 6B4D7D7982B0F8A9EB8B9FAE7AB4E28054252E90
"My watch says 30 chickens" Google, 2018

firedropdl

  • Full Member
  • ***
  • Posts: 111
    • View Profile
Re: The nature of NES and FDS headers
« Reply #29 on: August 20, 2017, 01:23:15 pm »
Awesome.  Thank you, my friend.   :)

Found an immediate match for that with the header.  It's "Indora no Hikari (J) [!].nes" in GoodNES 3.23b.

Just an FYI for you if you wanted to edit your patch page for the game, this is the info you had for the original rom before the expansion:

Original ROM, including header (128 KB):
CRC32: B2352DD2
MD5: 0E879C9A3C023132C504A010C0DEF083
SHA-1: 80DA4536C10294E83EC5EE8DD98C23F5BE7B0BB9



Also, the after expansion you just put here matches the patch page and is still wrong.  (I found this out by patching my result anyway and got a match on your final post/expansion-post/patch result.


Here's the info I have for all three rom statuses:

Original, unexpanded, 128KB Japanese ROM (with header):
Header: 4E 45 53 1A 08 00 12 00 00 00 00 00 00 00 00 00
CRC32: FF03672E
MD5: EFEDDAF077ECFA95E82C635BE042D58C
SHA-1: 55055548C7AFDF41980493CF49F1592DE0A4ED87
(without the header, the ROM matches the No-Intro DAT.
With header is Indora no Hikari (J) [!].nes in GoodNES 3.23b)


Expanded to 256KB but not yet translated:
(again, with header)
CRC32: F51AFBD7
MD5: d4210a45fb2430036bc24e087fdf10ec
SHA-1: 041957c838392bae5df3680b54c77662a28800c5


Translated:
CRC32: 3B5CCB05
MD5: 8DAB0AFEFD940DA8B0D96FBDE66A8A79
SHA-1: 6B4D7D7982B0F8A9EB8B9FAE7AB4E28054252E90


Thanks again.  I'll update that on the NES page.  Only 8 more to go.  Does pacnsacdave ever post around here?  Clearing up a few with him would leave only 1 or 2 more to go. 



EDIT:  I haven't really tested more than a few dozen of the NES translations yet since they were much easier to figure out than FDS and all of the ones that I did test had no problems.  I will eventually get around to that.

I did want to go out of my way to test this one now though since you went out of your way providing the info. 

Everything works!  Looks great man.  Thanks again.  :)
« Last Edit: August 20, 2017, 01:37:11 pm by firedropdl »