News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: CD romhacks often create corrupted cds  (Read 2521 times)

SCO

  • Full Member
  • ***
  • Posts: 129
    • View Profile
CD romhacks often create corrupted cds
« on: January 05, 2018, 08:23:58 pm »
So i was doing a little tool to extract games ids from cds (i want to be able to have a way to rename Retroarch configs to any dump type and name, so i decided to extract a invariant id from the dumps) using libmirage (part of the cdemu cd emulator) and pycdlib, and when testing it on my collection, many of the hardpatched cd translations gave a parsing error in pycdlib. There is a common error on originals which is probably a library bug ('year is out of range').

I'm not sure if most of these weren't already on the originals, but for what it's worth, i only verified that the original iso is not 'corrupted' like that on the policenauts translation. They all play fine (obviously) so this is just a curiosity. The list is just checking the first cd of multicd games to not get too long:

http://www.romhacking.net/translations/1422/ (verified original doesn't have the problem)

'pycdlib.pycdlibexception.PyCdlibInvalidISO: Invalid root directory entry identifier', relevant pycdlib is here.

https://www.romhacking.net/translations/265/ (translation says it inserted the modified files from the original distributed as a iso and create a ppf)

pycdlib.pycdlibexception.PyCdlibInvalidISO: data in 3rd unused field not zero


https://www.romhacking.net/translations/2380/

pycdlib.pycdlibexception.PyCdlibInvalidISO: File structure version expected to be 1


https://www.romhacking.net/translations/859/

pycdlib.pycdlibexception.PyCdlibInvalidISO: File structure version expected to be 1


As i said, this is probably useless and the library guy will probably relax these anyway since the games obviously play, but it's interesting.

It might also be my fault by way of libmirage since it loads dumps in 'most' formats to a in-memory format and the script is just trying to plug in the first data track to the pycdlib iso parser. Maybe some information is getting lost.


« Last Edit: January 05, 2018, 08:39:09 pm by SCO »

weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: CD romhacks often create corrupted cds
« Reply #1 on: January 05, 2018, 08:44:27 pm »
Quote
extract a invariant id from the dumps
Do you know what part(s) of the disk is being checked to get the ID?

SCO

  • Full Member
  • ***
  • Posts: 129
    • View Profile
Re: CD romhacks often create corrupted cds
« Reply #2 on: January 05, 2018, 11:38:11 pm »
Sure.

For Sega consoles (all of them i know), it's actually the first 256 bytes which are the header. libmirage allows me to get the 'real' data without having to care about fileformat gunk (cue/bin MODE2/2352 has a little prefix for instance) or ecc bytes (this one is not really a problem since all cd modes sectors are larger than 2048 at least). A exception: as far as i can tell, Sega mega-cd has two headers glued together to give 512 bytes at the start of sector 0. Since redump uses that second one (AFAIC) i'm using that.

After extracting the bytes, i do a md5sum of them. Since those headers are part of the original data, all (non-corrupted) versions of any iso/nrg/cue|bin/etc will have the same md5sum.

There was another reason to do it like this, from a cursory look at the redump site on sega cd, it's not easy to parse out the the correct 'actual serial' across versions because a great deal of them look incorrect (but the header is correct, since it's incredibly hard to get wrong).


For ps1, the serial is quite standardized (with some exceptions i've encountered).
It's basically the name of the executable in the 'BOOT' line inside system.cnf which is the console boot file. If that file doesn't exist, i try the cd volume descriptor and sanity check it (only game i 'have' that i've had this necessary was King's Field(Japan)). In either case, i then transform it into the redump standard (remove dots, uppercase, add '-' separator if missing). This was actually the main reason libmirage and iso parser was needed, because it's not exactly easy to find a utility that can access most dump formats and still extract files without mounting.

It's still not portable to windows (mostly because libmirage is not portable - as far as i know).

I'm planning to use the script on m3u files (that mednafen retroarch cores support) as a target of configurations names (find config with filename 'id', copy it over to the m3u name). Since m3u support multiple cds in retroarch you would only need a single config (cd1 on the m3u) to apply to all cd changes).

If only i can convince the people doing those configs to use this scheme ofc. That's the part i'm not confident on (see no windows support).


I've not investigated nintendo consoles yet.
« Last Edit: January 06, 2018, 06:13:45 am by SCO »

weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: CD romhacks often create corrupted cds
« Reply #3 on: January 06, 2018, 09:03:01 pm »
I see. King's Field was one of the original PS1 titles for Playstation so it's not surprising that it doesn't follow the standardized pattern. If I ever get to it, I might try looking at the Policenauts patched image and see what's going on.

SCO

  • Full Member
  • ***
  • Posts: 129
    • View Profile
Re: CD romhacks often create corrupted cds
« Reply #4 on: January 09, 2018, 12:56:44 am »
Actually don't bother debugging weissvulv (at least for now). It seems like a library screw up for now. Something about not expecting 2352 bytes sectors in isos, i'm bug reporting it. It is just a bit strange that hacks have different errors but it's better to wait until at least the original dumps problems are fixed before the hacks are analyzed.

weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: CD romhacks often create corrupted cds
« Reply #5 on: January 09, 2018, 04:30:23 am »
Multiple errors from that is strange, but I suppose it could depend on what random data happened to be in the incorrect read-area. Anyways, good you worked it and thanks for letting me know.  :thumbsup:

STARWIN

  • Sr. Member
  • ****
  • Posts: 449
    • View Profile
Re: CD romhacks often create corrupted cds
« Reply #6 on: January 09, 2018, 05:14:17 am »
IIRC isos are supposed to contain only the 2048 bytes of user data per sector, and not the raw 2352 bytes.

SCO

  • Full Member
  • ***
  • Posts: 129
    • View Profile
Re: CD romhacks often create corrupted cds
« Reply #7 on: January 09, 2018, 06:55:02 am »
edit: deleted all the confusing info.

Yes, that's true, but ps1 discs were true mixed mode cds, and always had some sectors at the beginning that had 2324 user data (for copy protection) and sometimes videos or audio with the same (for space efficiency).

I tried to modify the adapter so it could 'read(index)' with these sectors with a complicated scheme of list memoization of the 'naughty' sectors and calculation of the right sector by transversing a list of the naughty sectors and calculating their extent. But pycdlib always objected (even if i can't find the bug). So i decided not to care and pretend the 2324 sectors are 2048 only. It apparently still works to extract the system.cnf.

Meanwhile, the author of pycdlib relaxed the parsing of the problematic cases so it seems it's all good.

Except i got a headache: apparently ps1 and ps2 have the same system identifier 'PLAYSTATION'. There goes the console autodetect. This is just not right.


I also found that that cotton translation nukes the system description, but leaves a new volume description "Cotton RIP Trans". It's obvious that it was done very poorly.
« Last Edit: January 11, 2018, 02:06:17 am by SCO »