News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Poll

can help

yes
0 (0%)
no
0 (0%)

Total Members Voted: 0

Author Topic: how to create programs for unpacking files and archives  (Read 166 times)

Lynol

  • Jr. Member
  • **
  • Posts: 5
    • View Profile
how to create programs for unpacking files and archives
« on: July 22, 2021, 12:09:08 pm »
hello everyone. I'm having trouble again. how do I create programs to unpack files? Someone can give a link to a video with examples or articles on this topic(also with examples). I will be very happy. Thank you in advance. And sorry for my bad English(I'm from Russia)

Cyneprepou4uk

  • Hero Member
  • *****
  • Posts: 662
  • I am the baldest romhacker
    • View Profile
Re: how to create programs for unpacking files and archives
« Reply #1 on: July 22, 2021, 12:48:33 pm »
https://habr.com/ru/company/mailru/blog/490790/

Гугли алгоритмы сжатия zip

FAST6191

  • Hero Member
  • *****
  • Posts: 3164
    • View Profile
Re: how to create programs for unpacking files and archives
« Reply #2 on: July 22, 2021, 02:16:46 pm »
I don't have anything in Russian, though there are some very good Russian ROM hackers. https://gbx.ru http://www.magicteam.net/ being two links good to start with. Something there might be useful.

There are some programs that are scripting languages, or have plugin support. You can often program these to do what you want without having to make a full program to do it yourself.
http://www.romhacking.net/?page=utilities&category=21&platform=&game=&author=&os=&level=&perpage=20&title=&desc=&utilsearch=Go has many programs for extracting things, and many will have source code you can look at.
http://www.romhacking.net/utilities/779/ is an example of a scripting program, though do read the license it comes with.

Generally though files have a few aspects

1) Any kind of file details, including names of things it contains.

2) Pointers.

3) Data itself.

Unpacking is typically using the pointers to locate the files and then slicing off each part of the data section into its own file. If the game does not have pointers we have seen some programs do detection methods for things (some GBA editors, Golden Sun ones, being among the more noted). If there is compression then that too.

You can do a crude version of this without much programming skill. I like a program called filecutter ( https://web.archive.org/web/20170218180937/http://min.midco.net/cracker/filecutter.zip ) but there are others. With that you can tell it to slice this many bytes out at this point in the file. Make a batch file containing a few hundred commands and you have your extraction script for that file at least. For a ROM that will be the same for everybody and same forever then that is not so bad. You can get the locations and lengths from the pointers which a hex editor can dump (if you know a pointer and the next pointer you can subtract and get length, assuming length is not there as part of it all).

If you don't want to do that then you need a programming language that can take in files, read things in binary, and write files back out. This is not all programming languages but it is most of them; C, C++, C#, python, java, ruby, Lua... all good for that. Stuff that will be harder or more annoying is usually web languages (php, javascript, HTML), things focused mostly on text (perl, latex), things for maths (R, matlab) and databases (sql has many forms, and the modern world probably wants to include nosql), though many of those in more modern forms will do the job as well.
http://programming-motherfucker.com/become.html
https://www.youtube.com/watch?v=hE7l6Adoiiw&list=PL6B940F08B9773B9F&index=1
Neither of those are Russian but my usual two links for people wanting to learn general programming and not assembly. Hopefully you can find a translation of some of those books mentioned (many have them to many languages).

If the additional file details are relevant you probably want to note them as well. For instance sound files might note what volume to play things, what instruments to use, what speed to play things and a few other things like that.

Also if you aim to one day repack them you will likely want to note any relevant information, file ordering or the like that simple files might not have to use to recreate the file so it works with the game.

There are hundreds of arrangements of parts of files just for a basic archive, and hundreds of little quirks. For instance in one case on the DS I had a file format that shifted its numbers (effectively multiplied them by 4) for reasons best known to itself. Another common DS archive format called NARC has the high bits of its pointers set if the file is a subdirectory within the archive. This sort of thing can take a long time to learn and to spot, however it is pretty much the definition of what ROM hacking is.
For examples of formats
http://www.romhacking.net/documents/469
https://wiki.multimedia.cx/index.php?title=Category:Game_Formats
http://wiki.xentax.com/index.php/Game_File_Format_Central
https://sites.google.com/site/kiwids/sdat.html
http://kiwi.ds.googlepages.com/nsbmd.html
https://wiki.vg-resource.com/Nitro_Files
Several of those pages will use a format to describe the contents based around what C programmers will use so might want to learn that even if you don't learn/use C in your extraction program.