[Auryn]

Hi everybody,
saddly i am lacking programming skills so I am here to ask if somebody would make me a little tool for windows or command line (dos).

The tool should be a file splitter that splits a big file into pieces by reading the offsets from an txt file and naturally the opposite too.

Maybe an example will make it more clear:
let's say we have a txt with this 3 offsets inside:
00 00 00 00
00 00 A0 00
00 B0 00 00

A big file called big.bin

the program should take big.bin and split it into 1.bin (data from 00 00 00 00 to 00 00 A0 00), 2.bin (data from 00 00 A0 00 till 00 B0 00 00) and 3.bin (data from 00 B0 00 00 till EoF).

Naturally the opposite too:
take 1.bin, 2.bin and 3.bin and place it inside a file named big.bin at the offsets in the txt. If one of the pieces bin is too big as the place disponible between the 2 offset, it should not insert it and give me an error message of wich file is too big.
If one or more of the pieces bin is smaller as the place between the 2 offsets, pad the space with 00

For limits, i am not sure because i didn't split the files manually to check the max but i would say that big.bin should be max 100MB and have 300 files on it.

I hope it's clear and that somebody can make me this little tool.

Thanks

[andlabs]

For a splitter, I'm sure many hacking communities have had this tool, but here's the one the Sonic hacking scene uses; I've used it myself for Mega Drive/Genesis research. The general format of each line is "#split" followed by a tab (NOT SPACES) followed by three comma-separated fields. The first field is the start address as a C hex number. The second field is the end address (that is, the address of the first byte AFTER the data) as a C hex number. The third field is the output filename. You can also create directories with lines beginning "#dir" followed by a tab and then the directory name.

Here's an example, from the Sonic 2 disassembly on Sonic Retro's subversion:

Code: [Select]

/***********************Directories***************************/
#dir    art
#dir    art/palettes
#dir    art/nemesis
#dir    art/kosinski
#dir    art/uncompressed
#dir    mappings
#dir    mappings/16x16
#dir    mappings/128x128
...
/***********************Palettes***************************/
#split  0x001E5A,0x001E7A,art/palettes/Title Water.bin
#split  0x001E7A,0x001E9A,art/palettes/EHZ ARZ Water.bin
#split  0x001E9A,0x001F1A,art/palettes/Hill Top Lava.bin
#split  0x001F1A,0x001F2A,art/palettes/Wood Conveyor.bin
#split  0x001F2A,0x001F36,art/palettes/MTZ Cycle 1.bin
#split  0x001F36,0x001F42,art/palettes/MTZ Cycle 2.bin
#split  0x001F42,0x001F56,art/palettes/MTZ Cycle 3.bin
#split  0x001F56,0x001F66,art/palettes/HPZ Water Cycle.bin
#split  0x001F66,0x001F76,art/palettes/HPZ Underwater Cycle.bin
#split  0x001F76,0x001F86,art/palettes/OOZ Oil.bin
#split  0x001F86,0x001F8E,art/palettes/MCZ Lantern.bin
Usage is

Code: [Select]

splitrom.exe ROM.bin split_list.txt
For combining, if you just want to merge files together, most command-line copy or concatenate command:

Code: [Select]

(windows) copy /b file1+file2+file3 bigfile
(unix) cat file1 file2 file3 > bigFor padding with zeroes, if your assembler can import raw binaries (a directive like incbin or binclude), you should be able to use it...

[Auryn]

Thanks, I didn't try it yet but from the look at it, it will do the trick even if i wanted to have a little bit more security especially in the recombining part.

[KingMike]

Isn't there such a functionality already in file archiving programs, like 7zip or WinZip or something similar?

[Auryn]

This tool is not perfect but at least for the splitting part it's ok.
I have to check for something better to insert the files in the big.file.
Anyway i don't think that 7zip or winzip can extract parts of files especially if my files are not compressed in 7/win zip.

What i need is more like TIMmay in the utility section but without the search function for TIM.
Lucky me for one of my project i can misuse TIMmay but i need an alternative for the other project.

[messiaen]

In the end you may  spend more time searching for something like this than writing your simple tool.

Just read a basic file I/O tutorial on any language (my suggestion: cprogramming.com) and you should be able do this in no time, this is really straightforward stuff.

Nothing better to start learning a programming language than having practical problms.

[andlabs]

If you don't mind me asking: what exactly are you intending on doing with this?

[Klarth]

This tool is not perfect but at least for the splitting part it's ok.
I have to check for something better to insert the files in the big.file.
Anyway i don't think that 7zip or winzip can extract parts of files especially if my files are not compressed in 7/win zip.

What i need is more like TIMmay in the utility section but without the search function for TIM.
Lucky me for one of my project i can misuse TIMmay but i need an alternative for the other project.

PM me if TIMmay doesn't work out so I can fix it.  It wasn't well-tested to be honest and I'm half-surprised that anybody finds it useful.

[Auryn]

@messiaen: That's my last resort but for now i want to work on my project that are waiting long enougth.

@andlabs: since the playstation one, passing from GBA/NDS to the psp, there is a tendence to put many files in one big files in some sort of virtual/custom file system and because i don't want to ask a tool for each project i have/will have, i ask for a tool that is general and flexible.
The 2 main purposes of this tools for now are the splitting of the data.bin and all_mes.bin of the Phoenix Write serie and the slitting 400+ compressed tims and other files from some archives on my FF7 International 4°CD project :
http://www.youtube.com/watch?v=Uq37IdRjLTI
Almost everything that has still japanese on it (apart the information screens and some point where u can click on the world map) need to be splitted with this tool, then decompressed, modified/translated and reinsered with this tool.

@Klarth: You tool is sure usefull. If my memory is not wrong, there are a view tools that can extract multi tims from a file (yuri, multi tim rip, etc..) but none can insert them again especially not on block.
There are many psx games that have hundreds of multi tims: apart from my project, there is maybe 2 other future project that have alot of tims on it: Dice de chocobo and Tales of famdom Vol1.
Anyway, i just tried and i can't use TIMmay to extract my compressed tims 
I was thinking that he just look for the magic word x10 00 00 00 (this is present in the compressed tims) but timmay probably will need the dimentions of the time to calculate the ammount of data to extract (and in my case it's already compressed at that point)
Now i am too tired but maybe (if u didn't build a tim check on the file to reinsert) i can trick timmay to re insert the compressed tims in base of manually made log . Or take the idea of messiaen and mess around with your code :p
Thanks anyway and if i see any error, i will tell u.

[Klarth]

It sounds like your approach might be wrong with the PS titles.  Normally this data is structured (as unstructured as it may appear if you're just using a TIM ripper) inside of a Virtual File System, which is one big file containing many (sometimes in the ten thousands) smaller files.  Building a utility to deal with this will save you more hours and frustration than you can imagine, especially if you need to resize anything.  Which inevitably will happen when you're dealing with edited graphics.

As far as TIMmay goes, you'd need to make a few code changes for your uses.  The TIM detection (and size calculation) will be off.  It also uses the log not just for position data, but replaces TIM specific information in the file that may be lost in the editing process, such as Image X/Y and CLUT X/Y.

[Auryn]

I know about the Virtual file system and what i am asking is exactly the tool u are talking about. The only difference between your and my thinking, it's that u want to make a specific tool for each game but i want one that i can adapt to all games i want to mess with.

In the specific case, my "archive" is somewhere between the LZS (of the psx version) and LGP (of the pc version) archives of FF7.
I can search the magic word inside my archive and i get a list of all offsets i need from Hex workshop; copy that list to a txt correcting it to make the extracter/splitter tool 4 bytes before the magic word so that the lzs decompressor can handle the splitted files.

In the other project, there are some pointers somewhere in the game to point to 8 different headers of sub-files all in one single bin file:

header 1
pointer table 1
data1
header 2
pointer table 2
data 2
...
I am faster to check the data size on the headers and write it into the txt than go to put breakpoints in some disassembler (i remember that i not really good on programming) and can use the tool to split the 8 blocks into pieces and then copy/paste the specific pointer table to a txt and extract all sub-files that are any data possible all with the same tool.

Maybe it's me that is stupid and like to make the things "manually" or maybe the big guys in the scene just are too "lazy" to do some hand job. At the end, i don't know if it's more time consuming creating a tool for each specific game or having a "general" tool and some hand job.


As far as TIMmay, if u programmed like that, i can't use it at all then. This is exactly why i want a "general" tool and not a specific tool. The size thing is not a problem, the padding or warning would just be in case of errors.

[Karatorian]

Maybe it's me that is stupid and like to make the things "manually" or maybe the big guys in the scene just are too "lazy" to do some hand job. At the end, i don't know if it's more time consuming creating a tool for each specific game or having a "general" tool and some hand job.

The thing is, once you learn how to program, writing a tool is easier than doing stuff by hand. (That's why they invented computers after all.) On the other tentacle, there is something to be said for generic tools. For example, I do all my file splitting with dd and shell or make scripts. You might want to see if there's a decent dd port for win32. Then you could just use batch files with the offsets and stuff for splitting.

[DeGamer]

Normally this data is structured (as unstructured as it may appear if you're just using a TIM ripper) inside of a Virtual File System, which is one big file containing many (sometimes in the ten thousands) smaller files.  Building a utility to deal with this will save you more hours and frustration than you can imagine, especially if you need to resize anything.  Which inevitably will happen when you're dealing with edited graphics.

One of the games I'm working one has one big file like that. So you're saying if I edited a graphic and needed to make it larger in size, a custom tool would allow me to do that without shifting/screwing up the rest of the data and the game would still work?

Something like this:

code, CLUTs, code, graphics, pointers, code, graphics

into something like this:
code, CLUTs, code, edited larger graphics, pointers, code, graphics

I always thought that my larger edited data would shift any data after it causing pointers to then point to the wrong places unless you changed all of them manually (not to mention any code after would no longer work properly).

[Auryn]

No, he means that a custom tool would resize/repoint everything for u.

This is where i am a bit in conflict with the answer of Karatorian. U will need to spend so much time to figure out how everything is structured and how everything is saved that sure, the working tool will save alot of time to the user but i am not sure it will save alot of time to the builder.

Anyway, u are talkin about CLUTS so i think u are working on a psx game.
I would say at least 75% of the psx games have "empty holes" between the files that u can use.
If u have a nice pointer table on the beginning of your file, i would repoint the edited image to the end of the file and add the larger data there, then try to fit the file in the original place by using the "holes" or move the whole file to the end of the disk with http://www.romhacking.net/utils/466/

[urbanopsilva]

Just in case someone still needs this kind of a job...

I have recently uploaded here the rigth tool to it.

It is caled HexTraImp

here is the post:
http://www.romhacking.net/utilities/1001/