[hackfleisch]

Hi, i am looking for information about the binary format of PSP and GB rom images and how to decode/unpack them.
Where should i start my research?

[Disch]

I have no clue about PSP (a bit too modern for me), but as for GB:

The GB ROM file is just a straight dump of the binary data that exists on the ROM chip on the cartridge.  It's not really a "file format" in a traditional sense in that there's not really any structure -- it's just all the data as laid out on the cart.  So there isn't really anything to "unpack".

GB emulators will just take that data, load appropriate blocks of it into the GB's memory space, and then start emulating.

That said... GB games do have an "internal header" where information about themselves is stored... and emulators can use that information for certain things.  I found a header outline here:

http://gbdev.gg8.se/wiki/articles/The_Cartridge_Header


Other information about the GB can be found here:

http://www.devrs.com/gb/docs.php#docsmisc
http://www.devrs.com/gb/files/gbspec.txt
http://stackoverflow.com/questions/3690919/a-reference-on-the-layout-and-structure-of-gameboy-color-roms

[FAST6191]

There is no unpacking GB/GBC ROM images. Generally speaking then before the DS if it did not come on a floppy disc or CD then it did not have a file system, minor exception for some pieces of homebrew code. Various games could also have used so called memory bank controllers (mbcs) which function kind of similar to NES mappers in that they allow extra space, resources and processing for some things. http://problemkaputt.de/pandocs.htm#memorybankcontrollers has more. There is probably a simple trick to finding things like the binary in GB/GBC games (on the GBA you look at the start of the ROM, it will then usually jump to the end of the header, set up some IO and then jump to somewhere in the ROM which was the start of the binary, minor exception for things like phantasy star collection -- http://doc.kodewerx.org/hacking_gba.html#nonstandard ).
 http://doc.kodewerx.org/hacking_gb.html is probably also worth a look.
If I am linking curios then http://webcache.googleusercontent.com/search?q=cache:vRJYKY10nrMJ:www.pagetable.com/%3Fp%3D28+&cd=1&hl=en&ct=clnk&gl=uk , http://www.pagetable.com/?p=28 is the usual page but it seems to be down right now. https://web.archive.org/web/20160309234903/http://www.pagetable.com/?p=28 also has a copy.

PSP isos are fairly standard isos, can't recall offhand exactly what sub spec (load up something like imgburn and see all the various disc formats and file name supports that might stem from that). Most use the tool known as umdgen to extract them and manipulate them (it extracts, adds, adds different sized files if you want and unlike most can even relink them).
The binary format is usually eboot.bin, depending upon what age of guide you are using you might be told about the encryption on them which had to use a PSP to decrypt. Sony very nicely gave us all the keys when the PS3 was hacked (they included private keys on every PS3).
I am not sure what people are using for a disassembler there, the PPSSPP emulator has a decent enough take on debugging capabilities and depending upon what you want to do to the PSP then a lot of people did things in hardware and linked back to the PC over USB. Afraid I have not got a nice reference for the PSP like the pandocs I linked for the GB/GBC or http://problemkaputt.de/gbatek.htm but if you poke around http://www.ppsspp.org/ or its code then you will find some stuff. Not sure what happened to the old dawiki which had some nice stuff as well.

[hackfleisch]

WOW, THANKS A LOT, YOU BOTH ! 

This give a good start. Now i have learn about the hardware and the layout of the ROM data. I also could "decompile" some ROM-Infos from the information found in the links:
The ROM is "Star-Trek Generations - Beyond the Nexus". The file-suffix is ".gb".
 - Its game title at 0x134 is "GENERATIONS"
 - It has 128k in size and ist of cartridge type 0x01 (ROM+MBC1)
 - Byte 0x149 tells 0x00, so the cartridge has no extra RAM on it (ROM-Only cartridge)
 - Byte 0x143 is 0x00, so it is a black+white only game (plain old native GB). No color.
 - The entry-point at 0x100 tells: F3 C3 51 03 (in hex) I decoded them by the Z80 Opcode list found here http://imrannazar.com/Gameboy-Z80-Opcode-Map this Assembler code:
F3 = DI (Disable interrupts)
C3 = JP (2 bytes with addess follow, LSB first, so it will be "JP 0x0351"
So, code goes on at this address.
Next it put the ROM file into an Disassembler i found on the web here https://www.onlinedisassembler.com/odaweb/.

Well, but how to go on now? My Assembler skills are a bit rusty ;-)

The code at 0x0351 look like this:
ld sp,0xe000
ld hl,0xc000
ld bc,0x2000
xor a
ld (0x780b),hl
or c
jr nz,0x035a
ld hl,0xff8a
ld c,0x75
xor a
ld (0x200d),hl
call m,0x79cd
inc b
ret p
adc a,h
and 0x0f
cp 0x0f
jp z,0x036b
call 0x0738
call 0x0470
call 0x0392
call 0x03ed
call 0x1dc5
call 0x05cf
call 0x0835
call 0x0150
jp 0x071e
(i'm not shure if it is allowed to post or attach the whole disas file here...)

I also found this GB Startscreen image in the header. It is called "bitmap" in doc, but it this really a simple bitmap graphic?
And how to arrange the bits to get the logo? The Screen resolution is 160x144 Pixels, but this would give about 3k of data for one screen. The are much more less bytes in the logo.

As you see, i begin to learn and it makes a LOT of fun!!!

[Disch]

Well, but how to go on now?

That depends on what your goal is.  Are you trying to disassemble and document this ROM so that it is fully understood?  Or are you just trying to learning more about GB architecture?

I also found this GB Startscreen image in the header. It is called "bitmap" in doc, but it this really a simple bitmap graphic?

Technically yes it's a bitmap.
But not a "*.bmp" windows style bitmap file that you might be thinking of.  You won't be able to load it up in Paint or anything.

All "bitmap" means is that you have a "map" of "bits", where groups of bits represent a pixel.

GB graphics are stored in a 2bpp interleaved planar format, where each group of 16 bytes represents an 8x8 pixel "tile"... and multiple "tiles" are grouped together to form a larger image.

GB's 2BPP format is outlined here:

http://wiki.superfamicom.org/snes/show/Learning+the+GFX+Format+-+2BPP+Gameboy+and+SNES

You can also use tile editors like Tile Molester, YY-CHR, or Tile Layer Pro (all of which are available on this site in the utilities section) to view the graphics in the ROM.



The Nintendo Logo in the header is $30 bytes, which is 3 tiles, which probably forms a 24x8 pixel image.  EDIT:  but even that seems small.... maybe it's stored 1BPP to make a 48x8 image?  I'd have to research it.  =x

[hackfleisch]

Disch, i understand the concepts of bitmap images. Whats missing was how they are arranged (the planar configuration). Even this is similar to Tiff without headers...

Ok, i research this myself. Just another step in understanding the console. My goal is indeed to understand how everything works.

How can a software find images in the ROM? Are there lookuptables at some well known location?
I also wonder which programming language was used to build the games. I don't think they where coded in ASM. Maybe C or C++ ? Or did rhey use Game Enginges in that era of computer games?

Well, how can i go further in understanding the code found at position 0x351 ?
Are ypu able to interpret what is going on there?
The list of calls seems like a being a main loop.

[AWJ]

Disch, i understand the concepts of bitmap images. Whats missing was how they are arranged (the planar configuration). Even this is similar to Tiff without headers...

Ok, i research this myself. Just another step in understanding the console. My goal is indeed to understand how everything works.

How can a software find images in the ROM? Are there lookuptables at some well known location?
I also wonder which programming language was used to build the games. I don't think they where coded in ASM. Maybe C or C++ ? Or did rhey use Game Enginges in that era of computer games?

Well, how can i go further in understanding the code found at position 0x351 ?
Are ypu able to interpret what is going on there?
The list of calls seems like a being a main loop.

Generally, you find graphics in a ROM by loading it up in a tile editor like Tile Layer Pro, scrolling through the pixel garbage and looking for anything recognizable. If you don't find the graphics you're looking for, they're probably compressed in some way, and you'll have to reverse engineer the compression scheme.

Commercial Game Boy games were almost always programmed in straight assembly language. Even today C compilers don't do especially well with 8-bit CPUs, and compilers in the 1980s-1990s were much more primitive.

Be careful: the CPU used in the Game Boy (Color) is not a standard Z80; it has quite a few added, removed and changed instructions. You'll need a GB-specific disassembler to disassemble GB ROM images correctly (in the disassembly you've pasted, the ld (0x200d),hl and everything after it is wrong)

[Disch]

I also wonder which programming language was used to build the games. I don't think they where coded in ASM. Maybe C or C++ ? Or did rhey use Game Enginges in that era of computer games?

The majority (all?) of commercial NES and GB games were coded directly in ASM.  Probably SNES, too.  Those systems have very, very slow processors and the overhead involved with a high level language like C would have been too much of a burden.

Well, how can i go further in understanding the code found at position 0x351 ?
Are ypu able to interpret what is going on there?
The list of calls seems like a being a main loop.

As AWJ mentioned -- be careful about disassembly.  GB-Z80 is not traditional Z80.

As far as interpretting it goes --- for my disassembly of FF1, I first tried to figure out what each area in RAM represented, so that I could replace raw addresses with variable names.  That makes it way easier to understand what the code is trying to do.

It also helps if you're extremely familiar with the game -- as I was with FF1.

[RyanfaeScotland]

(i'm not shure if it is allowed to post or attach the whole disas file here...)

There probably aren't any rules against it but please don't. 

Congrats on the effort shown so far.

I have an on-going project you might want to check out where I'm attempting to document the whole of Toejam and Earl's disassembly. It's Sega Megadrive so different ball game from your work but the principles are the same: http://www.romhacking.net/forum/index.php/topic,21155.0.html

[hackfleisch]

Well, my first efforts where to decode the Ninendo logo at 0104h. Docs tell that it is bitmap, so a stream of bits was fast generated. But how to arrange them?

Because it is raw imagedata without any additional informations like size and such, i was not able to do it. So i search the web and found it described here http://stackoverflow.com/questions/21119904/how-to-decode-the-nintendo-logo-from-gameboy

Well, i guess this is one thing one just had to know about, to do the job. An i also guess there are many more things i had to learn about the console ;-)

Thanks a lot for pointing out that code is not Z80 but a similar kind of it. So i had to get the right disassembler to read the code. Any hints here?

Next thing would be to understand what is going on in the code. Because CPU codes don't do anything itself, i guess all depends on side effects of writing to magic memory addresses, registers or io-ports. So it seems a good idea to replace the addresses found in the disassembled code with functional names.

Yes, i will try to understand the whole game and every function. I think later i know where graphics, sound and text are stored and how they get used. Maybe i am able to change the code to do like i want (patch lifes, levels, spirtes or texts). This is my final goal.




March 19, 2016, 09:32:45 am - (Auto Merged - Double Posts are not allowed before 7 days.)

The Nintendo Logo in the header is $30 bytes, which is 3 tiles, which probably forms a 24x8 pixel image.  EDIT:  but even that seems small.... maybe it's stored 1BPP to make a 48x8 image?  I'd have to research it.  =x

It seems to be 1bpp. For some reason i don't know, the bytes of the logo:

Code: [Select]

CE ED 66 66 CC 0D
00 0B 03 73 00 83
00 0C 00 0D 00 08
11 1F 88 89 00 0E
DC CC 6E E6 DD DD
D9 99 BB BB 67 63
6E 0E EC CC DD DC
99 9F BB B9 33 3E
have to be arranged like this:

Code: [Select]

C 6 C 0 0 0 0 0 0 1 8 0
E 6 C 0 3 0 0 0 0 1 8 0
E 6 0 0 7 8 0 0 0 1 8 0
D 6 D B 3 3 C D 8 F 9 E
D 6 D D B 6 6 E D 9 B 3
C E D 9 B 7 E C D 9 B 3
C E D 9 B 6 0 C D 9 B 3
C 6 D 9 B 3 E C C F 9 E
As you can see, there are 6 bytes makeing up one row. And 8 rows for the whole image. In binary the later codeblock will look like this:

Code: [Select]

110001101100000000000000000000000000000110000000
111001101100000000110000000000000000000110000000
111001100000000001111000000000000000000110000000
110101101101101100110011110011011000111110011110
110101101101110110110110011011101101100110110011
110011101101100110110111111011001101100110110011
110011101101100110110110000011001101100110110011
110001101101100110110011111011001100111110011110

To improve the visuality of the image, i had replaced the zeros with blank:

Code: [Select]

11   11 11                             11       
111  11 11        11                   11       
111  11          1111                  11       
11 1 11 11 11 11  11  1111  11 11   11111  1111
11 1 11 11 111 11 11 11  11 111 11 11  11 11  11
11  111 11 11  11 11 111111 11  11 11  11 11  11
11  111 11 11  11 11 11     11  11 11  11 11  11
11   11 11 11  11 11  11111 11  11  11111  1111

But i'm really confused about the bit-arrangement of the bytes. It's mixed up with nibbles (4-Bits) of bytes found.

March 19, 2016, 10:06:06 am - (Auto Merged - Double Posts are not allowed before 7 days.)So, i downloaded "Gameboy Assembler Pro" from the utilities-section of this site for disassembling the code. Now, we can be shure that the instructions are correct.

Starting at 351h, here are the first lines. Could anybody help me to get their meaning?

Code: [Select]

#org 351

Label351:
 ld sp,E000  ; load 16-Bit register SP (Stackpointer) with e000h. SP is used for callback-addresses or pushed values
 ld hl,C000   ; load 16-Bit register HL (H and L) with value c000h
 ld bc,2000  ; load 16-Bit register BC (B and C) with value 2000h

Label35A:
 xor a,a      ; xor value a with itself. Maybe some kind of decryption? xor itself twice will result in originate byte
 ldi (hl),a    ; store value of "a" at memory location pointed by HL (initially c000h) and then increment HL (because of ldi command)
 dec bc      ; decrement BC value. It seems to contain the length of the block to handle
 ld a,b       ; load accumulator "a" with value of "b" (MSB of 16 Bit register BC)
 or a,c       ; binary or it with value of "c". So "a" is only zero if B and C have the value zero
 jr nz,Label35A   ; this seems to be a loop until register a (accu) is zero.

; ok, i'm pretty shure this will decode data inplace at memory location c000h for 2000h bytes.

ld hl,FF8A  ; this seems to be a magic address...
 ld c,75
 xor a,a

Label367:
 ldi (hl),a
 dec c
 jr nz,Label367  ; another decoder?



[AWJ]

The logo isn't in the standard hardware graphics format (as you've noticed, it's 1bpp and column-oriented) There's a small bootloader program in an internal ROM on the Game Boy CPU that decodes it (while pixel-doubling it in the process), displays it on the screen, and transfers control to the cartridge only if the logo matches the copy in the internal ROM byte-for-byte. It's a weak and silly "protection" against unlicensed games. You can find a disassembly of the bootloader online if you look.

xor a,a results in zero. The code you've pasted clears the memory from 0xC000 to 0xDFFF (which is the GB's 8KB work RAM)

[KingMike]

That last part in the disassembled code clears an additional part of RAM known as H-RAM.
It is 128 bytes from $FF80-FFFF, though as FFFF is another IO register, that's effectively 127 bytes usable.
My guess is the routine is not clearing $FF80-FF89 for some reason.

I'd really look up information on the GB hardware registers.
Such as the no$gmb/no$gba site, it has documents on the GB and GBA hardware, respectively.

[AWJ]

That last part in the disassembled code clears an additional part of RAM known as H-RAM.
It is 128 bytes from $FF80-FFFF, though as FFFF is another IO register, that's effectively 127 bytes usable.
My guess is the routine is not clearing $FF80-FF89 for some reason.

I'd really look up information on the GB hardware registers.
Such as the no$gmb/no$gba site, it has documents on the GB and GBA hardware, respectively.

FF80-FF89 is where most games copy their OAM DMA routine (which has to run out of H-RAM on the GB because of how the hardware works)

[hackfleisch]

xor a,a results in zero. The code you've pasted clears the memory from 0xC000 to 0xDFFF (which is the GB's 8KB work RAM)

Ups  Guess i was barking up the wrong tree...
At first "xor a,a" (or sometimes written as just "xor a") looks silly and one would expect "ld a,00h" to do the job. But looking into the Mnemonics tells that "xor" takes 1 machine cycle and "ld" takes 2. Wow, every cycle counts, eh? ;-) Why those Z80 developer did not give a command like "clear a" or such... well...

There's a small bootloader program in an internal ROM on the Game Boy CPU that decodes it

Oh, i see. I found the bootloader here http://gbdev.gg8.se/wiki/articles/Gameboy_Bootstrap_ROM. It would take some time for me to understand it.

Next, i will follow the advice to study the registers.

[AWJ]

Ups  Guess i was barking up the wrong tree...
At first "xor a,a" (or sometimes written as just "xor a") looks silly and one would expect "ld a,00h" to do the job. But looking into the Mnemonics tells that "xor" takes 1 machine cycle and "ld" takes 2. Wow, every cycle counts, eh? ;-) Why those Z80 developer did not give a command like "clear a" or such... well...
Oh, i see. I found the bootloader here http://gbdev.gg8.se/wiki/articles/Gameboy_Bootstrap_ROM. It would take some time for me to understand it.

Next, i will follow the advice to study the registers.

xor reg,reg is a standard assembly language idiom in many architectures to load a register with zero. As you've noticed it's smaller and therefore decodes faster and/or takes up less cache space (in modern CPUs with instruction caches) than an instruction with an immediate operand does.

Quick overview of the Game Boy memory map:

0000-7FFF is ROM. In almost all games, the first half (0000-3FFF) is fixed to the first 16K of ROM and the second half (4000-7FFF) is bankswitched. Confusingly, bankswitching is controlled by writing to a ROM address (the exact address depends on the cartridge; it's usually 0x2100)
8000-9FFF is VRAM. The first three quarters is tile patterns (384 8x8x2bpp tiles) and the last quarter is two 32x32 tilemaps. The CPU can only access VRAM during horizontal and vertical blanking periods (this is a common limitation on consoles, especially Nintendo ones) so you'll see VRAM access routines wrapped in busy-wait loops and/or guarded by IRQs.
A000-BFFF is cartridge RAM if present. If there's more than 8KB it's bankswitched just like ROM.
C000-DFFF is work RAM.
E000-FDFF is unused (games aren't supposed to access it, if you do you get a mirror of work RAM)
FE00-FE9F is OAM (the sprite list, 40 sprites x 4 bytes per sprite) Almost all games generate their sprite list in work RAM and upload it here via DMA, rather than accessing this range directly.
FE90-FEFF is unused (nothing here at all)
FF00-FF7F is all the video/audio/system registers.
FF80-FFFE is more work RAM. This RAM is actually on the CPU die and needs to be used for OAM DMA. Commonly-used variables are also often put here because the GB CPU has custom instructions to address this range in only one byte (like zero page on a 6502).
FFFF is one more register (interrupt enable, one bit for each hardware interrupt source)

In case it isn't clear, graphic tiles have to be copied from ROM to VRAM to be displayed. The GB video hardware can't directly access ROM. So graphics aren't necessarily in any standard arrangement in ROM, aren't necessarily aligned to 16 bytes or even in native format (text fonts are often reduced to 1bpp, other graphics might be compressed with various custom codecs)

[hackfleisch]

Thanks, AWJ. I also read about the memory map.
At this point i really like to make this console doing some action on my command to test the register functions. But therefore i need to have :
 - an GB-Z80 assembler to write my code
 - some kind of EPROM emulator with a cartridge pcb connector

A goal could be to simply show some graphics on the screen or play some sound. Of course i could write an ROM-file and execute it with an GB-Emulator on the PC. But this is only half fun. Maybe i can sacrifice an old cartridge to solder a circuit on the slot-lanes. Maybe there is a chance to emulate the ROM-Access from the console by using an Arduino or connect it to the PC somehow. Because access-rates are relativly slow, it should not be a problem. But to start, an emulator should be fine...

In case it isn't clear, graphic tiles have to be copied from ROM to VRAM to be displayed. The GB video hardware can't directly access ROM. So graphics aren't necessarily in any standard arrangement in ROM, aren't necessarily aligned to 16 bytes or even in native format (text fonts are often reduced to 1bpp, other graphics might be compressed with various custom codecs)

I understand! So this is also why it is not possible to simply "scan" for graphicsdata in the ROM. So the only place where it is structured and clear is at the moment it gets written to the VRAM. But if one would manipulate the content, he must look for the code which transfers this from ROM to VRAM.

If i understand the docs, the Nintendo logo is first placed from the ROM onto the screen, get scrolled and sound is played and after that, it is compared with the internal copy of the data. If they don't match, the console is halted. So i could place a custom image there and it should be displayed, or?
How about the checksums found in the header? I did not read anything about where they get used.

Hmm, i must admit that i first thought that older consoles are easier to hack and understand, but it seems contrarywise. Because nothing is really fixed and every game developer could do like he want (no Game Engines or IDEs) it's always reverse engineering. But it's fun! ;-)

[AWJ]

If i understand the docs, the Nintendo logo is first placed from the ROM onto the screen, get scrolled and sound is played and after that, it is compared with the internal copy of the data. If they don't match, the console is halted. So i could place a custom image there and it should be displayed, or?
How about the checksums found in the header? I did not read anything about where they get used.

If you know where to look you can buy (unofficial) flash memory cartridges to run homemade code on. No need to build one from scratch or hack up an original cartridge.

If you put a custom image in place of the Nintendo logo, the bootloader would display it but would then halt instead of running the cartridge program.

The bootloader does verify the header checksum at 014D. It doesn't do anything with the global checksum at 014E-014F (checksumming the entire ROM at the GB's CPU speed would take too long)

[FAST6191]

If you did want to build a flash cart rather than buy an everdrive (if you also want commercial games to do well) or something else (usually more homebrew focused, or more accurately focused on LSDJ/little sound DJ, at the cost of functionality with commercial games) a bit cheaper then http://reinerziegler.de/readplus.htm#Home%20made%20carts has some good info.

On arduinos and such like then most things I ever see are more for ripping games and saves than emulation of carts. People doing cart level emulation usually go right to FPGAs.

[hackfleisch]

Back again 
I've learned a lot by studing the bootloader of GB, not just how the logo is displayed...
There are a few things i don't understand.
In the graphics load routine starting at 0095h (please see my notes for questions):

Code: [Select]

LD C,A ; $0095     C is loaded with value of A
LD B,$04 ; $0096
Addr_0098:
PUSH BC ; $0098
RL C ; $0099   Rotates C left by one bit using carry-flag
RLA ; $009b   Same as C but without carry
POP BC ; $009c   Why is C rotated? Is lost now because of the pop...
RL C ; $009d   C is never bin used in code. What's it's purpose?
RLA ; $009f    A rotatet left again
DEC B ; $00a0
JR NZ, Addr_0098 ; $00a1     Will loop this 4 times. Every loop A is rotatet by 2 bits. After 4 iterations it the same as before?!?!
LD (HL+),A ; $00a3
INC HL ; $00a4
LD (HL+),A ; $00a5
INC HL ; $00a6
RET ; $00a7


[AWJ]

Back again 
I've learned a lot by studing the bootloader of GB, not just how the logo is displayed...
There are a few things i don't understand.
In the graphics load routine starting at 0095h (please see my notes for questions):

Code: [Select]

LD C,A ; $0095     C is loaded with value of A
LD B,$04 ; $0096
Addr_0098:
PUSH BC ; $0098
RL C ; $0099   Rotates C left by one bit using carry-flag
RLA ; $009b   Same as C but without carry
POP BC ; $009c   Why is C rotated? Is lost now because of the pop...
RL C ; $009d   C is never bin used in code. What's it's purpose?
RLA ; $009f    A rotatet left again
DEC B ; $00a0
JR NZ, Addr_0098 ; $00a1     Will loop this 4 times. Every loop A is rotatet by 2 bits. After 4 iterations it the same as before?!?!
LD (HL+),A ; $00a3
INC HL ; $00a4
LD (HL+),A ; $00a5
INC HL ; $00a6
RET ; $00a7


The "RL/RR" and "RLC/RRC" Z80 instructions are a bit confusingly named. Both of them do use the carry flag (otherwise they'd be "shifts", not "rotates").  RL/RR transfers the "head" bit into the carry flag, and the previous value of the carry bit into the "tail" bit. In other words it uses the carry flag as the 9th place in the rotation--if you do nine RLAs or RRAs, you end up with what you started with, both in the register and in the carry flag.  RLC/RRC transfers the "head" bit directly into the "tail" bit and also into the carry flag, simply discarding the previous carry bit.

The loop you've pasted doubles up bits while rotating them out of C and into A--it transforms ABCDEFGH into AABBCCDD. It's used to scale up the logo by doubling each bit/pixel.