UMDgen is the main tool most use for manipulating PSP Isos.
Text does not have to be in shiftJIS, or another known format.
If there is some shiftJIS then four broad scenarios.
1) You just found some debug strings, something the devs left behind, menus or something boring. AKA different parts of the game (even things on screen at the same time or one after the other) can use radically different things.
2) Formatting. I have seen games use 8 bit commands between 16 bit shiftJIS. This means your basic hex editor/viewer will trip up and when it blindly decodes things will fail, until the next 8 bit command and then it starts working again, and then another and it fails, and another and it works...
Should be fairly obvious in the hex ( http://www.rikai.com/library/kanjitables/kanji_codes.sjis.shtml
); most shiftJIS starts each 16 bit run with an 8 or 9, and failing that then E or F so you will see that in the hex but shifted along by 8 bits. You might also try compressing it (covered in a second) as already compressed text will tend not to compress much more where plain text will tend to compress at least a little bit.
3) Compression. The PSP is a modern device that is more than capable of handling a half decent compression routine. Kind of odd to use it on text when you have the same size of UMD either way but far from unheard of.
4) The text of the game is not text but pictures. Fairly common in low text games like puzzle games, though such a thing also extends to visual novels at times and it is far from unusual to see such a thing.
If you are playing with the eboot file then some later ones were encrypted, can be sorted now as we have all the keys so no need to fiddle with things on the PSP any more but you might have to consider that.
At this point we would normally point you at the getting started section on the main page/navigation of the site, around here will also be a starter pack thing linked up. I don't know of much explicitly aimed at the PSP for hacking but you might gain a bit from http://www.romhacking.net/forum/index.php/topic,14708.0.html
. I wrote it for the GBA/DS but the PSP shares a lot in common with the DS as far as general theories of operation -- at the time they were contemporary fairly high power devices with a focus on a file system and file formats, as opposed to most previous devices that were not on a CD or floppy disc that don't do file formats and file systems. Though I will note the PSP often used higher level formats -- you would be hard pressed to find anything other than the web browser for the DS which used JPG, PNG and BMP type formats within the game but I would not be surprised at all to find in a PSP game a common graphics, 3d modelling or similar format I could open up in an otherwise vanilla version of whatever popular or relevant graphics/3d modelling software of the time was.