News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: Shiren the Wanderer 4 Plus (PSP) - English Translation [WIP]  (Read 3714 times)

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Shiren the Wanderer 4 Plus (PSP) - English Translation [WIP]
« on: August 29, 2020, 12:54:24 pm »
Hey there, it's time for another Mystery Dungeon post!
So this Shiren 4 project is partly why the Asuka Kenzan translation has been on hold, and it's far enough along now that I think it's time to share some of the progress.

The project started when someone who goes by DhrGR managed to write some scripts to handle text extraction & insertion for the game, and I've been working on the translation for a few months now. (Menus, signs, dungeon messages, tutorials, main story events/scenes, and some monster descriptions and other bits and pieces are translated).

Some screenshots, showcasing both PSP and DS versions:

(Main menu)


(Village)


(Monster library)

And a video of the intro, if you want to see it in action: https://www.youtube.com/watch?v=RcEGtXPyCw4

Exciting stuff, yeah? But unfortunately, it's not all smooth sailing. We've encountered a number of issues that need to be solved on the hacking side, which we could use some help with.

1. "「" needs to be changed to ":" when characters talk.


2. Either the font width needs to be narrowed a bit, or window sizes need to be expanded a bit.


3. The control character to make a line break to a new line on a word doesn't appear to be working as expected on the DS version.


4. And most severely, the game now crashes at the title screen on the PSP version, making the game unplayable.
Video of the crash: https://youtu.be/p_x6TC6asSc

What we think is happening for No.4 is that English text takes up more space than Japanese text, so when it gets read into memory, it's flowing over/overwriting other things, making the game crash. The current idea we have is to somehow make it so that the game uses UTF-8 or ASCII instead of UTF-16LE, since we don't need the Japanese characters in the translation. But we don't exactly know how that could be accomplished.

example: Currently "E" = "45 00" when it could just be "45"


So yeah, there you have it. Thanks for reading, and please let me know if you'd be interested in helping solve these issues.
« Last Edit: April 10, 2021, 10:56:14 am by sharksnack »

[Unknown]

  • Jr. Member
  • **
  • Posts: 49
    • View Profile
    • PPSSPP
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #1 on: August 30, 2020, 03:22:32 am »
How are you inserting?

One thing I like doing with text insertion is "translating" in a dumb way, replacing each letter with i.e. 'A' (except control codes.)  If doing this causes problems still, it may indicate your insertion is buggy.  If it works fine and everything is 'AAAAA', then it's probably a RAM problem indeed.

Assuming you're using PPSSPP, make sure you have ignore illegal reads/writes and fast memory disabled.  Also look at the log for memory allocation failures - that would be a sure sign that it became too big.  RAM is not fixed-allocated in PSP games (unless the text is in the ELF), so usually length problems show up as failures to allocate memory.

Also, I've recently - at request for another translation - created a branch for PPSSPP that allows you to crank up the total available RAM in a plugin.  See here: https://github.com/hrydgard/ppsspp/pull/13335.  This may be a decent way to validate if it's memory constraints as well.

As far as UTF-16 to UTF-8/ASCII, the viability of that depends on...
 - Are you sure it's UTF-16 and not UCS-2?  It's likely not handling surrogate pairs unless it calls sceCcc or something.
 - Are control codes also 2 bytes or are they sometimes an odd number of bytes (i.e. 1 or 3)?
 - Do you know yet how many different places reference the text?  for example, strlen, drawing, etc. funcs.

A good way to determine the first is to use a memory breakpoint in PPSSPP.  Set it on the string and review the code reading the bytes.  If it's a lhu/lh, then it means there's probably other places it assumes alignment, and you'll need to find them.

That leads to the final question.  Aside from using breakpoints (and the stack traces when they hit), the next step is to see if there's a code pattern you can search for.  For example, if the game always uses a 0xFFFF control code, you might review each of the 0xFFFF occurrences in the ELF (this is far more realistic if it's an uncommon sequence, 0xFFFF would be terrible.)

-[Unknown]

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #2 on: August 30, 2020, 06:50:25 am »
Thanks for the detailed response! I work mostly on the language/translation side, but I'll try to answer to the best of my ability.

Quote
How are you inserting?

The text insertion script can rebuild a 1:1 copy of the original file, and does some form of alignment to 16 bytes since the original file was aligned that way. The crashing issue occurs after a certain amount of translation into English has been made across all files, rather than editing any particular file. (Text ins't in the ELF)

I disabled those settings and checked the log, and the error that shows up when the game crashes is:
"E[MEMMAP]: Core\MemmapFunctions.cpp:55 Unknown GetPointer 00006240 PC 08b14478 LR 08b14480"


(The errors above that one appear even in the unmodified game)

I'll see if I can build that branch of PPSSPP and give the increased RAM via plugins a try.

I don't know if this would matter, but the PSP version is an enhanced port of the DS version - so I wonder if the Nintendo DS uses fixed-allocated RAM, and maybe they left that unchanged? (I'm not really familiar with the DS, so that might not be the case at all)

Quote
Are you sure it's UTF-16 and not UCS-2?

I'm not sure on this one. I was told that it's UTF-16LE by someone who had previously done some work on the game, but they disappeared a while back so I don't think I can contact them to verify - is there a way I can confirm myself? (Both HxD and wxMEdit don't appear to have UCS-2 as a display option)

Quote
Are control codes also 2 bytes or are they sometimes an odd number of bytes (i.e. 1 or 3)?

I'm not sure. The text in the binary file is wrapped (sir0), and the extracted text I work with is temporarily converted to UTF-8 (JSON), so I haven't personally seen what the control codes look like in the binary.

Quote
Do you know yet how many different places reference the text?  for example, strlen, drawing, etc. funcs.

We don't know yet. Basically all that's happened on the technical/hacking side of the project is text extraction and insertion, and as far as I'm aware, no one has used a debugger to investigate anything. I'm willing to try it myself, but the translation workload is immense already and the others are currently busy with real life stuff or such, so ideally it'd be great if another interested person could take that up. (Otherwise it'd likely be months from now before I can get to it)

[Unknown]

  • Jr. Member
  • **
  • Posts: 49
    • View Profile
    • PPSSPP
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #3 on: August 30, 2020, 10:53:17 am »
Hm, if the game calls sceKernelStopUnloadSelfModuleWithStatus, that's probably a bug and it'd be good if you could create an issue on github.  I don't have the game to look into it currently, but any game calling that means a bug in PPSSPP.

The next three errors are possibly normal, though, and may even happen on actual PSPs.

The get pointer error definitely indicates a game crash, so it does look likely that the larger file caused allocation issues unless it accidentally overwrote something incorrectly.  If the game uses a lookup table or has headers indicating the sizes of things, and those aren't updated, that could cause this too.

-[Unknown]

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #4 on: September 01, 2020, 08:51:21 pm »
I managed to build the branch of PPSSPP you linked to, but it looks like I might be out of luck on Windows for compiling the prx plugin. (It seems pspsdk requires psptoolchain, which is only for Linux/OSX).

Would it be possible for you to upload a sample prx plugin? I'd like to test if the RAM expansion can be used to side-step the crashing issue.

And noted, I'll create an issue for the sceKernelStopUnloadSelfModuleWithStatus error later this week.

[Unknown]

  • Jr. Member
  • **
  • Posts: 49
    • View Profile
    • PPSSPP
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #5 on: September 02, 2020, 12:47:17 am »
I recommend minpspw on Windows:
https://sourceforge.net/projects/minpspw/

You can expand RAM with just an ini though, as noted in the pull description - it does not actually require a prx.  Just leave type and filename out.

-[Unknown]

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #6 on: September 14, 2020, 05:04:54 pm »
Thanks, I gave the expanded RAM option a try now, and got a similar result (didn't have the load plugins option turned on the first time):

Code: [Select]
Critical: [g_MemorySize < MAX_MMAP_SIZE * 3] ACK - too much memory for three mmap views.
...
34:13:881 user_main    D[SCEKERNEL]: HLE\sceKernelInterrupt.cpp:626 sceKernelMemcpy(dest=040b56f8, src=09e05e08, size=8)
34:13:881 user_main    D[SCEKERNEL]: HLE\sceKernelInterrupt.cpp:626 sceKernelMemcpy(dest=040b57f8, src=09e05e10, size=8)
34:13:881 user_main    D[SCEKERNEL]: HLE\sceKernelInterrupt.cpp:626 sceKernelMemcpy(dest=08d0f360, src=08d76670, size=5120)
34:13:881 user_main    D[SCEKERNEL]: HLE\sceKernelInterrupt.cpp:626 sceKernelMemcpy(dest=08d10760, src=08d77a70, size=5120)
34:13:881 user_main    D[SCEKERNEL]: HLE\sceKernelInterrupt.cpp:626 sceKernelMemcpy(dest=08d11b60, src=08d78e70, size=5120)
34:13:881 user_main    D[SCEKERNEL]: HLE\sceKernelInterrupt.cpp:626 sceKernelMemcpy(dest=09df6310, src=08fd28a0, size=3744)
34:13:884 user_main    W[MEMMAP]: Core\Core.cpp:423 Read/Write Block: Invalid address 00006240 PC 08b14478 LR 08b14480

Looks like the "ACK" error might be something related to 32-bit systems judging by the comment, so I don't think it means anything? (I'm on Windows 10 64-bit)

So now I'm not sure what the next step would be... maybe the game has a hard-coded size for the text somewhere, seeing how expanded RAM didn't seem to make a difference?

(Side note: all 366 monster library descriptions have been translated now. Next up is item descriptions and more npc dialogue)

[Unknown]

  • Jr. Member
  • **
  • Posts: 49
    • View Profile
    • PPSSPP
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #7 on: September 14, 2020, 08:52:23 pm »
Oops, I'll fix that check.  It's just a bug in the assert.

I do think this indicates that something is overwriting something else.  Could be hardcoded sizes, or just another field in an existing file that indicates the size and needs updating.

There are probably two paths to take:

 * Determine what bytes it's reading that become wrong, set a breakpoint there, and find where it's overwriting to figure out how it determines the size.
 * "Bisect" the problem by reducing the inserted size in halves, until you narrow down the max size it happens at.  This might help you determine a reduced test case where you change the minimum number of things.

We actually don't know it's about size yet.  For example, it could be there's a formatting code that's wrong in just one place, or some string that's not terminated in a special way you don't realize it needs.

-[Unknown]

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #8 on: September 17, 2020, 01:45:18 pm »
I managed to narrow it down to where it crashes after you add a single letter now.
I only modified explain.bin for this test, and the following sizes are what's displayed when right-clicking the file in explorer and selecting properties.

original/untouched:

size: 277 KB (283,648 bytes)
size on disk: 280 KB (286,720 bytes)

doesn't crash, showing the last entry:
Code: [Select]
~Special Abilities~
 Absorbs status conditions.
~Description~
 Elephant that absorbs various
 status conditions to power up.
 It won't grant wishes when it
 absorbs status conditions!
AA

size: 341 KB (349,520 bytes)
size on disk: 344 KB (352,256 bytes)

crashes after adding letter 'A' to the monster description:
Code: [Select]
~Special Abilities~
 Absorbs status conditions.
~Description~
 Elephant that absorbs various
 status conditions to power up.
 It won't grant wishes when it
 absorbs status conditions!
AAA

size: 341 KB (349,536 bytes)
size on disk: 344 KB (352,256 bytes)

And for whatever it's worth, it appears the explain.bin file for this test case (crash with 'AAA') spans from 95AACE0 to 95FC160 in memory.

8800000 + 00DAACE0 = 95AACE0
8800000 + 00DFC160 = 95FC160

Would the next step be to do something with a memory breakpoint there? I haven't really used a debugger before, but I have a general idea how they're used... and it looks like PSP is MIPS R4000, so this would apply, I think? https://groups.csail.mit.edu/cag/raw/documents/R4400_Uman_book_Ed2.pdf

Edit - Not sure if this helps either, but I tried comparing the end of the explain.bin section in RAM between the different test cases, and this is what it looks like:
« Last Edit: September 17, 2020, 05:49:49 pm by sharksnack »

[Unknown]

  • Jr. Member
  • **
  • Posts: 49
    • View Profile
    • PPSSPP
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #9 on: September 19, 2020, 11:43:33 am »
Size on disk is just about your disk's sector sizes, and won't affect insertion or RAM.

That range (095AACE0 - 095FC160) would be 332,928 bytes.  If it's really that fixed size, that would explain the crash when you go over that size.

The doc you linked is better for the PS2, but should largely work for the PSP.  Note that there are no double (64-bit) instructions on the PSP.  Its registers are all 32-bit.

You can set a memory breakpoint at any address and then do something to make the game access that address.  When it trips, it'll show you which instructions accessed that memory.  For example, let's say it's read directly from file via sceIoRead.  If you set a write breakpoint at 095AACE0, it'd trip on the sceIoRead call (but because the write happened within HLE, it will highlight another instruction - view the log to see what thread it tripped on.)

The end you've highlighted seems to have a series of pointers after it - which seem to point into your space.  It might even be the problem is after that sequence.

-[Unknown]

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #10 on: September 19, 2020, 02:19:50 pm »
Thanks, I think I'm starting to get the hang of using PPSSPP's debugger a little bit now.

You're right that there's a pointer offset list after the text/content of explain.bin, I forgot to account for that.
(It's a SIR0 wrapped file: https://projectpokemon.org/docs/mystery-dungeon-nds/sir0siro-format-r46/)

So I was wrong about it being a fixed range, I think I found where the start address gets set (breakpoint at 0x08BD5C74 and repeated until I saw explain.bin), and the size differs between the three versions:

original:
start address = 095BAE30
size = 45400

no crash (AA):
start address = 095AACE0
size = 55550

crash (AAA):
start address = 095AACD0
size = 55560

I noticed that the AAA version of explain.bin wasn't aligned like the other two in the pointer list section, so I added some additional letters so that it lines up - but it still crashed in the same spot.

It seems the game loads a bunch of files one after the next from the message folder, so maybe if I can figure out a list of them looking at the RAM dump, I can get an idea how much space is reserved for text total? (If it's a defined value)

The crash happens a little while after explain.bin is read into memory, so I still don't have an exact idea how explain.bin might be messing things up. (I set a read breakpoint on the explain.bin text, but it didn't seem to result in a hit before the crash)

Image showing the start addresses/size of explain.bin for each version:

Ponch

  • Jr. Member
  • **
  • Posts: 8
    • View Profile
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #11 on: September 20, 2020, 10:46:53 pm »
I don't have any technical skill to help , but just wanted to show my support.
Progress looks great! The Shiren franchise is a venerable one and a translation of another of it's titles would more than welcomed. Wish you success!

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Re: Shiren the Wanderer 4 (DS/PSP) - English Translation [Help Needed]
« Reply #12 on: April 10, 2021, 10:55:19 am »
A dormant project thread often means a project is on hiatus, but not in this case! I'm just way more active on discord, and kind of forgot that I had a project thread here.

So what's new?
  • PSP version no longer crashes on startup. (Can switch between UTF-16LE and ASCII on demand)
  • Name entry screen options are in English.
  • The「 has been changed to : when characters talk.
  • Font widths have been adjusted, making text look much nicer.
  • Some windows were made a bit larger to accommodate longer strings.
  • Most gameplay areas have been translated, so it's basically NPC dialogue and post-game story scenes left.

The hacking/technical items above are all thanks to Arc Impulse. It really wouldn't have been possible without their help, and I can't thank them enough. There's still a lot of work left, but things are progressing pretty nicely.

Here's some gameplay showcasing some of the listed changed: https://youtu.be/tbD4m4ZVdnE

And now for some bad news:

1. I found the version 3.10 update for the game, but the game no longer runs in PPSSPP if I apply the patch using iso_tool 1.981 and dump the game (to ISO format) from my PSP to PC. I'm guessing it's some sort of DRM that I have no clue how to bypass, since the update works fine on console:



It's not exactly a dire thing, since version 2.00 works. But there's a bug in 2.00 that freezes the game under a certain condition, so it would have been nice to find a way to get 3.10 working. (Plus 3.10 has an extra game settings option)

2. The translation for the DS version of Shiren 4 is discontinued. It's too much of a hassle to work within the limitations of that version, and I don't see the point in spending time on a vastly inferior experience.

[Unknown]

  • Jr. Member
  • **
  • Posts: 49
    • View Profile
    • PPSSPP
Re: Shiren the Wanderer 4 Plus (PSP) - English Translation [WIP]
« Reply #13 on: April 10, 2021, 01:39:12 pm »
Hm, the magic number should normally be 0x5053507e not 0x50535000.  Maybe the patch tool didn't update that right and maybe the PSP doesn't actually care.  If you can compile PPSSPP (it's easy) try changing this in Core/HLE/sceKernelModule.cpp:

Code: [Select]
if (*magic == 0x5053507e) { // "~PSP"
To:

Code: [Select]
if ((*magic & 0xFFFFFF00) == 0x50535000) { // "~PSP"
And see if anything improves.  Alternatively, hex edit that file on the ISO to put 7E there in the header instead.

-[Unknown]

sharksnack

  • Jr. Member
  • **
  • Posts: 28
    • View Profile
Re: Shiren the Wanderer 4 Plus (PSP) - English Translation [WIP]
« Reply #14 on: April 10, 2021, 04:27:43 pm »
It looks like that got rid of the magic number error, but it's still failing to load /NPJH50698/f5psp.sprx
(I moved the folder with that file to PPSSPP's memory stick, so I don't think that's the issue?):



I think you might be right that iso tool is somehow messing things up.
« Last Edit: April 10, 2021, 04:51:00 pm by sharksnack »

[Unknown]

  • Jr. Member
  • **
  • Posts: 49
    • View Profile
    • PPSSPP
Re: Shiren the Wanderer 4 Plus (PSP) - English Translation [WIP]
« Reply #15 on: April 10, 2021, 06:21:10 pm »
Seems like that means the header is corrupt.  It's a bug that the error code is wrong, but it must mean the header indicated a size larger than the ELF size.

-[Unknown]