Raj Here (making games more accessable to the blind)

Started by GoodRaju, August 23, 2012, 12:54:42 PM

Previous topic - Next topic

GoodRaju

Hello all. I have an eye condition known as Retinitis Pigmentosa. I used to be an avid gamer and loved to play all sorts of games, but as time passed, my vision slowly decreased to a point where I can't play most games and can't read the text on the screen.  My dream is to somehow make these main stream games accessible to the blind community by adding special sounds and figuring out a way to make the in game text read by a text to speech engine.

Obviously, I have no idea how to go about doing this, which is why I am here. Hopefully one or several of you intelligent people out there can help me make this happen. Any help in this matter would be greatly appreciated.

Thanks. looking forward to learning lots and hopefully making the dreams into a reality.

(MOD EDIT: added more descriptive thread title)

Garoth Moulinoski

Your wish may come true in a different way, via bionic eyes. Hopefully, this becomes easily accessible to most people with your condition.

Link
Who will quote me next?
Disclaimer: If it sounds wrong, I may have been posting while asleep.

creeperton

.
#2
.

GoodRaju

#3
Bionic eyes eh? That sounds pretty cool. Not sure how long until that becomes main stream, but definitely something to look forward to.

I am mainly interested in classic RPG games, like Final Fantasy, Chrono Trigger, Zelda etc and Pokemon series too. I am open to action games as well, but I think that would be a lot harder to do than RPG, could be wrong though. Thanks for your help! :)

henke37

You are in about the same boat as the people who hack in variable width fonts. You will need to inject code in the text drawing routine that performs black magic to escape the sandbox of the emulator and send the text to the speech engine. Well, or at the very least find a suitable memory buffer that contains the text to be shown.

This is extremely similar to the people who tries to hook up automatic translation tools to games. But those people are luckier than you. They have the game running as a native process. They don't need to deal with nonstandard character sets and nonstandard character encodings nearly as often. They don't need to poke a hole in the emulator, they can just hook things directly in memory. And they often cheat, they take advantage of the fact that nearly all games uses some system provided function to process the in game text. It's easy to find and hook those functions.

Your best bet is to try reading raw game memory with LUA scripts in the emulator.

Jorpho

Why not do something external to the emulator?  Take screen captures, OCR the text, and send the result to a text-to-speech converter?  Of course, it would be a little tricky to programmatically locate the text box and determine when the text has finished writing, but on the other hand since there's generally only one low-resolution bitmap font to worry about in each game, the OCR could be straightforward.  No more difficult than poking around in game memory, I would think.
This signature is an illusion and is a trap devised by Satan. Go ahead dauntlessly! Make rapid progres!

Klarth

Quote from: Jorpho on August 25, 2012, 12:07:19 PM
Take screen captures, OCR the text, and send the result to a text-to-speech converter?
My experience with free OCRs is poor accuracy.  You have to train the AI based on the particular font and you still get fairly poor recognition.  In the case where you do a pixel-matching "OCR", you'd need to make a new configuration for each individual game and can grow pretty complex due to layouts.  I would personally use an approach like henke suggests.

This is a very interesting project, at least to think about.  The core of this community has been to bring games to people who couldn't experience them before.  While the text-to-speech deal isn't impossible by any means, I'm more intrigued about how a blind person could navigate a visual game with acceptable accuracy.  Maybe have the controller vibrate when you walk into obstacles or have a "board" that represents what's currently on screen.  This board can dynamically change its texture as you move around the screen.  It could work for slower RPGs where you move one full tile per step.

Jorpho

I presume from Mr. Raj's OP that elements aside from text are still discernible.
This signature is an illusion and is a trap devised by Satan. Go ahead dauntlessly! Make rapid progres!

GoodRaju

Was busy over the weekend, just got a chance to check the responses. Interesting suggestions, both screen capture to OCR, and the other complicated sounding one. :)
What would I be looking for while using the script program?

As for navigating the world, simple sound cues would help a lot.  For example, let's say I run into a wall, tree, insert whatever border here, it would play some sort of sound that indicates that you can't go further in that direction. Like a sound of hitting your shoe on the wall as you step forward. That would help with the boundries of the world.

Pokemon for example, when walking through grassy areas, it makes that grassy sound for lack of a better description. That tells you where you are walking. Change in the sounds of the foot falls would be another big help to identify what you are walking on.

Little beeping beacons that can be activated to indicate directions. Let's say you are on the world map and trying to go to the Shinra HQ, a beeping indicates the direction that you need to travel. The beep getting louder as you get closer to it. Same can be done for loot and other items. Once people become familiar with the lay of the land, they won't even need to use those and just sort of go by memory.

In the above example of the Pokemon game, there are enough sound cues in the game already to make navigation fairly easy, the only thing that prevents me from actually doing anything significant now days is the text. So if all the text could be read as it appears on screen, via a screen reader like JAWS or standard TTS, then that game would be considered as accessible.

Same thing with other RPG games, Final Fantasy, Chrono Trigger. Just making the text readable would bring the game to like 75 percent accessible. The remaining can be done by adding those simple sound beeps and boops as mentioned.

Sorry for the rambling. :)

Klarth

This is what I would do if I were interested in doing only text to speech.

1. Identify emulators with LUA scripting built-in.
2. Identify a well-documented game for a system with an emulator that supports LUA.  The game should have a simple text system, table available, and some documentation regarding its text routine.
3. "Hook" the text routine for the game for the text pointer to be loaded.  Then read in the text data via the LUA script and convert it to a standard encoding using the table.
4. Use a text-to-speech library to output sound.  Dragon Speaking Naturally has one of these (not sure if free), but there may be some decent ones that don't require purchase of a program for the end user and the purchase of a Software Development Kit for you.

The core of your script will be largely reusable, but this simple model won't work with all games with how I described the hook in #3.  You likely will have to hook the game where it prints each visible character, which means waiting until the end of a line, page break, or end message until you start text to speech output.  This isn't exactly desirable, but not bad in games where you can greatly speed up text.

If you do use a good text to speech library, you should be able to interpret the script so that you can use a different voice for each main character and maybe a couple shared between NPCs.

Spikeman

Quote from: henke37 on August 25, 2012, 09:58:30 AM
You are in about the same boat as the people who hack in variable width fonts. You will need to inject code in the text drawing routine that performs black magic to escape the sandbox of the emulator and send the text to the speech engine. Well, or at the very least find a suitable memory buffer that contains the text to be shown.

This is extremely similar to the people who tries to hook up automatic translation tools to games. But those people are luckier than you. They have the game running as a native process. They don't need to deal with nonstandard character sets and nonstandard character encodings nearly as often. They don't need to poke a hole in the emulator, they can just hook things directly in memory. And they often cheat, they take advantage of the fact that nearly all games uses some system provided function to process the in game text. It's easy to find and hook those functions.

Your best bet is to try reading raw game memory with LUA scripts in the emulator.

The GBA has a debug function that's used to print text to a window. I know that VBA supports it. I did a proof of concept hack about a year ago (for the purpose of trying to hook an automatic translation program onto the emulator). If a screen reader already supports reading text from programs, this would probably be an easy way to make it work. If there are issues, VBA is open source and could be modified to hook into a screen reader. Personally, I think that using this debug function is easier than the LUA option (although, a LUA script might be able to hook straight into the screen reader.)

Raj, if there is any particular GBA game you're interested in playing I'd be willing to make a hack. Could you provide some more details on how your text-to-speech software works?
Open Source Hacking Projects: Guru Logic Champ, Telefang 2, (Want more? Check out my GitHub!)