[RHDNBot]

Update By: Nightcrawler

Nightcrawler has been working hard on some things that may be of interest to the Translation Community. The first of which is a document aiming to be: A reference to the file format, explanation for newcomers, and a helpful reference for programmers. A first draft has already been written. Suggestions and feedback are welcome in hopes that some may choose to adopt it. Adopting a standard would allow for standardized table file features and compatible utilities going forward. Additional details can be found on TransCorp.

The second item of interest is a yet to be named GUI Generic Dumper aiming to meet or exceed Cartographer in functionality for it's first release. It will be the first utility to follow the new table file format standard (when finalized). An option to output in Atlas 1.1 compatible format is expected. Complimentary native insertion functionality is planned, but the first release will focus on dumping functionality only.

In addition, Nightcrawler has been working on several other items. He's been tinkering with developing UVWF (SNES Universal VWF) to be applied to the sprite based text in Terranigma. He's been working on continuing the Tenshi No Uta Project. Lastly, he's been working with electrochip on experimenting with code that doesn't work on a home made flash cart, but does on copiers and BSNES to better understand bare bones initialization on the hardware for a potential learner's kit.

Plenty of additional information on all subjects can be found over at TransCorp!

Relevant Link: (http://transcorp.parodius.com)

[tc]

Oh wow! I always treated my press for a Terranigma VWF as a running joke. I had NO idea anyone really was listening to me.

[Next Gen Cowboy]

This is really something, considering he has been working on it for a long while now, with tons various complications, and than bam! Must have been hauling ass the last few weeks?

[C_CliFF]

I'm really looking forward to this. Keep up the good work NC! 

It's also nice to see that Mattias is still active in our small swedish RH community. I thought I was the only one left.
We talked about the VWF in Terranigma seven years ago so I'm looking forward to see this in his translation.

Say hello from me if you talk to him. 

-C_CliFF

[Nightcrawler]

Only interest in Terranigma?

Nobody but Gil Galad and DaMarsMan are interested in the standard?  Wow! I must have a golden standard that will be adopted by all. Or... maybe it's so bad, nobody can even begin to talk about it and it will just be totally ignored. Ha! At least it will be over-glorified notes for myself then! 

[DarknessSavior]

Only interest in Terranigma?

Nobody but Gil Galad and DaMarsMan are interested in the standard?  Wow! I must have a golden standard that will be adopted by all. Or... maybe it's so bad, nobody can even begin to talk about it and it will just be totally ignored. Ha! At least it will be over-glorified notes for myself then! 

I'm more interested in your dumper/inserter than anything else. I haven't had any trouble working with tables in the past, regardless of the way the game handled text. Dumping and inserting text, depending on the game, can be a real pain, though.

~DS

[Gideon Zhi]

Honestly, I'm almost past the point where I need a dumper. At this point, most of my scripts are at least translated if not inserted. The generic dumper could save some time in some cases since it dumps into Atlas format, but I'm at the point where the games it'll be useful on would almost require a custom job, or at the very least a hacky workaround (such as dumping from RAM and inserting to ROM, which no generic dumper is probably going to be able to handle.)

[Pennywise]

I suppose I'm interested in it. For the most part, Cartographer gets the job done, but a GUI would be nice.

[Nightcrawler]

DS:
The dumper/inserter will use the established table standard, so they go together. You say you haven't had any trouble working with 'tables' in the past. What tables? The one you use with Windhex? The one you use with Thingy? The one with Hexposure? The one with Translhextion? The one you use with your own custom utility? The one with Atlas? That's the kind of thing I'd like to see go away moving forward with utilities. We can have some compatibility between utilities, set an abstraction layer for tables (more interesting for utility creators), and rely on standard feature set. Oh well. Our community likes to stay in the stone ages. Why should this be any different? They like their stones.

Gideon:
So, we've declared nothing here is of use to you? Fantastic! Wait... How is this contributing to the thread again?


Boy, I'll come back in 10 years and try to push you guys forward again. Timing seems to be way off on this one. 

[DarknessSavior]

DS:
The dumper/inserter will use the established table standard, so they go together. You say you haven't had any trouble working with 'tables' in the past. What tables? The one you use with Windhex? The one you use with Thingy? The one with Hexposure? The one with Translhextion? The one you use with your own custom utility? The one with Atlas? That's the kind of thing I'd like to see go away moving forward with utilities. We can have some compatibility between utilities, set an abstraction layer for tables (more interesting for utility creators), and rely on standard feature set. Oh well. Our community likes to stay in the stone ages. Why should this be any different? They like their stones.

Gideon:
So, we've declared nothing here is of use to you? Fantastic! Wait... How is this contributing to the thread again?


Boy, I'll come back in 10 years and try to push you guys forward again. Timing seems to be way off on this one. 

I've used just about every hex editor there is for romhacking, including Thingy, Thing32, Goldfinger, Windhex, Hexposure, and I'm sure I could dig up some others. And I've used the same type of table for each one, and never had an issue. I've used them to dump romjuice scripts (though, I admit I make a separate table with the "\r" and "\n" stuff) and insert Atlas ones with no problem.

If anything, I think perhaps there should be a standard for the basic <End> and <Linebreak> control codes. Some sorta standard value so that you don't have to set up the table to dump them properly.

But I'm more interested in the dumper/inserter because I don't really like romjuice, Cartographer is still unfinished (and I couldn't get it to dump ActRaiser 2, for some reason), and Atlas has always given me trouble. I'd like something a little more streamlined and simple to work with. Oh, and a GUI. I don't like stones. =P

~DS

[tc]

Closest thing to 'tables' I'd poked around in a few minutes, was an NES game typo. Turns out the corrected version was too long to fit and would've needed real hacking.

[Tater Bear]

Personally it sounds like a great idea. I am all for it, put an article on the site for newbies to use the new proposed standard and the new generation of romhacker can/will reap the rewards and benefits 

[C_CliFF]

Only interest in Terranigma?

Actually, I was more interested in the generic dumper. The problems I've had in Cartographer has been about the base pointer function. How will this function work in your program? Will you be able to add and subtract values? (the Base Pointer)

-C_CliFF

[Nightcrawler]

In the testing version right now, all pointers are added to the global offset. If the pointers are relative, you'd set the global offset to the pointer table start. If the global pointer is 0, the pointers are effectively absolute. Then, there's the option to use the pointer location as the base.

It certainly has not been finalized it yet. If you give me a few scenarios of what you've been dealing with and how Cartographer doesn't work, I can see if they fit into the system I have or if I can do something to handle them. I certainly have not thought of every pointer scenario and would be happy to take a look at as many as possible.

[Kajitani-Eizan]

one problem i've had with cartographer is when i have a bunch of pointers (GBA; 4 byte absolute pointers with 0x08000000 offset to be subtracted) in an area, but they're not necessarily neatly arranged. they might have some other data in between them, not necessarily in any easily-defined pattern. i want to dump all of them. so i tried just having it dump every 4 bytes. but cartographer crashes if you try that, because instead of hitting an invalid pointer and going "oh, that's invalid, let me just output an error for that one and move on to the next", it goes "durr no error checking whatsoever, i are crash"

and another feature request: specify a pointer type and possibly a pointer range, specify the text block, and then dump from the text block but search for pointers as you go. so kind of like the RAW method, except you actually get pointers out of it. this is pretty much the method i've been using for my personal custom tools but it would be nice for it to be universally available.

depending on how you implement everything, this might be a pain, but one more feature request: same as above, except it sorts the text/#WRITE commands by location of the pointer. so if the text block is World <$00> Hello <$00>, and it finds a pointer to World at $200 and Hello at $100, the output is #W32($100) / Hello<END> / #W32($200) / World<END>, or somesuch.

[Nightcrawler]

one problem i've had with cartographer is when i have a bunch of pointers (GBA; 4 byte absolute pointers with 0x08000000 offset to be subtracted) in an area, but they're not necessarily neatly arranged. they might have some other data in between them, not necessarily in any easily-defined pattern. i want to dump all of them. so i tried just having it dump every 4 bytes. but cartographer crashes if you try that, because instead of hitting an invalid pointer and going "oh, that's invalid, let me just output an error for that one and move on to the next", it goes "durr no error checking whatsoever, i are crash"

I think you are asking for the same or similar thing DaMarsMan did on TransCorp:
http://transcorp.parodius.com/forum/?num=1273690996/4#4

I would support pointer lists for properly supporting these kinds of cases. The drawback is you would need to create the pointer list manually. It would probably be the responsibility of another utility to help automate the scanning or detection of  pointers and generate a list so you don't have to do it manually. I am not too interested in features that don't result in something tangible. Such 'scanning' functions will always generate false positives and misses.

I could also probably make your workaround of just dumping every 4 to work. The only issue I see is probably what crashes Cartographer. You get a pointer that exceeds the bounds of the file. I could handle that exception and output an empty string for something like that. However, it does make sense to generate an error upon an invalid pointer. I suppose an option to force/skip invalid pointers could accommodate both. I'll give it some some thought. I wouldn't imagine it comes up too often where you'd actually want to dump invalid pointers. At least it doesn't for me. Typically an invalid pointer indicates to me that I don't really understand what's going on and need to figure it out so it's not invalid anymore.

Along these lines, I've always thought about a 'rule' based scanner that can find proper pointers based on the rules you feed it about the game's scripting blocks. It's never 'random' spacing between pointers. You just need to learn what the data is in between so you can scan the block and find the pointers using known information. I've done something similar for projects of mine. I parse the block with known rules and find valid text to dump.

With linked entries you can almost do that already if you know enough commands. If you set up your table fancy with line breaks after known linked entries/commands and their parameters, you can single out the actual text when it gets to it. Basically, you set up your table to 'parse' the data like:

Say you have this hex string:

$f2 $01 $03 $25 $37 $55 $45 valid text here $45 $62

In your table you'd have something like:

$f2=\n\r<create_window>,3
$37=\n\r<clear_window>,1
$45=\n\r<text_command>/r

You'd get an output of something like:

//<create_window><$01><$03><$25>

//<clear_window><$55>

//<text_command>
//valid text here

So, you see you can also pretty much parse the block that way too with a RAW dump.

and another feature request: specify a pointer type and possibly a pointer range, specify the text block, and then dump from the text block but search for pointers as you go. so kind of like the RAW method, except you actually get pointers out of it. this is pretty much the method i've been using for my personal custom tools but it would be nice for it to be universally available.

How does that differ from what you can accomplish with the pointer table range here?

http://transcorp.parodius.com/scratchpad/TextAngel.png

Not sure what you mean by searching for pointers.

depending on how you implement everything, this might be a pain, but one more feature request: same as above, except it sorts the text/#WRITE commands by location of the pointer. so if the text block is World <$00> Hello <$00>, and it finds a pointer to World at $200 and Hello at $100, the output is #W32($100) / Hello<END> / #W32($200) / World<END>, or somesuch.

I guess I'm not getting why you keep saying you're in RAW mode, but you're finding pointers somehow. If you're RAW mode, the only thing you can derive is knowing where an string end token is and assuming the next string starts right after that. Even then, you've collected all the information. What do you do with it then to 'find' pointers? I think this circles back to non tangible scanning algorithms.

[Kajitani-Eizan]

Along these lines, I've always thought about a 'rule' based scanner that can find proper pointers based on the rules you feed it about the game's scripting blocks. It's never 'random' spacing between pointers.

yes, it can be "random". especially if the pointers are simply embedded in the code in between functions, or if they just dumped a bunch of data somewhere that happens to include pointers as well as other stuff. that's why you need pointer searching/scanning, or brute force dumps. and you do get something tangible from those -- the output atlas script will have #WRITEs for each pointer it found. you can modify those at will to remove false positives and perhaps add in pointers that the routine might have missed. (though it really shouldn't be missing any, unless they fall outside the pointer search area you specify.)

How does that differ from what you can accomplish with the pointer table range here?

assuming that screen is showing functionality similar to what is currently in cartographer, it will go through a pointer table and dump the pointers and the text that the pointers point to. that is not the same as what i am saying. what i am saying is for it to go through the text area and find the pointers that point to the text, and dump those pointers and the text.

I guess I'm not getting why you keep saying you're in RAW mode, but you're finding pointers somehow. If you're RAW mode, the only thing you can derive is knowing where an string end token is and assuming the next string starts right after that. Even then, you've collected all the information. What do you do with it then to 'find' pointers? I think this circles back to non tangible scanning algorithms.

? what is a "tangible" scanning algorithm? here is the sort of algorithm i am proposing and have implemented in my (crappy) tools, and i know others have as well:

1. aha i've found the start of a string.
2. let me make a note of what offset this string starts at.
3. let me search a specified area for pointers that point to this offset.
4. ok, i found it. let me dump that pointer along with the string.
5. ok, next string. goto 2.
6. repeat until end of text block is reached.

if you're confused about how one might search for pointers... for example, the GBA addresses the ROM starting at 0x08000000. so, if my string is at 0x00001000 in the ROM, i can add 0x08000000 to that to get 0x08001000, and search for this value. so i'd start from the beginning of the specified "pointer search area", and look for a group of 32bit-word-aligned bytes that look like 00 10 00 08. if i find that, i have potentially found a pointer to the text at 0x00001000. obviously that particular set of bytes could very easily be a false positive, but something like 56 32 1C 08 is much less likely to be a false positive. and of course you can double check it yourself, manually.

[Nightcrawler]

yes, it can be "random". especially if the pointers are simply embedded in the code in between functions, or if they just dumped a bunch of data somewhere that happens to include pointers as well as other stuff. that's why you need pointer searching/scanning, or brute force dumps. and you do get something tangible from those -- the output atlas script will have #WRITEs for each pointer it found. you can modify those at will to remove false positives and perhaps add in pointers that the routine might have missed. (though it really shouldn't be missing any, unless they fall outside the pointer search area you specify.)

That's not tangible. That's just a guess from an algorithm. The very fact that you have manually go back through them attests to that. If you have hard coded pointers in the assembly code, you can locate them in seconds with a debugger and add to a pointer list. Or if you know the instructions, you can search via instruction pattern. I will support pointer lists. But this is a dumper/inserter, not a scanning or pattern utility for possible pointers. I have no interest in these non tangible methods. I will have only features that result in correct/verifiable/reliable output.

assuming that screen is showing functionality similar to what is currently in cartographer, it will go through a pointer table and dump the pointers and the text that the pointers point to. that is not the same as what i am saying. what i am saying is for it to go through the text area and find the pointers that point to the text, and dump those pointers and the text.

The results are unreliable guessing based on your searching algorithm.

? what is a "tangible" scanning algorithm? here is the sort of algorithm i am proposing and have implemented in my (crappy) tools, and i know others have as well:

1. aha i've found the start of a string.
2. let me make a note of what offset this string starts at.
3. let me search a specified area for pointers that point to this offset.
4. ok, i found it. let me dump that pointer along with the string.
5. ok, next string. goto 2.
6. repeat until end of text block is reached.

if you're confused about how one might search for pointers... for example, the GBA addresses the ROM starting at 0x08000000. so, if my string is at 0x00001000 in the ROM, i can add 0x08000000 to that to get 0x08001000, and search for this value. so i'd start from the beginning of the specified "pointer search area", and look for a group of 32bit-word-aligned bytes that look like 00 10 00 08. if i find that, i have potentially found a pointer to the text at 0x00001000. obviously that particular set of bytes could very easily be a false positive, but something like 56 32 1C 08 is much less likely to be a false positive. and of course you can double check it yourself, manually.

There are no tangible scanning algorithms. That was my point. Anything you use will result is something with false positives or misses in the methods you describe. This is not within the scope of the utility I am creating. It fails at step one. In a raw blind dump, you don't know where the start of a string is. You're guessing based on where an end token is or other unreliable means.

Sorry, you'll have to use pointer lists for these or stick to your own tools that do this job so well for you.

[Klarth]

Some thoughts on the table standard:

• I agree with the deprecated table entry types such as line break, bookmarks, etc
• I like the notion of having a table entry that switches tables automatically, but I don't yet understand how many cases your implementation will cover.
• I don't believe that kanji arrays have a place in a table standard.  I only know of 2-3 games that use this.
• There are a few spots in the doc where you use /r and /n instead of \r and \n.

Other thoughts (not fully related):

I implemented UTF-8 support in Atlas a few days back, so that should be out of the way.  It's not true UTF-8 support, just that the commands, comments, etc have to be ASCII...the actual text portion of Atlas files is considered encoding-neutral because it only needs to be mapped to another binary value, not displayed.

Interest pretty much died in the spreadsheet project, so I put it on the backburner until someone comes up with an XML standard or until early next year.

On the dumper:

It'll be interesting to see what your approach amounts to.  I'm positive it'll be more user-friendly than Atlas (and probably the rest of the inserters/dumpers out there), which is a great thing for people starting out or dealing with simpler games...most people like to play with a gadget out of a box before reading an instruction manual.  I do believe that the approach is going to disappoint when it comes to games more complex than "Here's a relatively neat pointer table and/or here's a nice, neat bank of text.".  But maybe it's been because I've seen too many of Gideon's scripts and too many PSX scripts and know they require game-specific context, which admittedly, Atlas relies on the dumper (or user) for solving.

[DaMarsMan]

Along these lines, I've always thought about a 'rule' based scanner that can find proper pointers based on the rules you feed it about the game's scripting blocks. It's never 'random' spacing between pointers.

yes, it can be "random". especially if the pointers are simply embedded in the code in between functions, or if they just dumped a bunch of data somewhere that happens to include pointers as well as other stuff. that's why you need pointer searching/scanning, or brute force dumps. and you do get something tangible from those -- the output atlas script will have #WRITEs for each pointer it found. you can modify those at will to remove false positives and perhaps add in pointers that the routine might have missed. (though it really shouldn't be missing any, unless they fall outside the pointer search area you specify.)

How does that differ from what you can accomplish with the pointer table range here?

assuming that screen is showing functionality similar to what is currently in cartographer, it will go through a pointer table and dump the pointers and the text that the pointers point to. that is not the same as what i am saying. what i am saying is for it to go through the text area and find the pointers that point to the text, and dump those pointers and the text.

I guess I'm not getting why you keep saying you're in RAW mode, but you're finding pointers somehow. If you're RAW mode, the only thing you can derive is knowing where an string end token is and assuming the next string starts right after that. Even then, you've collected all the information. What do you do with it then to 'find' pointers? I think this circles back to non tangible scanning algorithms.

? what is a "tangible" scanning algorithm? here is the sort of algorithm i am proposing and have implemented in my (crappy) tools, and i know others have as well:

1. aha i've found the start of a string.
2. let me make a note of what offset this string starts at.
3. let me search a specified area for pointers that point to this offset.
4. ok, i found it. let me dump that pointer along with the string.
5. ok, next string. goto 2.
6. repeat until end of text block is reached.

if you're confused about how one might search for pointers... for example, the GBA addresses the ROM starting at 0x08000000. so, if my string is at 0x00001000 in the ROM, i can add 0x08000000 to that to get 0x08001000, and search for this value. so i'd start from the beginning of the specified "pointer search area", and look for a group of 32bit-word-aligned bytes that look like 00 10 00 08. if i find that, i have potentially found a pointer to the text at 0x00001000. obviously that particular set of bytes could very easily be a false positive, but something like 56 32 1C 08 is much less likely to be a false positive. and of course you can double check it yourself, manually.

I've also had this problem where the text chunk is in order but the pointers are out of order. Dumping from the table gives a very confusing script sometimes. Not to mention that you can have duplicate text chunks. It would be a great option to have.