News: 11 March 2016 - Forum Rules
Current Moderators - DarkSol, KingMike, MathOnNapkins, Azkadellia, Danke

Author Topic: [Utility] Visual SAK - Visual Swiss Army Knife  (Read 7597 times)

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
[Utility] Visual SAK - Visual Swiss Army Knife
« on: January 28, 2018, 09:04:23 pm »
Visual SAK: Visual Swiss Army Knife

A new tool developed that will be more like the real Swiss Army Knife - a toolbox in one package, easy to use, does few things but do them with precision!
The toolbox will be focused on ripping of visuals from different ROM - Sprites, Graphics, Palettes, Tiles, ... but not casual ripping rather - finding, documenting, exporting and maybe in the feature importing.

Main Features:
- Mapping of graphics
- Naming of graphic objects
- Saving all findings as a document/project for easy reuse
- Multiple BPP format
- Multiple compression formats
- Exporting of graphic objects in PNG
- easy to manipulate attributes to find out desired graphic (Width, Height, Offset)
- customizable palette for displaying a graphic with the correct colors

Visual Swiss Army Knife (Visual SAK) - proof of concept:




Download:
 RHND submission: 3.1
 Latest exe (Version 4.0)

Versions:
 1.0 - First release. Manual address adding for tiles and palette. Save/Load project. Template system for interpreting the images. Currently only GBA 4BPP LZ10 is supported.
 2.0 - Template system improved. Added a new Template: GBA 4bpp (no compression). Some visual changes and many, many internals rewritten to take advantage in the new Template system
 2.1 - unreleased version. Design change - added status bar. Non-edit info moved there
 2.2 - Added 2 new Templates: SNES 3/4bpp (no compression). Major workflow changes thanks to weissvulf

 3.0 - Many changes. "Double buffer" engine. Export of the graphics in PNG. Removing of Template system. New 2bpp SNES. New 8bpp PC. New compression format: LZSS-F5/LZSS-F6 (used in SNES versions of FF5/FF6). Longer names (20->30 character).
 3.1 - New 8bpp SNES Mode 7(2). New 8bpp SNES Mode 3. New compression format: LZSS-CT (used in SNES version of Chrono Trigger). New feature: importing a palette from ZSENS save-state files!
 3.2 - New 4bpp SNES FX. New features: import/export to Photoshop aco/act format. Export to ZSNES save-state. New feature: contextual menu and buttons for the graphics list. New operations over the list - sorting, moving. New feature: speedy add of a new address, right after selected one.

 4.0 - Again huge update. New shiny GUI. New Templates - 2bpp for NES/NGP, 4bpp for PC, 15bpp (BGR 555 - SNES/GBA/PSX palette). New scle-able file format for your work. Import of graphics. Save of a changed ROM

ToDo List:
 - adding Search ability. Initially for LZ10 compression.
 - reading compressed Palettes. Initially LZ10 only
 - support for other compression (on request)
 - support of other console format (on request)
« Last Edit: August 03, 2018, 10:36:44 am by Squall_FF8 »

weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: Sprite ripping tool
« Reply #1 on: January 28, 2018, 11:37:17 pm »
I think you should decide the scope for your tool before you begin. I find that tools which try to support too many formats end up with cluttered controls and an inconvenient workflow.

One possible workaround is to have a 'mode selector' where you selected the platform you are working with, and all unrealted controls would be filtered out. For example, if you select GBA, all settings/controls SNES features would be hidden.

Other than that, some general features I like in tools:
 Minimal confirmation message boxes. For example, rather than "File exists do you want to overwrite?", just auto save the original old to a 'backups' folder.

 Don't clutter up the control area with multiple controls that do the same thing. For example, a "save" button and also a "File > Save" menu.

 Whenever pertinent, save settings to a file that can be easily shared with others. For example, if you're ripping graphics from a particular game, let the user save the location of all the graphic to a project file. They could then share the project file so that others could easily rip the same graphics.

 I wouldn't bother adding any editing features since most people use their own favorite photo editor.

 Inserting as well as extracting graphics would be nice.
« Last Edit: January 29, 2018, 12:00:11 am by weissvulf »

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Sprite ripping tool
« Reply #2 on: January 29, 2018, 03:58:48 am »
Hehe weissvulf, its like you are reading my mind, but express the thoughts with different words  :thumbsup:
- minimal confirmation
- (not) using of menus
- freedom to save by the user
- NO editing!
- supporting too many options/formats may lead to reduced effectiveness/inconvenient workflow.

I would love to do inserting. There are some technical problems with that - since it need to be compressed the size will be different. When the new size is smaller its ok, but if it's larger then its a problem.

BTW maybe 'Sprite ripping' is not accurate expression of what I mean. Well definitely the ripping part is involved as possibility, but also a tool that easy find visuals, accumulate a knowledge of how it was done for the purpose of hack/mod or even making a ROM-Map of Visuals ...
For example currently I need to find the job sprites of FF5a. I'm not interested in actual ripping, rather finding them in the rom, understanding how they are stored internally, finding the master table that contain all job sprite indexes/pointers,...

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Sprite ripping tool - Visual Swiss Army Knife (Visual SAK)
« Reply #3 on: February 23, 2018, 08:09:03 am »
Finally version 1.0 is ready! Here is an extended list of implemented features:
 - New Project
 - Load/Save of current project
 - Template system for recognizing/interpreting an image
 - Currently only GBA 4BPP LZ10 format is supported!
 - Manually adding address for Tiles & Palette
 - Support of decimal and hexadecimal format of the address
 - Loading a Palette from the ROM
 - Some basic operation with palette - Grayscale 16, Grayscale 256
 - Zoom of the drawing
 - Transparent drawing

* Download from main post.
* Check the Offset property. It mean that in decompressed stream visualization will start after skipping few bytes! This is an unique feature that non of existing tools have! Such option is crucial, because most of the Final Fantasy titles have header after decompression that needs to be skipped!

Please download and check this tool and share any feedback that you have! Lets together make a tool that is useful as famous swiss army knives! The tool doesn't require extra layer (dll, .net, DirectX,..), so its just an .exe file.

weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: Visual Swiss Army Knife (Visual SAK)
« Reply #4 on: February 23, 2018, 06:10:45 pm »
I thought of a feature that might be handy. If this tool is designed to map the graphics in a game and save the map to share with other modders, an option to assign graphics to a sub-group which the user could 'show' or 'hide' would be nice. For games that have hundreds or thousands of sprites, being able to thin the pool down categorically could be handy.

I tried the tool and I'm not really sure how to use it. The interface seems self explanatory, but I don't get the expected results. Maybe it would be good to include a basic VSK file to get people started?

1) I get a 'S:\...FFV advance.gba not found' message at start up.

2) I loaded a FFV USA.gba file, but I'm not sure where graphics are in the ROM for testing. I enter the address shown in the tool's screen-shot and get an 'Access violation: write of address' message. I get a similar message when I adjust zoom, width, height, offset, or name.

3) There seems to be an anchoring issue for the controls on my PC: the 'load Palette, add/remove address' buttons are anchored in place, but the address inputs and palette view are anchored right. Also, if I resize the window enough to remove the lower scroll-bar, the palette-view is cut off. I think I have my PC's view-settings tweaked, so that may be causing the issues.     
Spoiler:
« Last Edit: February 23, 2018, 06:22:04 pm by weissvulf »

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Visual Swiss Army Knife (Visual SAK)
« Reply #5 on: February 25, 2018, 06:59:24 pm »
Version 2.0 is out! Here is an extended list of implemented features:
 - Template system improved. Now all the code is based on a template, rather couple of individual characteristics
 - Added possibility to change a Template at any time
 - Added a new Template - GBA 4bpp (no compression)
 - added some restrictions in interactions, to avoid some 'Access violation '
 - small bug-fixes
* Download - from main post!

I thought of a feature that might be handy. If this tool is designed to map the graphics in a game and save the map to share with other modders, an option to assign graphics to a sub-group which the user could 'show' or 'hide' would be nice. For games that have hundreds or thousands of sprites, being able to thin the pool down categorically could be handy.
'map the graphics' - very well said! You really have this ability to express what I'm trying to say just in a better way  :beer:
I was think the same - when we have to many entries it will be not so easy to navigate/search. So that is an excellent idea to organize them in groups! Thank you! :thumbsup:

Thank you for the time to report the problems. Hopefully they were all fixed now in this version. I've put everything in an appropriate container, added some min/max info so hopefully no problem with anchoring/resizing. In the rush to use better buttons I didn't see that these one have no Anchor property ... The first problem come from left-over code while testing  :-[
I'm a little worried about the second reported bug ... please tell me if you have it with the version!


weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #6 on: February 26, 2018, 05:47:51 pm »
It working, nice! Here is some test feedback.
Spoiler:

Access violations:
Enter name before selecting Template
Load new ROM with old values still in place (memory reads greater than new file size)

Crash:
Increasing width or height excessively. Maybe happens when input dimensions exceed open-file's size.
Increasing PalNum beyond allowable range

Controls:
Displayed palette colors go up with PalNum+load palette, but not down.
Loading new ROM does not clear old image.
"Add Address to list" blanks all settings; better to keep them.
No way to edit "Tile Address" for existing List-Entry?
Displayed image disappears if it goes behind another window, or off screen.
Image display starts with left and top cut off behind controls.
White border at palette right and bottom.

Other:
Maybe set starting Tile/Palette Address to 0, width/height to common size.
Maybe change PalNum to 'Colors'
Think 'make palette' buttons better docked next to 'load palette'

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #7 on: February 27, 2018, 02:07:54 pm »
Version 2.2 is out!
 - Added Template: SNES 3bpp (no compression)
 - Added Template: SNES 4bpp (no compression)
 - Major workflow change (thanks to the great suggestions from weissvulf
 - Stability improvement (thanks to the extended testing and report of weissvulf

weissvulf, Thank you for taking your time to use, test and report your observations. I personally tried all your suggestions and they are rock solid, based on practical application!!! So here is what I've done.

Fixed:
 - Enter name before selecting Template (access violation)
 - Displayed palette colors go up with PalNum+load palette, but not down.
 - "Add Address to list" blanks all settings; better to keep them.
 - Maybe set starting Tile/Palette Address to 0, width/height to common size.
 - Maybe change PalNum to 'Colors'
 - Think 'make palette' buttons better docked next to 'load palette'

Unconfirmed (I tried on 3 different PC with different Windows):
 - Image display starts with left and top cut off behind controls. (Do you use a huge window border? like 5+ pixels?)
 - White border at palette right and bottom.
 - Increasing PalNum beyond allowable range(access violation). The range is 1-256, how much did you enter?

Need some talk:
*Load new ROM with old values still in place (memory reads greater than new file size).
Do you think I should make a 'New Document' when new ROM is loaded? In general that is a good idea, but there are some practical implications of not doing so. Like if you map one version, then you open another version and you'll find out whether the map is relevant(Eu <-> Us <-> Jap)

*No way to edit "Tile Address" for existing List-Entry?
For now Tile Address is my main attribute ... think of it as record ID. Changing it is like having a new entry. If the change is big, maybe its better to really make a new entry? Especially now with your suggestion of 'keeping last used values' it will be as easy as changing the address. If the change is small, you can play with the Offset. This attribute is unique for this tool, so maybe that's why it is neglected. Keep in mind that it allows positive and negative values.

P.S. I need help to test 8bpp GBA graphics. If you know the addresses of graphic tiles of 8bpp in any GBA ROM (non-compressed or LZ77-10) please share some and the name of the ROM. If you happen to know their palette address that will be even better :thumbsup:
Do you know games for GBA that use direct-color (non palette)?
Do you know GBA games that internally store graphics in bpp different of 4/8?

weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #8 on: February 28, 2018, 01:25:29 am »
It's working nicely!

As far as "display cut off behind controls." and "White border palette", I did some testing and it's caused by Window's 'icon size' setting under 'Personalize > Display". It resizes SAK's frame, but not the image/palette display. I wouldn't worry about it, if there is no easy fix. Maybe just add a note in the readme.
Spoiler:

Quote
Increasing PalNum beyond allowable range(access violation). The range is 1-256, how much did you enter?
Yes, higher than 256. I was emulating 'dumb guy' who mashes buttons till something glitches, grunts "This program stinks", and writes a nasty message to the developer. :P The 'up button' is properly capped at 256 now, but I can still type large numbers which cause a crash on 'Load Palette".

Quote
Do you think I should make a 'New Document' when new ROM is loaded?
No, I would not do that, for the reason you mention. I tried the same thing in the latest version (expanding the expected dimensions beyond actual file size),  and it no longer crashes- just an error message. That is fine in my opinion. Users have to exercise some common sense!

Quote
*No way to edit "Tile Address" for existing List-Entry?
Like you say, preserving the old settings when making a new entry is enough in my opinion.

I have never dug into GBA images so I can't help with finding samples. Hopefully someone will speak up.  :thumbsup:

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #9 on: February 28, 2018, 04:33:37 am »
It's good to know that 'Scaling' cause these artifacts, because at first I thought its because of different Windows version! I always assume that this is done by scaling the DC of controls, but it seems now that it only scale the size of a control and the Paint process is responsible to take that into an account  :(

Quote
Yes, higher than 256. ... The 'up button' is properly capped at 256 now, but I can still type large numbers which cause a crash on 'Load Palette".
Yeah that SpinEdit control causes some troubles with typing ... In feature I may tray to get a better one or just fix the code of it  ::)

weissvulf I'm counting you will continue to monitor the progress and play the 'dumb guy' who mashes buttons till something glitches, because that is extremely, extremely helpful to me!  :beer:
BTW what platform/game is your major knowledge on? What do you dig atm?

I'm also interested in implementing other (non SNES/GBA) formats of graphics, probably Sega, Sony consoles ones as far as I have enough info for internals (compression methods, how a tile is positioned in the memory).
But that will be after I implement all main features. For now 8bpp/Direct Color for GBA/SNES is something that I need examples (doesn't matter in what game)  8)

weissvulf

  • Sr. Member
  • ****
  • Posts: 324
  • Good news! An anomaly solved the enigma.
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #10 on: February 28, 2018, 10:11:29 pm »
Quote
BTW what platform/game is your major knowledge on? What do you dig atm?
I've done most of my work on the PC and PS1. I'm not infatuated with those systems; it's just that people haven't done much with the Sony systems compared to mow many games were released for them, so there are a massive number of great PS1 titles ripe for the picking.

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #11 on: March 27, 2018, 04:54:49 am »
Version 3.0 is out!!!
 - Internal engine rewritten! That change took me like 2-3 weeks, but is worth the time. Before I was using kind of Virtual Tile Engine. Now I use a 'double buffer', similar to what most of contemporary games do. Now drawing is FAST, no tear, no artifacts. That also allows me to easy export GFX
 - New feature: Export of Gfx (PNG only for now, other formats on demand)
 - Removed feature: Templates from 2.2. Now BPP is separated from compression (as intended in version 1). The main reason for this is that the list of Templates grows exponentially with increasing the supported BPP/compression formats.
 - Added new BPP: 2BPP (SNES)
 - Added new BPP: 8bpp (PC). PC yes, I was surprised that a SNES game would use it, but there are some like Wolfenstein 3D
 - Added new compression: LZSS-FF5, a modification of LZSS used in FF5 SNES
 - Added new compression: LZSS-FF6, a modification of LZSS used in FF6 SNES
 - Increased: The length of Names from 20 to 30 characters. Now you can be more descriptive  :laugh:

*Download from the first post.

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #12 on: April 18, 2018, 12:40:05 pm »
Version 3.1 is out!!!
 - New feature: Import of a palette from zSnes savestate (.zst, .zs?).
 - New BPP: 8BPP (SNES Mode 7). No planes. In mode 7 a tile takes 128 bites - because in the memory tiles are interwoven with the tile map
 - New BPP: 8bpp (SNES Mode 7 - II) same as above except it contain only the tiles (64 bytes each)
 - New BPP: 8bpp (SNES Mode 3) - each tile is 64 bytes, but each pixels is divided in 8 planes according to the SNES model of planes
 - New Compression: LZSS-CT, a modification of LZSS used in Chrono Trigger SNES

*Download from the first post.

Special thanks to Mauron for providing info for CT compression :thumbsup:

« Last Edit: April 19, 2018, 03:08:32 am by Squall_FF8 »

spiiin

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #13 on: April 21, 2018, 07:32:24 am »
What programming language did you use?
Why didn't you post sources? It may be interesting to have examples of building snes graphics to me.

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #14 on: April 21, 2018, 08:45:23 am »
What programming language did you use?
I'm using Delphi with Pascal as language for programming.
The application is compiled with old version Delphi 7 to produce much smaller code.
You don't need external DLLs, frameworks or whatever, just Windows  :laugh:

Why didn't you post sources? It may be interesting to have examples of building snes graphics to me.
I will think about it. It mainly depend on the reception ... if it will be received well and community around helps (ideas, test, code fragments/algorithms, icons. design, ...) my biggest thanks will be releasing the source as public. :thumbsup:

As for SNES I would love to help you. If you have specific questions/interests feel free to post here  :beer:

Also I would love to hear you first impressions with the tool, could you use it with 0 help/readme? (I tried to make it intuitive.) Could you think of new cool features?

spiiin

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #15 on: April 21, 2018, 10:46:32 am »
It crashes on shutdown sometimes with EInvalidPointer exception.
I understand, for the program should do, but can't do it on real ROMS, because of no examples and I don't know much any SNES/GBA addresses for any game :) I can try to see it from screenshots, but why no put some example projects with the program?

I like Delphi 7 and use it a long time ago, but I think, that's there is no pascal community now. It's nice for single programmers to use simple instruments like Delphi/visual basic, but there is no chance to get a power of the synergic use of several programs together. At least, you can add an option to export palettes/tile maps to uncompressed files (if your tool hasn't it, I can't check all features).


 

Fire-WSP

  • Full Member
  • ***
  • Posts: 120
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #16 on: April 21, 2018, 06:29:45 pm »
Awesome, will check the new version very soon.

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #17 on: April 21, 2018, 07:37:47 pm »
Quote
It crashes on shutdown sometimes with EInvalidPointer exception.
Yes ... I gave a freedom to decompress even if there is no real compression and that can create serious memory problems :(

Quote
I like Delphi 7 and use it a long time ago, but I think, that's there is no pascal community now. It's nice for single programmers to use simple instruments like Delphi/visual basic, but there is no chance to get a power of the synergic use of several programs together.
Hehe if you have really used Delphi 7 you would know that is all but 'simple instrument'. It is called a RAD (Rapid Application Development) not by chance ;)

Quote
At least, you can add an option to export palettes/tile maps to uncompressed files (if your tool hasn't it, I can't check all features).
Currently you can export the graphics in PNG format.
I will focus more on Palettes operations in later versions. There will be export to Photoshop aco, Windows RIFF pal. I would like to have export to some tools that are made by the community too.

P.S. spiiin really if have specific questions about SNES and maybe some games (Final Fantasy 5), don't hesitate to ask!

spiiin

  • Jr. Member
  • **
  • Posts: 43
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #18 on: April 22, 2018, 07:13:54 am »
Quote
Hehe if you have really used Delphi 7 you would know that is all but 'simple instrument'. It is called a RAD (Rapid Application Development) not by chance ;)
I mean "simple" not as "primitive", but "comfortable for programmer".
My criteria for choosing language for creating tools:

Simple for programmer (imho)
 + Delphi, vb, C#, python
 - C++, Java

Simple for the user (no need to install some dlls or runtime libraries):
 + Delphi, C++, C# (.net framework preinstalled on modern windows)
 +/- Python (can be compiled with one Exe with some magic and big size of app)
 - Java

Easy to implementation extension system
 + Python (as every interpreted languages)
 +/- C (with embedded interpreter of python or Lua), C# (I just use CSScript for using dinamic compiling C# as scripts)
 -/+ Delphi (with DLL plugins, for example, and any other languages)

Language is "alive" - bugfixes, updating 3rd party libraries for different tasks, possible cross-platform, chances to make team with other programmers, the advantage to training programmer skills for professional coding, etc.
 + Python, C#, C++
 - Delphi

Also, one of authors of Delphi is author of C# too.

Sorry for offtop and for mistakes, my English is poor.

Quote
P.S. spiiin really if have specific questions about SNES and maybe some games (Final Fantasy 5), don't hesitate to ask!
Ok, when I start to implement plugin for SNES for my editor, I pm to you, thanks.

Squall_FF8

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: [New Tool] Visual SAK - Visual Swiss Army Knife
« Reply #19 on: April 22, 2018, 11:05:43 am »
I mean "simple" not as "primitive", but "comfortable for programmer".
I see :laugh:

Also, one of authors of Delphi is author of C# too.
Yeah. He was part of Borland team that is responsible for implementing components, property system of components, all the things that Microsoft market as 'Visual'. Later on he changed the company and made the same with C, implemented in C#. That is the reason it so easy for me to read C# code, although I never studying at all :)
Some times I use the Microsoft trick: when people ask me what is Delphi, I answer "Visual Pascal" and they node with understanding. :laugh: