Thanks for the every so often "wow you are old" for the "old arcade game... runs on linux".
Game names do help if one of us is to have a look. Can do the general case though.
Anyway custom video formats are seldom fun to figure out. If it is simple and you have some indication of what goes then you have a chance short of hoping for a leak of source or spending a dozen hours with a disassembler.
Between rounds is a slightly odd one, most games we see with a quick turnaround for matches will be in engine if they have anything at all, but if it is was showing off in an arcade then OK.
In the most basic format they are one picture after another, possibly compressed on a per frame basis with whatever compression format is viable for the machines of the day (mjpeg, that being motion jpeg, being one of the most well known examples of this concept). Wind in some audio, data on the stream (though for a game it might be hardcoded into the decoder. Still resolution, colour depth/profile, framerate, possibly length and a few other choice bits if info tend to want to be known to do the video thing) and you are there. Interlacing support can also mix things up and if it is old arcade designed for CRT then it is a possibility.
You also have the separate video and audio dilemma. Separate makes fiddling easy but also means a lot of random reads which is not great for discs and not all that much better on carts. Interleaved (guess what the I in AVI stands for?) then being the frame count and audio sample count then match up. Even to this day games are no stranger to having their audio be separate in the normal format, or maybe another format entirely, while a silent video plays.
If it is simple pictures then you may or may not have a group of pictures (usually seen as GOP in encoders) such that things can grab a set number of frames + audio, decode it all and then if it messes up there is a kind of inbuilt reset -- while not quite the same thing if you have ever had a video corrupt, and then come back in with motion before all of a sudden becoming just fine then you have witnessed this.
Compression sizes with this approach are fractional though and as one frame is similar to the next for most video then you just figured out motion interpolation and estimation. This usually sees video have one frame help determine the next (p-frame and b-frame if you want the usual terms). If it is this then yeah.
Anyway from what you said.
TGA is a seldom seen image format these days but is a fairly well known one which is nice.
Check to see if you can find any indication of the format (magic stamps, resolution and whatever matching the machine it plays on...)http://paulbourke.net/dataformats/tga/
Sometimes these formats will strip any header data (they are all the same known size and format after all) but it would be where I start.
RLE is part of the TGA spec so doing it twice (once in the file itself and once for final) is unlikely to yield many gains and might make things unwieldy if you have to decode several seconds of footage at once before you can feed it to the playback options. If he had to implement it as part of it though then it might have caused a headache that he remembers and passed along.
Whether each individual image file will be end to end or pad it out I don't know; there are benefits to each approach.
Comparing two files does occasionally yield some good info -- what differs and what stays the same could well be a key to something. Comparing between two different videos could be another -- if the title screen/normal videos are larger in resolution than the small animations then yeah.
Anyway once you figured all this out comes time to play encoder. If it is just a sequence of TGA in a nice archive, maybe with some audio thrown in, then good for you as you don't have to write a nice custom encoder ( http://andrewduncan.net/mpeg/mpeg-1.html
is just MPEG1, the impossibly ancient format that you possibly have not seen in a decade at this point). Instead get a video editor that will spit out a sequence of images (virtualdub should have the option but there are plenty of other choices here), as the video editor is unlikely to do much other than BMP then batch convert these to a suitable version of TGA (I like irfanview here, though I must confess I have never used its TGA abilities), somewhere in the previous two steps then you want to have made it a suitable resolution, and now you get to pack these into a file* and make up a suitable header/footer, possibly along with any audio.
*this you can do with a batch file. copy /b on Windows or cat on Linux. If I need to trim a fixed (or easy to generate list) of things from a file then I quite like filecutter ( https://web.archive.org/web/20170218180937/http://min.midco.net/cracker/filecutter.zip
). Padding should also be doable with this (make a larger than the whole frame/section padded file, copy /b it after each image and trim to size accordingly).