ROM checksums
Sound Rate data (Can cause desyncs on some games)
Up+Down/Left+Right checked bit
SRM information(/16k BRM for when SCD is made TASable)
Perfect Syncro checked bit (For when SCD is made TASable)
Am I missing anything?
How fleeting are all human passions compared with the massive continuity of ducks.
maybe record the console type, in case of a (very hypothetic now) sms/gg support ? or should it be recorded in another container, with another extension ? or just another emulator -__-
I never sleep, 'cause sleep is the cousin of death - NAS
maybe record the console type, in case of a (very hypothetic now) sms/gg support ? or should it be recorded in another container, with another extension ? or just another emulator -__-
SMS/GG TASing will be done on a different emulator. Meka is the only emulator which doesn't freeze on a number of games, and the author is willing to help improve it for TASing purposes.
How fleeting are all human passions compared with the massive continuity of ducks.
The initial GMV format have many flaws. The main flaws were :
-Lack of space for extra features
-Hard-coded limit on movie data (2 6 buttonx or 3 3 buttons)
-Lenght of movie based on lenght of file
By the developpmenet of Gens 9, I considered switching the format (I called it GM2). I was thinking about a modular sturcture to preserve backward compatibility (not with original GMV but with all version of GM2) without sacrificating expandibility.
[FILE HEADER]
-File identifier (similar to SMV, 4 bytes to itendity the file as as Gens Movie)
-File Checksum
-Module count (4 byte integer)
Then, for each module :
-Module type id (4 byte, like a string of 4 caracters)
-Module index in file (Indicate where in the file the module starts) 4 byte int
[MODULE HEADER]
(let me explain the idea of module : every elements in the movie file is encapsuled in a module. The movie data is a module, the autor info is a module, savestate information is a module, etc.)
-Module header size (4 byte int)
-Module data size (4 byte int)
-Module data checksum
-Module minimum requirements size (4 byte int)
-Module minimum requirements (4 byte identifier, one byte critera and 4 byte qualificator)
(let me explain the idea of minimum requirements : it tells the emulator if it can process the module without problem. For exemple, if a movie module require Gens 212 and at least movie TEST 3, identifier 1 could be "GENS", qualificator 1 could be 212, critera 1 wocould be '=',identifier 2 could be "MOVT", qualificator 2 could be 3, ant critera 2 could be ']' . This way, the emulator will know that to play the movie properly Gens 2.12 is required and at least movie test 3 is required. Modules like Movie Author info won't have any minimum requirements. If different Gens mod authors want to add their own module, they would include a minimum requirement for their mod. Example : I create a mod of Gens with Audio commentars (let's name it Gens LOUD). I create the module "SNDC" with requirement "LOUD" ']' 1. If I play it on another version of Gens, this module will be ignored. If another author implements sound commentarys but in MP3, take unintentionaly the same module name "SNDC" but with minimum requirement "GMP3" ']' 1, the Gens LOUD version won't be confused.)
Criteras :
'=' is equal
'-' is not equal
'>' is greater than
'<' is lesser than
']' is greater or equal
'[' is lesser or equal
'o' is optional
[MODULE DATA]
Raw module data
Exemples of modules :
"AUTH" : Author info, include author name and author comment
"ROM " : Include rom name, rom checksum, and country
"CONT" : Include controller information
"SYST" : Include video frequency
"SCFG" : Include sound configuration
"SUBT" : Subtitles : Frame, string,delay,color,etc...
"SVST" : Savestates with description
"SRAM" : Sram with description
"MOVI" : Movie data, up to 8 players, with description and rerecord count
"ZMOV" : Gziped movie data
"MVBK" : Backup of the movie, In case you accidently destroyed your current file with a misplaced F5
You can have multiple modules of the same kind in a single file, meaning you can have a movie file with multiple savestates at differents points of the movies, or different versions of the movie. Want to add a custom soundtrack with audio commentary ? Want to have some parts of the movie played by default in slow motion ? Want to include a module with a webcam filming yourself playing the game ? The possibilities are infinite. While still maintaining backward compatibility, expandibility and conflict resolution. Tools would allow conversion between GMV and to a Hex Editing friendly version
But that is going to be an hard job to implement, and I never found the motivation to do it.
[thinking about it, a format like that could be used universally with multiple emulators, so once an author code the main library to manage this file format, only minor modifiactions would be required to adapt to different emulators]
This shouldn't go in the format, because it only affects recording (not playback).
upthorn wrote:
Sound Rate data (Can cause desyncs on some games)
Ideally, Gens would be made to not emulate differently with different sound rates set, but it might be too late for that to be worth correcting. As things are, people should only be recording movies at 44100 rate, otherwise the sound information of that higher frequency is lost.
upthorn wrote:
Am I missing anything?
- Console type (Sega Genesis vs. Sega CD vs. 32x), because it might be useful for something to be able to extract that from the movie file
- Internal ROM name (so you have something readable to show about the ROM in the play/record dialog besides a checksum)
- Maybe something about which BIOS file was used, if not the built-in one
- It shouldn't affect the synchronization, but if it does, maybe the Sprite Limit option
- Embedded savestate file for the "Record from now" option
Also, SRM data shouldn't be included unless some "Record from SRAM" option is checked, otherwise the emulator should simply clear it on starting the movie.
About what Jyzero said: I thought we weren't aiming for such a big overhaul of the format here, but this module idea sounds good. It's similar to what FCEU does for its savestate format.
(If we are changing the format in a non-backwards-compatible way, we might as well also invert all of the keypress bits, because it's sort of confusing that 0 means pressed instead of unpressed.)
Also, "Length" is not spelled "Lenght", contrary to what the Gens source code would lead one to believe.
In the interests of lazieness, I'd like to point out that this will be a lot of work to replace something that's already good enough. The current GMV format won't win any awards, but it won't end up on The Daily WTF either. Perhaps the best way to overhaul this format is to consider everything from a user's perspective. A nice extensible module-based format may be the way to go, but we should think about how it will make recording and playing movies easier.
I'd like to see a new format and I can think of a few ways it could make life easier for everyone involved, but it takes a lot of energy to get started actually coding. There's no sense producing an unnecessarily complex specification that none of us will bother to implement.
An on board hex friendly editing tool system would be very useful. It sucks realising something you did way back could be improved, but you can't do it because it'll wipe out all the good stuff after that and have to go through the whole process again.
hex editing is hard, not because tools lack, but because random number generators make the run desynchronize after the copy/paste.
hence, a tool wouldn't help much, since rngs are inherent in games, not in tools/emulators... it would just be more practical :)
I never sleep, 'cause sleep is the cousin of death - NAS
What I want:
- SRAM and/or savestate included in moviefile
- ROM checksum and name
- All options possibly causing desynch saved
- Country setting (and that Gens defaults to this contry when playing back the movie)
- One byte for the number of players
- One byte per player for the type of controller (3-button, 6-button, etc...)
- Hex-edit friendly (i.e. definitely fixed number of bytes per frame, preferably fixed location where they appear)
- A lot of extra space in the header
I think you have most if not all of this covered already. If you want to do something a lot more fancy, that's fine with me, but like IdeaMagnate said fancy ideas have a smaller likelyhood of actually being implemented. As long as the stuff above is in, I'm happy.
Jyzero, I think your idea is has some potential if it's pared down a little. The first thing is that I don't see any reason for adding any metadata to modules other than size and checksum. My thinking is that requirements won't help Gens figure out what a module does if it doesn't know, and won't be necessary if it already knows. If a newer version of a module is likely (e.g. mouse support), the module can be renamed or can include version info in its data.
It seems like a good idea to include both per-file and per-module checksums. If the file checksum is good, we'd know that all the modules are good and could skip checking them. If it's not, we'd be able to figure out which modules were broken, either ignoring their data or failing gracefully.
Edit: In order to make the format both self-verifying and hexedit friendly, it will need to be possible to bypass checksumming and length checking and to store the frame data at the end, to be able to figure out when the data ends.
GMV isn't office 2007 so the the format specs I described are a little extreme for the envergure of Gens. Checksums could all be scraped since file integrity isn't an issue (except for ROM checksum). Version numbers for modules could also go away except for movie data. As you said, if Gens doesn't recognize a module name it will just ignore it. As for being Hex friendly, I think a little converter GMV->Raw could do the job.
okay, guys, I think we really ought to get back to this discussion.
Nitsuja, IdeaMagnate, and any other active Gens coders, we need to figure out what's going to change in the next version of GMV.
How fleeting are all human passions compared with the massive continuity of ducks.
Hexedit-friendly
The GMV format is relatively hexedit-friendly, although GM2 should store a button press as a 1-bit instead of a 0. However, the format may be hedexit-unfriendly as long as there is a portable editor or GM2<->text convertor available. Alternately, the ability to directly load and save an ascii verion of GM2 would also fulfil this requirement.
Compression
Button-press recordings compress very well, but GMV does not allow for compression. GM2 should allow movies to be optionally compressed for publication. This prevents the need for separate compression/decompression steps when submitting movies or viewing submitted movies.
These two goals conflict each other, to me. I am against compression. The reason is that it does not take long to compress a movie file before submitting it. It takes much longer to run a pre-compressed movie format through a converter a total of 2X number of hex-edits you want to try, which are usually more than one if you try any at all. Very annoying. (I'm looking at you, FCEU/Nesmock.) Otherwise this looks very nice.
<Swordless> Go hug a tree, you vegetarian (I bet you really are one)
The goals of space-efficiency and hexeditability don't have to conflict. They just can't be met in the same file. If Gens can natively save and load compressed, uncompressed and plaintext movies, I think everyone will be happy.
The text version of the format would take care of hexediting. All you'd need to do is set a "save as text" option in Gens and you'd be able to use Notepad for all your hexediting needs without any conversion overhead.
I made some comments on the proposed file format page. Respond to my stupidities there.
I'm also a bit suspicious about compression, but if there's a program which opens GM2 files for editing, I guess it's okay.
Good job on the wiki. Since I can't add my comments here there are :
-File header should contain something about the Gens version it was created.
-Config should contain info about wich sound chip is enabled. I don't know if it's still the case but some games had to diasble some sound chips to avoid desynch.
-Correction : 3 player input was avaible since Gens 9d, before Nitsuja.
-Would this format work for Master System too ?
I'm responding here because the forums are better at facilitating discussion.
Truncated wrote:
This (u/d l/r enable) is only important while recording, not while replaying. Gens should not filter the recorded button presses so it is not necessary to save this variable.
I thought it would be a convenience to movie makers. Gens would only filter actual button presses, so the setting would be preserved across recording sessions. As long as Gens spits out a message when settings between its config file and the movie file are different, I think this will be useful without causing extra problems.
Truncated wrote:
<reset chunk stuff>
Good idea.
Truncated wrote:
Isn't it confusing, if editing with a hex-editor and also while writing the code, that the first byte changes meaning depending on controller type? Second byte could be x,y,z,mode instead.
Yes, it is. That was a strange decision on my part and I like your idea better.
Truncated wrote:
Pick a locale-independent character encoding system for these. Unicode?
I should probably do that. UTF-16 it is!
Truncated wrote:
Number of states (in state chunk)? Addresses to them, or traverse the list to find them?
A simpler solution would be to have the frame stored with the other two values. That way, it would be easy for Gens to make an internal sorted linked list of states when the loading the file, and it wouldn't have to decompress anything until they were needed. Having a more complex data structure in the file isn't a good idea because we want to read/write the file as little as possible, and we won't know in advance which states a viewer will want.
Truncated wrote:
* Should (upper right corner of first letter) be upper left?
* Number of strings?
Yes. I thought about that before I wrote it, but clearly not enough.
I can't think of any case when the string count would be useful. Care to elaborate?
Truncated wrote:
Things i can't find:
* Start from SRAM/savestate/reset (and which chuch would this go in?)
I made a new chunk for it. It also includes the state/srm, if needed.
Thanks for taking the time to critique this.
Jyzero:
I added a version chunk. I don't know if it will be useful for anything other than information, but it's there. I only added a string because it will be difficult to compare version across forks.
I reworked the config chunk to contain the sound chip information. Thanks for pointing that out. I've run into games that are picky about this too.
I also fixed the 3-player info.
Whether this is used for the SMS depends on who implements SMS rerecording. It's there if they want it.
These two goals conflict each other, to me. I am against compression. The reason is that it does not take long to compress a movie file before submitting it. It takes much longer to run a pre-compressed movie format through a converter a total of 2X number of hex-edits you want to try, which are usually more than one if you try any at all. Very annoying. (I'm looking at you, FCEU/Nesmock.) Otherwise this looks very nice.
I have a solution for this which doesn't even require using a special command to save and load for hex-editing. Well, it does, but it doesn't require it everytime you want to hex edit.
We have an option to save in human-readable format, which sets some flags in the header, converts all the input data to a text format ala
Frame 1:
P1 S
Frame 2:
Frame 3:
P1 UDLRABCSXYZM,
P2 S.
And then saves the input data chunk to a separate file, named (gm2file).input.txt
It would then read the input from this file until the user chooses the option to "recompile" the movie data, at which point the input would be converted back into the compressed format.
How fleeting are all human passions compared with the massive continuity of ducks.
Here's another reason I wouldn't care to much about how big the movies are when they're uncompressed: At least in Windows, you can tell the operating system to compress any file (just set the file attribute to compressed). The file takes up as little disk space as if it were compressed, but hex editors and other programs see it as uncompressed. (I'm not too familiar with what method it's actually using to do this, so let me know if there's major disadvantage of using it.)
EDIT: I meant, you know, in those cases where the file might actually get large enough that you care about compressing it. Probably an even better (and more universal) reason than the one I posted above is that the vast majority of the time, the file won't get that large.
IdeaMagnate>I thought it would be a convenience to movie makers. Gens would only filter actual button presses, so the setting would be preserved across recording sessions. As long as Gens spits out a message when settings between its config file and the movie file are different, I think this will be useful without causing extra problems.
My concern was that Gens would behave like Snes9x, which plays back the same movie file differently depending on if the "Accept left+right" flag is set. (And that's stupid.) Including and using it in this way, I have no problem with it.
>That way, it would be easy for Gens to make an internal sorted linked list of states when the loading the file
I get it now. Smart. Strike all the "number of" fields for the chunks that don't need it.
nitsuja (on GM2 page)>What's the point of storing these "x enabled" options? Why should anyone ever record a movie that must be played back with these sound options or an entire sound CPU disabled?
I think it is because some games cannot sync properly when they are on. I remember that /*- made a movie where DAC had to be turned off, but I do not remember which game.
(I might very well be mistaken on this.)
>Can these "x improvement" options possibly cause desync? If not, they are a nuisance to store because viewers who prefer a certain option will need to change it whenever the recorder didn't happen to use it.
If they don't cause desynch, the viewer won't really need to change them for playback, right? If it does, it's good to know which settings were used.
It's outside the scope of the movie format, but Gens could perhaps (temporarily) set all these variables to movie settings on command in the play movie dialog.
-----
About compression... how about this method?
* Savestates/SRAM in movie file is always compressed.
* Any other data in movie file (including buttons pressed) is never compressed.
* Gens accepts reading and writing to movie files in .zip archives, like it does with ROMs.
Effects:
* Maintains user hexeditability (just leave your file unzipped while you record/hexedit it)
* Movie can be zipped if you want to save space
* No need for Bisqwit to change formats that he accepts for this site
* No extra step of decompressing when downloading others' files
Also, I'd like to point out that the loss of compressing a movie with compressed SRAM/savestate is really minimal (at most 5 bytes lost per ~65000 bytes).
Let's remember that not everyone uses windows. ;)
Somehow I missed that Gens can read zipped ROMs directly. I can't say that zip is my favorite compression method, but since the code is already there and since Bisqwit's submission system accepts zip files, it'd be best to eliminate all compression from the format and make Gens able to directly create zipped movies. This would solve a couple of issues.
@Nitsuja:
Peripherals? I can see how it already fits in, but it's worth explaining.
Feel free to fix this. My brainpower seems to be waning (and I still have to go to work).
The checksum should be optional, off when the movie frame data is uncompressed, otherwise it makes the format very hex-edit-unfriendly to need to manually calculate a checksum with every edit to avoid an error message.
If the text dump (which Gens could load/save directly) doesn't contain a checksum and is used as the only editable format, this won't be a problem. As it is now, even storing the length of a chunk makes manual hexediting trickier (though not unmanageably so). Assuming that Gens can natively read and write a nice text format, is there any reason the binary format would need to be hexedit-friendly?
Recording information...
I can't believe I forgot that.
Just for clarification, recorded frames is a rough indicator of how many frames the person recording the movie saw, vs. how many are in the actual movie?
I'm tempted to merge some of these small chunks into something like a "Movie Information" chunk just to keep the implementation's code size a little more reasonable.
I think this [sound rate] should always be 44100 for movies, eliminating the need to store its option here. 22050 can be used to produce a shorter movie in some cases, so leaving it in the format could be a liability, and 22050 sound quality can't be restored to 44100.
You're right. I was thinking that some current movies might need this setting, but we won't need to convert them if we keep the GMV code. If this is constant across all GM2 movies, there won't be a problem.
The same logic applies to the perfect synchro option.
sound options
I know some of these can cause desyncs. If you know for certain that some of the will never, feel free to take them out. I prefer to err on the side of caution since I don't fully understand how they all work.
Would not external CRC be more reliable?
If by "external" you mean "calculated," yes.
200-byte ROM filename (not including path) (null-terminated utf-8)
I'm hesitant to use the ROM filename for anything. It's usually reliable, but it seems more effective to rely on the ROM itself, which is harder to change. Storing it isn't a bad idea, since it provides a reasonably good idea of how the movie was made, but a name mismatch shouldn't generate any sort of warning, assuming that ROM checksums match.
Edit: I also fail to see the purpose of a unique identifier. What would this do that the movie CRC wouldn't?
Yeah, I know, I was just mentioning one method I knew of and I was pretty sure that other OS's can do something similar.
IdeaMagnate wrote:
Somehow I missed that Gens can read zipped ROMs directly. I can't say that zip is my favorite compression method, but since the code is already there and since Bisqwit's submission system accepts zip files, it'd be best to eliminate all compression from the format and make Gens able to directly create zipped movies. This would solve a couple of issues.
That would make things a lot simpler. And it wouldn't require going through Gens or some special converter in order to make a decompressed version of the movie.
IdeaMagnate wrote:
nitsuja wrote:
Peripherals? I can see how it already fits in, but it's worth explaining.
Feel free to fix this. My brainpower seems to be waning (and I still have to go to work).
I was thinking that, since you're reserving a whole byte for each controller's 3-button/6-button status (and already letting that change the meaning of the bytes stored for that controller) you could use that to store which peripheral type a controller is, and store whatever data that peripheral needs per frame like normal. (But, I don't know if Gens itself support any peripherals.)
IdeaMagnate wrote:
Just for clarification, recorded frames is a rough indicator of how many frames the person recording the movie saw, vs. how many are in the actual movie?
Yes.
IdeaMagnate wrote:
I'm tempted to merge some of these small chunks into something like a "Movie Information" chunk just to keep the implementation's code size a little more reasonable.
At first I thought it was a mistake that things like the re-record count and frame count weren't stored at a fixed offset in the header.
IdeaMagnate wrote:
I'm hesitant to use the ROM filename for anything. It's usually reliable, but it seems more effective to rely on the ROM itself, which is harder to change. Storing it isn't a bad idea, since it provides a reasonably good idea of how the movie was made, but a name mismatch shouldn't generate any sort of warning, assuming that ROM checksums match.
Right. It's not to be used for determining mismatches, only to give the viewer more information.
IdeaMagnate wrote:
I also fail to see the purpose of a unique identifier. What would this do that the movie CRC wouldn't?
It would not change, that's what it would do differently. A constant and unique (or, at least, pseudo-unique) identifier makes it much easier for the emulator to determine whether a savestate belongs to (that is, was saved during) a given movie. And storing the date/time seems like it can't hurt, anyway.
Since people seem to like the ideas of zip files, I will delete the "compressed?" flags. Should SRAM/states always be compressed within the movie file anyway, like I suggested? I think that's a good idea at least.
>Assuming that Gens can natively read and write a nice text format, is there any reason the binary format would need to be hexedit-friendly?
People might want to write their own tools for manipulating the frame data. There's no reason to make it hexedit-unfriendly at least, seeing how unpopular FCEUs precompressed format is.
Changes I made to the GM2 page:
- I suggest removing the PAL/NTSC flag and just having the country byte. See page.
- Need clarification on how the BIOS chunk works (I think what we have isn't enough).
- Removed "from reset" starting condition. I've seen enough submissions making the mistake of starting from reset instead of power-on, and we have the ability to record resets now, making this unnecessary.
- Names of states would be neat, but not necessary. Having them ordered by the frame they start would also be good, but perhaps it's not necessary to make it a format requirement
---------
About peripherals:
http://www.vidgame.net/SEGA/peripherals.htm
The ones that would need a number are
- Mega Mouse
- Sega Menacer (zapper)
- TeeVGolf (golf simulator)
- Batter Up! (baseball simulator)
The ones that probably don't are
- All the turbo controllers (we have autofire already)
- Activator ring (sends the same signals as a regular controller)
How do we handle Multitaps? I'm not sure - we have a number of controllers byte, but maybe it's not the same thing? Do some games detect multitaps? It seems like Gens only accepts either 1 or 4 controllers on one port anyway, so perhaps this needs to be taken into consideration.
One suggestion for solving it is having a multitap number and a "no controller" number. The number of ports on a Genesis is always two, both must be assigned a value.
0 = no controller
1 = 3-button
2 = 6-button
10 = multitap (next four numbers are controllers connected to it)
1-player:
2
0
3 players on multitap:
10
2
1
2
0
0
6 players on multitap, 2 on first team:
10
2
2
0
0
10
1
1
1
1
We could still have a "number of controllers" byte for the sake of searching through the file. It would always be 2, 6, or 12, and it wouldn't reflect the number of people that are actually playing. Probably someone has a better method for handling this.
------------
Someone mentioned SMS support. Without making ourselves much trouble, I think we could:
- reserve a number for type "SMS controller" in controller configuration
- reserve a flag/number for type "SMS game" in 'rom info'
- make a 'MSpauses'-chunk identical to the 'resets '-chunk for the pause button located on the SMS
Speaking of 'rom info', any reason why game type should be flags instead of a number? A game can't have several types at once. I changed it in the specs, change it back if I'm missing something.
I'd also suggest:
Merge 'record ' and 'author '.
Merge header and 'version '.
