I had a crazy idea a few days ago and was wondering if anyone more familiar with the draw behavior of the various consoles could comment on its viability, especially on SNES.
My theory is that, because pre-fifth-generation consoles all base their rendering on sprites and layers, it might be possible to do deblink on a per-layer basis instead of on the final frame. This has two distinct advantages:
- It could reduce artifacts from bleedthrough or bad background blends, since the background would be separable and blinking is almost always an intra-layer phenomenon
- Knowing sprite positions and scroll offsets could massively speed up motion search
The main challenge is whether recombining the layers is actually doable, especially on SNES with its various sprite blend modes. I guess Genesis also has some shadowing modes, which I'm not sure of the mechanics of.
Joined: 4/17/2010
Posts: 11479
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
As in, run the movie to figure out the very blinkings, then read the dumped data and apply transparency on program end, capturing the video. Quite hacky stuff though, I don't think any emucoder would be interested :D
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
As in, run the movie to figure out the very blinkings, then read the dumped data and apply transparency on program end, capturing the video. Quite hacky stuff though, I don't think any emucoder would be interested :D
It'd be more like, the emulator would dump the layers individually along with sprite/scroll information, then at the encode stage, deblink and recombine.
It doesn't seem TOO hacky if the emulators are already capable of hiding layers, but it might be if it interferes with accuracy somehow, or if recombining them isn't possible due to order-of-events or other technical issues with the layer interaction.
What would you do if the game uses color math (i.e. transparencies)?
The SNES calculates 512 color values per line, 256 for "main screen" and 256 for "sub screen". When color math is disabled it shows only mainscreen pixels, otherwise it adds/subtracts a color constant or the subscreen pixel. (Optionally the result can then be halved.) Both screens can be configured to show only specific layers (BGs or sprites), and this can be changed line-by-line.
(in hi-res modes it shows all 512 pixels per line)
What would you do if the game uses color math (i.e. transparencies)?
The SNES calculates 512 color values per line, 256 for "main screen" and 256 for "sub screen". When color math is disabled it shows only mainscreen pixels, otherwise it adds/subtracts a color constant or the subscreen pixel. Both screens can be configured to show only specific layers (BGs or sprites), and this can be changed line-by-line.
(in hi-res modes it shows all 512 pixels per line)
Multiple images would probably have to be dumped, but it's probably possible to break it into associative groups, like having one dump for all additive subscreen pixels from a layer, one for all subtractive subscreen pixels. If it doesn't do any multiply operations, that that should be enough.
If add/subtract/halve status is per-line, then dumping the per-frame line flags might be a better option and split it up internally, but lossless codecs cut solid-color frames down to practically nothing anyway so that may not be necessary.
I have a modified version of snes9x which identifies unique sprites, turns them into png images, and dumps their position and orientation on the screen for every frame. This was supposed to be the beginning of a sprite-based video codec that could be dumped directly from the emulator, but I got a bit too ambitious with the backgrounds (wanting to compose the segments of background that are in vram at any given time into the large, continuous background they represent), and the project was not completed (also, this would not have been able to handle any of the color arithmetic stuff the SNES uses). But it sounds a bit similar to what you are suggesting here, so perhaps some of it might be of use. It would give you both the position and shape of the sprite at any time, letting you identify exactly which pixels need to be changed.
To be honest, though. I think a much easier approach (though game dependent) would be to identify what controls the blinking, and simply turning it off. I'm not sure how easy that would be to find, though.
Another possibility is to monitor the OAM to look for and eliminate blinking behavior there.
Of course, turning off blinking would not be quite as good as replacing it by transparency.
Joined: 4/17/2010
Posts: 11479
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Guys, hacking the emulator to output already deblinked things sounds much more efficient to me than what you describe here (dumping thousands of images and messing with them). Remember that sprites can be put above and below the BG in the game at the same time, which makes reconstructing them back almost impossible. Your way also looks like too much effort to pull out each time you encode a movie.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
If it can be done in the emulator, that'd be even better, but the layers have to be able to be split up at least internally, otherwise the intra-layer motion search that's the entire point of attempting this won't work.
The main reason I'm for exporting layer info to the AVI or something is that I think tuning it via an avisynth plugin or something would be a better workflow than adjusting it in the emulator and having to re-dump everything if tweaks are made.
Joined: 4/17/2010
Posts: 11479
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
The layers get outputted to the client separately anyway, it's not the core side I believe, because otherwise it won't be possible to switch these layers. So, somewhere between the outputted separate layers and rendering the final bitmap we need to do our calculations.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
