Hello all,
I made this as a bit of a side project to an input viewer. The concept is fairly simple: record inputs while playing on original hardware, then have it saved as a BizHawk movie for playback afterwards. There are still a lot of kinks to be worked out for proper syncing and some other timing inconsistencies, but it didn't take too long to come up with the PoC below:
Link to video
This is currently based on heavily modified NintendoSpy code, with inputs recorded based on a combination of serial communication and host-side timing events. I'll eventually set it up on github or something when it doesn't look like such a steaming pile and is a bit more consistent.
This is useful for a few reasons:
-Alternative playback of standard RTAs. Simple enough, it just gives a more precise way to play the latest run with more insight into the system state and player errors.
-Testing for emulation inconsistencies for specific games. The usual approach of "make a TAS movie sync to console" is just reversed; if a correct recording of console inputs can't sync on emulator, there may be things worth investigating to iron it out.
There are some limitations with the current approach. The code is currently only designed for SNES, although it can be easily adapted to most of the other controller protocols. The second is that it syncs based on power-on; resets do not have an identifiable event across any of the controller signals. Runs requiring resets or power cycles will need to be explored further.
In any case, I'm still chugging away with this. The current code is ugly and has a lot of sync issues, but I'm redesigning the microcontroller code to get around a lot of the problems. For perfect consistency, I plan to set it up to log to mass storage devices as well which will help issues relating to not having a host RTOS. Any comments, suggestions, or questions are welcome.
Thanks,
~Omni
Joined: 12/8/2012
Posts: 706
Location: Missouri, USA
All I can say is that this is an amazing concept. This sounds like it could eventually help make more accurate emulators, also.
"But as it is written, Eye hath not seen, nor ear heard, neither have entered into the heart of man, the things which God hath prepared for them that love him." - 1 Corinthians 2:9
This is an excellent project. Going console -> emulator should be just as sync stable as emulator -> console. Whatever you do on console should be accurately reflected in an emulator, if that emulator is any good to begin with.
One thought that comes to mind is console -> console. I know, you just played it on console, but this would be a good test to see how stable the input log itself may be when played back on the console again.
One thought that comes to mind is console -> console. I know, you just played it on console, but this would be a good test to see how stable the input log itself may be when played back on the console again.
This makes me wonder how well consoles would sync between themselves. They might actually not sync in the long run. Sure, they may work identically in all respects, except perhaps one: Their CPU clocks might not tick exactly at the same rate. If there's even a tiny fraction of a difference in their clockrate, that will probably cause a desync eventually.
(On the other hand, that might not be the case if they sync to the TV's refresh rate. In other words, if they do some work to calculate the next frame, wait for the TV to start vertical retrace, do work to calculate the next frame, and so on. Since the TV is synced to the 50 or 60 Hz mains, that ought to keep all consoles in sync. Or at least this was so in the past. With current TV's I have no idea, really.)
This makes me wonder how well consoles would sync between themselves.
I was thinking about something sort of similar: with a system like this you could theoretically play networked multiplayer games on the original SNES. Each player's input would be read by the device, streamed over the network, and sent to the other player's console via a custom controller.
My guess is they'd desync after maybe a few minutes (even ignoring any latency issues due to the network), but it's a cute idea.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Emulator Coder, Site Developer, Site Owner, Expert player
(3574)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
This is a really cool idea, and I think it has a lot of intriguing possibilities for emulator development as well as RTA verification. a RTA attempt submitted as an input file would definitely give it more validity, and can be scrutinized more.
One thought that comes to mind is console -> console. I know, you just played it on console, but this would be a good test to see how stable the input log itself may be when played back on the console again.
This makes me wonder how well consoles would sync between themselves. They might actually not sync in the long run. Sure, they may work identically in all respects, except perhaps one: Their CPU clocks might not tick exactly at the same rate. If there's even a tiny fraction of a difference in their clockrate, that will probably cause a desync eventually.
(On the other hand, that might not be the case if they sync to the TV's refresh rate. In other words, if they do some work to calculate the next frame, wait for the TV to start vertical retrace, do work to calculate the next frame, and so on. Since the TV is synced to the 50 or 60 Hz mains, that ought to keep all consoles in sync. Or at least this was so in the past. With current TV's I have no idea, really.)
The sync points for this implementation at least are certain signals in the controller serial protocol, which should be stable at least until the point of environmental effects having an impact on processor operation. This means that even if the clocks between different consoles are slightly different, the controller input latching happens at regular intervals with a far longer period than the source clock. Unless lag frames are mysteriously skipped here and there, it should sync on one console just as well as the next. The only other factor would be subtle differences in the console hardware (such as PPU) that have a visible effect on gameplay such as here:
Link to video
This makes me wonder how well consoles would sync between themselves. They might actually not sync in the long run. Sure, they may work identically in all respects, except perhaps one: Their CPU clocks might not tick exactly at the same rate. If there's even a tiny fraction of a difference in their clockrate, that will probably cause a desync eventually.
According to byuu, real hardware units do differ slightly from each other. Usually that's not a problem, except when you have multiple clock sources (like CPU and APU).
Warp wrote:
(On the other hand, that might not be the case if they sync to the TV's refresh rate. In other words, if they do some work to calculate the next frame, wait for the TV to start vertical retrace, do work to calculate the next frame, and so on. Since the TV is synced to the 50 or 60 Hz mains, that ought to keep all consoles in sync. Or at least this was so in the past.)
No, the console sets the timing (of course roughly within the boundaries of NTSC spec and mains frequency) via the sync pulse. Remember that a TV has to be flexible enough to lock onto any broadcast channel's video signal, with no way to know when a field starts except from the signal itself.
Consoles even twist the signal a bit to achieve a progressive picture, and the resulting number of frame per second is different from the 30 (NTSC b/w) or 30/1.001 (NTSC color) frames per second.
Derakon wrote:
you could theoretically play networked multiplayer games on the original SNES. Each player's input would be read by the device, streamed over the network, and sent to the other player's console via a custom controller.
No, the console sets the timing (of course roughly within the boundaries of NTSC spec and mains frequency) via the sync pulse. Remember that a TV has to be flexible enough to lock onto any broadcast channel's video signal, with no way to know when a field starts except from the signal itself.
Ah, I knew that but I totally forgot. It indeed is so that it's the TV that syncs with the console (within certain limits), rather than the other way around. This would mean that the syncing problem between two consoles can be even worse.
So a few developments:
-Much-improved microcontroller code that should accurately and more efficiently capture input frames using SPI.
-Timer method for capturing no-latch frames. Can still desync if 1000+ consecutive no-latch frames occur, but that should be pretty unusual.
-Switched around some of the communication protocols so that the recording client should be more stable.
-Recordings now start at power-on rather than at the first latch.
Long-term sync issues are somewhat dampened, but not completely eliminated yet. A bigger issue is that occasional glitches appear on the 5V line I'm using as the power signal, so I will need to adjust my logic a bit more to account for 1-frame instances of signal loss.
On the other side of things, I've been thinking more about dealing with sync from the perspective of re-orienting movies so that latch-less frames are not recorded. This means that inputs are directed per latch instead of per-frame. There may be some other issues with this that I need to work out, but it could simplify both emulator sync and console playback issues. I need to understand better how emulators handle the controller latch signals though. Making a console-playback program should be pretty straightforward though if the movie is structured just to operate on latched frames.
As a side-point, does anybody have any information on how lagframes are determined in BizHawk or others? It makes sense if it's related to expected clocks per frame and counting assembly ops, but if there's a more trivial way of picking up on it that would be good to know.
Put everything up on SourceForge. Go get it, if you'd like. If you'd like to help out, let me know.
https://sourceforge.net/projects/inputcontrol/
I included some basic sample programs for both logging and playing back. Playback is a bit more iffy at this point, but it has worked on a handful of things that I've tried.
I reworked everything to completely ignore actual frame-sync and focus entirely on syncing to controller latches. This means that the new logs are not immediately compatible with BizHawk, but I threw together a lua script that would play the input logs in the same manner. One way or another, I have a handful of test movies that sync on both BizHawk and console.
Some current limitations:
-Resets and power cycles are not currently logged. Power cycles are used as delimiters of sorts between separate logs, and there's just no way to monitor for or exercise soft resets from the controller lines.
-Some games initialize RNG or otherwise via uninitialized memory values. Just like in emulator, you won't necessarily be able to force the same memory state for console playback.
-The pins on the Arduino seem to be unreliable sometimes. Too much noise or some motion can cause them to miss inputs, which desyncs all of the serial communication.
-Right now, you have to swap firmwares depending on whether you're doing logging or playback. Ideally it would just be one firmware and the host program would direct its operation.
-I don't know why, but most times when starting up playback the Arduino has 4 bytes already in its read buffer. I don't know if this is an artifact of the connection code, but right now I just toss the data into the bitbucket to get ready for the actual data. If there's ever more or less than 4 bytes to throw out, it will start off desynced.
This reminds me of importing Brawl and Mario Kart replays from the Wii to Dolphin. It used to be broken, but there was a timing fix probably about a year ago so it should be rock solid now.
Just as a minor update, I reworked a lot of things such that:
-The user interface and firmware files for logging/playback are the same. This means you can log a file, then play it back without having to reflash firmware.
-Incorporated my Stepmania-like input viewer into the program as an optional view.
-When I remade it, I migrated to a bit-banged SPI so it wouldn't be necessary to swap wires when switching modes. I think this actually made it less consistent, so I'm in the progress of reverting it back to hardware SPI.
That said, it isn't uploaded to the sourceforge and probably isn't quite ready for a release yet. My time has been taken up by other things lately, but I should be able to patch it up the rest of the way in a couple weeks.
I was working on a board that would fit into NES or SNES controllers to do input recording, mostly for this purpose. Never got it done. May revisit doing it at some point.
Was thinking of that too, but involves controller connectors and such...but I could modify some shitty third party NES or SNES extension cable to do this. I was thinking along the lines of something integrated but both ideas were on the table.
I dropped it for the time being but may revisit it. My MultiReplay hardware is capable of being an in-the-middle device but needs programmed and will require cutting up an extension cable to do this.
Was thinking of that too, but involves controller connectors and such...but I could modify some shitty third party NES or SNES extension cable to do this.
This is how it is done in much of the streaming community at present, built according to guides for NintendoSpy or otherwise. There are software and hardware limitations with the standard implementations of NintendoSpy that prevent it from being an accurate logger (typically only used for input displays, which don't need accuracy). That was why I started this up; to see if I could rearrange it into an accurate recording system. The playback portion was an expansion on that.
In any case, modifying extension cables is a fairly simple and cheap exercise so long as you have a soldering iron and some good wire strippers. The issue at present is that many of the folks selling prebuilt adapters conform to different software revisions which have different wire requirements. My current implementation also optionally uses the +5V line to indicate when the console is powered, but I don't believe any other implementation uses this.
Newest version of this project is up. Major new change is that the logging and playback software/firmwares have been merged so you don't need to load a new firmware whenever you want to switch. Also a much better GUI and more consistent logging in this version. Finally, I included a side project, my Input Mania inputs visualization, into it as well.
That said, still lots to improve moving forward. It works fairly well, but games that are especially susceptible to lag will desync both in BizHawk and on console playback sometimes.
In any case, binary and code available here: https://sourceforge.net/projects/inputcontrol/
How efficient would it be to take a console-verified TAS, re-sync it on console, record that input, and take that input to compare to the original movie?
You should be able to see areas where your code could improve, seems like.
I don't know what you mean by taking a "console-verified TAS" and "resyncing it on the console" - it has already been synced on the console, which is why it is console-verified.
If movies don't sync on NES, it is because emulation quality is poor. The solution is not to compensate inputs, it is to fix emulation.
If movies don't sync on SNES it is likely because the code being executed relies on bus timings that can't be met in hardware without extremely accurate clocks. The solution is not to compensate inputs, it is to inject clocks or otherwise consider the game unsyncable.
How efficient would it be to take a console-verified TAS, re-sync it on console, record that input, and take that input to compare to the original movie?
You should be able to see areas where your code could improve, seems like.
I'm not sure what you'd accomplish by doing this.
It should be simple enough to make your own inputs log from an already existing TAS, however I have not made a script to do this. Having it play back successfully on console will depend largely on the accuracy of the emulation for that particular game and possible inconsistencies in the hardware itself.
Let me try to explain a little better.
The fact that the console-verified TAS exists means that its inputs are of high-enough quality to be input to the real hardware and execute to completion.
If you take the output from the Input Logger and play it back on an emulator, you can detect inconsistencies between the real hardware and the emulator, or errors in the Input Logger (should there be any).
It stands to reason that if you:
1) take a known control input file that was "generated" via emulator
2) input it to the console
3) use Input Logger to record that same input
you could run a diff of the original input file vs. the Input Logger file and potentially find any discrepancies.
