Post subject: Emulating classic MacOS with SheepShaver for Ferazel's Wand
Joined: 8/28/2015
Posts: 3
Hi! Over the last year and a half or so, I've been working on a series of patches to SheepShaver to allow TASing games for the classic MacOS. The patches are all available on my github and provide, in short:
  • Deterministic playback
  • Input recording, re-recording, and replay
  • Frame advance
  • Savestates (mostly)
  • Audio/video export via libav
This work was all intended to allow me to make a TAS of one of my favorite computer games as a kid: Ferazel's Wand. With the patches applied, I was able to record some videos which I've posted to youtube: https://www.youtube.com/playlist?list=PL-Qewl8D5Gq2OUVK-C4foXiwhMfRM1eTZ These are mostly just a proof-of-concept, and there's a lot I know I need to improve on or completely redo in those videos. (That said, I am definitely interested in any feedback on the gameplay in these videos, so I can keep things in mind for the future.) However, to get this up on TASVideos at all, there's a few obstacles, which is why I'm posting in the forums. First, the most obvious thing is that SheepShaver is not an accepted emulator. I'd like to get it accepted! From looking over the emulator requirements and desired features, here's how SheepShaver stacks up: Movie files/authenticity Movies are (for now; this can change if necessary) stored in a format of my own devising. They can be saved and replayed without issue. Movies always start from power-on; starting from a savestate or loading a savestate is not possible. Movies mostly only contain input; the only exception to that is that there is an instruction in the movie format for "dump the JIT cache", for reasons I'll explain later. There's (currently) no length on a movie other than "time of last input", but I can add that if necessary. Re-recording I've tested this extensively in my own usage, and I've only rarely gotten desyncs. It is possible to get random desyncs in some circumstances, but it is quite rare. Unfortunately, I haven't been able to discover the cause of this yet. Emulation stability Movie files are portable and extensible, and at this point I don't think I'll have any issues maintaining backward-compatibility. Other movie information Not added yet, but trivial to do so. Accessibility All the source is available on my github. Right now, I've been doing all of my development only on linux, because SheepShaver has some interesting issues compiling for 64-bit and/or newer-than-10.6 OS X. I don't have a windows box around to test on, and I don't have very much familiarity with programming for a windows environment, but if necessary I can try to find someone to help me out here. Possible to make AVI Export is done with libav, and writes out a huge AVI. It's trivial to transcode after export, but I haven't looked much into doing the transcoding at export time. Emulation quality I saved this for last because it's kind of tricky. SheepShaver does do some patching of the system files and ROM. This is one of the areas of the codebase that I'm not particularly familiar with, though. I don't know what is considered standard or acceptable here overall, though. I think what it might come down to is that I don't think it would ever be possible to replay a movie file on real hardware, not because of the aforementioned patching, but because of the nature of the emulation. To be able to have precise, deterministic input replay, the emulator has to execute everything triggered by the 60Hz interrupt, and then execute some more instructions. The "execute some more instructions" bit is the tricky part; because this is emulating a whole general-purpose OS and a game, and not just a game intended to run directly on the hardware, it's much more difficult to determine "is the program just waiting for the next interrupt now?" The way I've solved this is just by simply limiting the number of cycles executed post-interrupt. The final nail in the coffin, as it were, is that this isn't a true "CPU cycles" count, but because SheepShaver has a JIT, it's effectively the number of JIT-translated blocks that have been jumped to. Getting back to what I said earlier about the JIT cache: the JIT will as-needed translate PPC instructions to x86 and then execute the x86 parts directly. As an optimization, the translated blocks are kept around so that they can be re-used when appropriate. However, this causes a problem with savestates. If block A has a one-time jump to block B, and block B has a one-time jump back to block A, that can be executed as one 'cycle', but only if both blocks are already translated. Otherwise, if block B is not already translated, this will be counted as multiple 'cycles', but only the first time. What this means is: if you've play along, translate block A, record a savestate, translate block B, and then load the savestate, the JIT could be in a different state, and the 'cycle count' will be off. This was causing desyncs until I figured it out, and my solution-at-the-time-that-actually-seems-to-work-okay is to dump the JIT cache immediately after reading or writing a savestate. Anyway, that's all about SheepShaver. Even if the emulator itself is accepted, there's a few other obstacles to getting a Ferazel's Wand video up on TASVideos. There's four things that someone would need in order to verify the movie:
  • The movie file itself, naturally
  • A New World ROM, which is (as far as I know) not able to be legally distributed
  • A copy of MacOS 9.0.4, which Apple (for whatever reason) still cares about, and as such is not able to be legally distributed
  • A copy of Ferazel's Wand, which Ambrosia Software is still selling (but only on CD-ROM), and as such is not able to be legally distributed
The latter three things on this list I all have, but couldn't give out. In this case, I'm not really sure what I can do. Working from a MacOS 9.0.4 installer CD image and the Ferazel's Wand CD image, I could make movies to prepare a hard drive, install the OS, and install the game, but this still requires files that I can't give out. I certainly could record the hash sums of the files that I have, but I don't know if there's any guarantee anyone else would get the same values with their own physical copies. I hope that these are all surmountable issues. But, since I haven't been able to come up with a solution myself, I hope that the collective wisdom of the forum will be able to help. Thank you for your time and consideration!
Editor, Experienced player (570)
Joined: 11/8/2010
Posts: 4036
This sounds promising. I'm not sure a Mac TAS emulator has even been attempted before, but you've come a long way with this and I hope the issues can get worked out and your emulator gets accepted here! I have one MacOS game I think I would like to TAS; it's a 3D game, but it sounds like your program should be able to run such a game already.
Patashu
He/Him
Joined: 10/2/2005
Posts: 4043
Wow, this is super cool! I wish you luck on getting this accepted.
My Chiptune music, made in Famitracker: http://soundcloud.com/patashu My twitch. I stream mostly shmups & rhythm games http://twitch.tv/patashu My youtube, again shmups and rhythm games and misc stuff: http://youtube.com/user/patashu
Joined: 7/2/2007
Posts: 3960
Good luck! Makes me wonder what old Mac games deserve TASes...better start thinking in preparation for the wishlist thread. :)
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Experienced player (690)
Joined: 11/23/2013
Posts: 2233
Location: Guatemala
Movie length could be really really handy. You should add that. Also a frame count and rerecord count.
Here, my YouTube channel: http://www.youtube.com/user/dekutony
Joined: 8/28/2015
Posts: 3
CoolKirby wrote:
This sounds promising. I'm not sure a Mac TAS emulator has even been attempted before, but you've come a long way with this and I hope the issues can get worked out and your emulator gets accepted here! I have one MacOS game I think I would like to TAS; it's a 3D game, but it sounds like your program should be able to run such a game already.
Thanks! Most things seem to run without major issues in SheepShaver. I've seen some things exhibit strange issues, but I haven't dug into it too much since Ferazel's Wand has been my biggest concern.
Kurabupengin wrote:
Movie length could be really really handy. You should add that. Also a frame count and rerecord count.
Yeah, I definitely need to add re-record count. Frame count would just be the number of frames an exported AVI would contain for a movie?
Joined: 10/28/2013
Posts: 130
Location: United States
Derakon wrote:
Good luck! Makes me wonder what old Mac games deserve TASes...better start thinking in preparation for the wishlist thread. :)
TaskMaker! That's the first one that comes to mind straightaway. Dark Castle would be another good one.
Joined: 7/2/2007
Posts: 3960
goldenband wrote:
Derakon wrote:
Good luck! Makes me wonder what old Mac games deserve TASes...better start thinking in preparation for the wishlist thread. :)
TaskMaker! That's the first one that comes to mind straightaway. Dark Castle would be another good one.
Yeah, TaskMaker has some pretty good options for luck manipulation. The Exile games would be pretty long, but they're classics; maybe someone would be interested in doing them. Harry the Handsome Executive (another Ambrosia SW game) might be good; it has weird movement that's tricky to do quickly. Man, most of the games I remember from that era were endless (and most of them were also clones of arcade games like Asteroids and Centipede). I remember Ambrosia did some kind of puzzle/action game involving pushing bubbles around, which might be interesting. Can't remember what it was called though.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Skilled player (1741)
Joined: 9/17/2009
Posts: 4981
Location: ̶C̶a̶n̶a̶d̶a̶ "Kanatah"
Does the movie sync back on other computers? I recall at least one TAS where it couldn't be published because it only synced on the submitter's computer.
Post subject: Re: Emulating classic MacOS with SheepShaver for Ferazel's Wand
Joined: 2/3/2013
Posts: 320
Location: Germany
habnabit wrote:
Hi! Over the last year and a half or so, I've been working on a series of patches to SheepShaver to allow TASing games for the classic MacOS. The patches are all available on my github and provide, in short: [...]
That's pretty awesome. I hope we can integrate it with the site soon. Does upstream know of this? It is always better to (try to) get it into mainline. As for your points: Movie files/authenticity It would be really good to have a proper specification for this sort of thing. A good indication of whether the format sucks or not is to write a parser for it (our site will need a parser for submissions too) and see how difficult that is. See Wiki: EmulatorResources (click on a movie format) to see what others look like. A "time to last input" field would be convenience and will need to be verified anyway. For simplicity's sake could you disable the JIT cache for TASing (and movie playback) to avoid the problem with the JIT cache altogether? How much of a performance cost would that be for TASing? Re-recording Spurious desyncs due to nondeterminism in the emulator are very frustrating. Nobody from site staff wants to dance the "find out who can sync the movie file reliably"-dance., so this'll need testing. Emulation stability Backward compatibility doesn't matter, but is nice to have, as long as old versions are easy to come by. A stable (or extensible) movie file format is more important (e.g. Dolphin's Wiki: DTM was missing information that lead to all completion times on the site being wrong, some still aren't fixed). Other movie information Nice to have would be things like author and a comment as well as the rerecord count. More necessarily it should contain information on what is needed to make the movie sync: Hashes of the ROM, OS and game, possibly emulator settings, that, when wrong, could lead to desync (Wiki: DTM has this). This is probably the most complicated part and not obvious to implement. Accessibility Cross-platform would be nice boon to have. Many emulators on this site only run really well on Windows. I'm not sure whether we have anyone using OS X, but we have Linux (e.g. me) and Windows users for testing. Possible to make AVI Here it would be good to have to possibility to write the raw video to a pipe for processing (maybe optionally packed in y4m or something (x264 can be patched to also accept RGB in y4m)), because both hardcoding all the encoding parameters and not hardcoding them (needs a frontend) suck. Generally a good thing is to encode lossless H264 (RGB) or FFV1 and lossless PCM audio and mux it into MKV (proper support for long files, please let AVI die) or something. We have been bitten by lots of terrible audio/video capture in emulators in the past. Here's the chance to make this good :) Emulation quality As long as you can TAS reliably with it (this is only for runners, maybe judges) and reliably verify the movies (this is for the site), all is well. Again, what would be the costs and benefit to disabling the entire JIT thing? Miscellaneous Ah yes, the good old proprietary software licensing issues... For the New World ROM and Mac OS 9.0.4 we can safely assume that it is abandonware (I found them without problem). The game itself doesn't seem to be abandoned, but I'm sure we can find a solution here. DOS and JPC-RR have similarly complicated setups, so this is nothing new (see Wiki: http://tasvideos.org/EmulatorResources/JPC.html) This is a pretty major contribution, I'll try to build your tree later on.
All syllogisms have three parts, therefore this is not a syllogism.
Joined: 8/28/2015
Posts: 3
Derakon wrote:
Yeah, TaskMaker has some pretty good options for luck manipulation. The Exile games would be pretty long, but they're classics; maybe someone would be interested in doing them. Harry the Handsome Executive (another Ambrosia SW game) might be good; it has weird movement that's tricky to do quickly. Man, most of the games I remember from that era were endless (and most of them were also clones of arcade games like Asteroids and Centipede). I remember Ambrosia did some kind of puzzle/action game involving pushing bubbles around, which might be interesting. Can't remember what it was called though.
Harry is another one of my favorite childhood games, and I've definitely thought about how to run it, but yeah, movement is nontrivial. Bubble Trouble is the game you're thinking of, though. Ambrosia is at least good about keeping archives of their old games around: http://www.ambrosiasw.com/games/old/
jlun2 wrote:
Does the movie sync back on other computers? I recall at least one TAS where it couldn't be published because it only synced on the submitter's computer.
Yes. It syncs without issue on at least two different physical computers and a VM.
RGamma wrote:
That's pretty awesome. I hope we can integrate it with the site soon. Does upstream know of this? It is always better to (try to) get it into mainline. As for your points:
Not yet. Some of the changes I've made are pretty invasive, especially around save-stating. The emulator was not built to be able to be able to persist and reload state from disk.
RGamma wrote:
It would be really good to have a proper specification for this sort of thing. A good indication of whether the format sucks or not is to write a parser for it (our site will need a parser for submissions too) and see how difficult that is. See Wiki: EmulatorResources (click on a movie format) to see what others look like. A "time to last input" field would be convenience and will need to be verified anyway. For simplicity's sake could you disable the JIT cache for TASing (and movie playback) to avoid the problem with the JIT cache altogether? How much of a performance cost would that be for TASing?
I've written parsers for the format in C++, python, and swift. It's not difficult to do, and I can write up a more formal spec. Regarding the JIT, the original reason that I cared about it at all was because I originally wasn't even considering the option of making the emulator itself export AVI, and so I was trying to make the emulator able to run in realtime. It's less of a concern now that I have libav integration, but it is definitely a noticable speed difference. I'll play around with it some more to see just how bad it is.
RGamma wrote:
Spurious desyncs due to nondeterminism in the emulator are very frustrating. Nobody from site staff wants to dance the "find out who can sync the movie file reliably"-dance., so this'll need testing.
So, I was just talking about this with some other folks, and I think I remember that basically all of the desync issues I was having were a consequence of not disabling input while inputs are being played back. Now that I'm thinking about it again, I'll put that on the top of my todo list to fix.
RGamma wrote:
Backward compatiability doesn't matter, but is nice to have, as long as old versions are easy to come by. A stable (or extensible) movie file format is more important (e.g. Dolphin's Wiki: DTM was missing information that lead to all completion times on the site being wrong, some still aren't fixed).
Okay. Since this is all in git, I don't think getting/building old versions should be difficult.
RGamma wrote:
Nice to have would be things like author and a comment as well as the rerecord count. More necessarily it should contain information on what is needed to make the movie sync: Hashes of the ROM, OS and game, possibly emulator settings, that, when wrong, could lead to desync (Wiki: DTM has this). This is probably the most complicated part and not obvious to implement.
Good ideas. I'll implement these soon too.
RGamma wrote:
Cross-platform would be nice boon to have. Many emulators on this site only run really well on Windows. I'm not sure whether we have anyone using OS X, but we have Linux (e.g. me) and Windows users for testing.
I'm on OS X, and I have a few different machines with a few different versions of OS X. I'm going to try to build it on 10.6 to see if that will still produce usable binaries, since as far as I know that's the only way to support OS X.
RGamma wrote:
Here it would be good to have to possibility to write the raw video to a pipe for processing (maybe optionally packed in y4m or something (x264 can be patched to also accept RGB in y4m)), because both hardcoding all the encoding parameters and not hardcoding them (needs a frontend) suck. Generally a good thing is to encode lossless H264 (RGB) or FFV1 and lossless PCM audio and mux it into MKV (proper support for long files, please let AVI die) or something. We have been bitten by lots of terrible audio/video capture in emulators in the past. Here's the chance to make this good :)
This is something that I'm not very familiar with overall, so some guidance would be appreciated. Right now I'm using libav to write out an AVI with YUV 420p video (transcoded by libavformat from the raw framebuffer) and PCM_S16LE audio (byteswapped from the native raw PCM_S16BE). So, would it be better to write out the raw audio/video and stuff it into a container after the fact? If so, where can I learn more about the 'raw' formats that a muxer would expect?
RGamma wrote:
As long as you can TAS reliably with it (this is only for runners, maybe judges) and reliably verify the movies (this is for the site), all is well. Again, what would be the costs and benefit to disabling the entire JIT thing?
Okay, phew. That was one of the things I was worried might make this infeasible. I'll see what I can do about JIT.
RGamma wrote:
Ah yes, the good old proprietary software licensing issues... For the New World ROM and Mac OS 9.0.4 we can safely assume that it is abandonware (I found them without problem). The game itself doesn't seem to be abandoned, but I'm sure we can find a solution here. DOS and JPC-RR have similarly complicated setups, so this is nothing new (see Wiki: http://tasvideos.org/EmulatorResources/JPC.html)
Okay. Adding support for verifying hash sums of the input files should probably be enough here, then.
RGamma wrote:
This is a pretty major contribution, I'll try to build your tree later on.
Let me know if you have any issues. I'm trying to fix the build system to not suck right now, and I think that's going to mean moving to CMake.