Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
Sour wrote:
Thanks, here are the results: -Both Bionic Commandos sync to the end. -With a small modification (altered the PPU/APU sync after reset in CPU::Reset), Battletoads syncs at first, but then desyncs a bit after the warp. -Both streemerz runs desync. streemerz_joe desyncs almost right away. The other one gets relatively far (maybe ~2 mins in?) and eventually desyncs in a room that had ~10 clowns. For the bugs after compilation, the build process is silly and you need to build 2x for it to work properly (otherwise the resource files will be missing from .exe) - it shouldn't be an issue except the very first time you build it. Also, I just commited support for MD5 hash checks in bk2 files (had only implemented SHA1 ones since it seemed like they were the only thing used), you'll need that if you want to try the Bionic Commando/Battletoads movies.
DId you mean altered PPU/CPU sync? because I don't think battletoads is affected by the APU in any way. Building twice worked! Ok cool now I can do tests. I'll do some more tests as time permits and try to track down where the two emulators differ.
Joined: 6/29/2016
Posts: 53
Alyosha wrote:
DId you mean altered PPU/CPU sync? because I don't think battletoads is affected by the APU in any way.
Yea, my bad. Changed it from 28 PPU cycles before CPU execution starts to 24, iirc (I'm pretty sure this has impact on a couple of test roms, though.)
Site Admin, Skilled player (1254)
Joined: 4/17/2010
Posts: 11479
Location: Lake Char­gogg­a­gogg­man­chaugg­a­gogg­chau­bun­a­gung­a­maugg
Ti_ says oam-decay-test isn't actually passed, and it isn't in any other emulator, because it's impossible to emulate, only to simulate. http://forums.nesdev.com/viewtopic.php?f=3&t=10499#p127049 https://yadi.sk/d/x6fXc5pY3GMJ68 http://forums.nesdev.com/viewtopic.php?f=2&t=96 https://yadi.sk/d/cSdOhqNs3HVNLc I asked him to elaborate or to post here, but he doesn't want :/
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Ti_
Former player
Joined: 12/4/2008
Posts: 27
Location: Russia
Also sprites decaying on dendy goes faster than on famicom (and some differ): https://youtu.be/bQ2_2c89ipk Dizzy The Adventurer (Camerica) (Aladdin) [!].nes game freezes only on hardware dendy, if you press pause or fall into water: (00:15) https://youtu.be/g_9weytf2Y4 fixed roms: _1 - corrected game ntsc/pal detector to properly detect dendy as ntsc - game not freezes, but some sprites decayed when you press pause. (00:40) _2 - hack to update oam every frame - game not freezes, but some sprites blinks (01:18) _FIX - fix1+fix2 = all ok. (01:48). https://yadi.sk/d/qOXyPGrc3HVXKa
Ti_
Former player
Joined: 12/4/2008
Posts: 27
Location: Russia
Famicom (and NES?): about (Pre-render scanline (-1, 261)) in most emulators you can write to ppu data $2007 if rendering is enabled. (but NOT on hardware). afaik only nintedulator gives closiest (but not same) result to real hardware. most games turns off rendering (#$00 to $2001) while vblank for vram updates. some games dont do this, but because designed to fit all vram writes into vblank time, this situtation never happens. But for hacks, translations, or homebrew this may occur (when modify to push more data to ppu). That means it's very important (however not emulated).
Joined: 6/29/2016
Posts: 53
For OAM decay, it's impossible to accurately model because it will most likely be different for every single console sold. Mesen has an option to at least simulate OAM decay, which is helpful to test homebrew software (and not much else, which is why it's disabled by default). Nintendulator seems to have quite a bit of extra logic for $2007 reads & writes, which is probably related to why it gets a much closer result for vram_access_scanline. A high-level description of what the test does and what the numbers shown on the screen mean would be really helpful in trying to figure this one out.
Ti_
Former player
Joined: 12/4/2008
Posts: 27
Location: Russia
Sour wrote:
Mesen has an option to at least simulate OAM decay.
OK. I understand. But your option very far from simulation. 1) you sprites seems just disappers instantly , not by time? 2) not some bits memory corrupted, your just clear all to zeros? you may simulate some bit corruption using random. 3) sprite 0 visually no changes on hardware (but as I understand 'sprite hit' logic broken) even after 4 seconds. Other record from hardware famicom: Wonderland dizzy (first rom version): https://youtu.be/FRLfJbRu_gY various sprite blinking and glitches. seems other not emulated thing. https://yadi.sk/d/NipOb5L93HVbiF
Ti_
Former player
Joined: 12/4/2008
Posts: 27
Location: Russia
Sour wrote:
A high-level description of what the test does and what the numbers shown on the screen mean would be really helpful in trying to figure this one out.
test first clear all vram to zeros. than does various writes and reads to/from PPU_DATA in the end of vblank and later (scanlines -1, 0) then read data back from ppu, and shows it at right side of screen: 178 78 - means value '78' at ppu adress 2178. (2ADDR :VAL) 3d1 d1 - means value d1 at ppu address 23d1. (checks only range $2000-$27FF). than compare it with different patterns happens on hardware.
Joined: 6/29/2016
Posts: 53
Ti_ wrote:
1) you sprites seems just disappers instantly , not by time? 2) not some bits memory corrupted, your just clear all to zeros? you may simulate some bit corruption using random. 3) sprite 0 visually no changes on hardware (but as I understand 'sprite hit' logic broken) even after 4 seconds.
Any unaccessed 4-byte OAM row is set to $00 after ~3000 CPU cycles, which is a known(?) upper limit around which OAM RAM can start decaying. The option is meant as a helpful option for developers to be able to quickly notice that they will encounter OAM decay problems on a real console - I could definitely add random decay values that progressively get worse, but emulating this "accurately" wasn't really too important since the actual behavior will vary wildly from one unit to another. About Wonderland Dizzy: Part of the glitches are caused by the fact that the game is being run on a Famicom. The game reads from $2004, which doesn't work on a Famicom. Enabling the option to disable $2004 reads in Mesen creates a similar glitch at the top and bottom of the status bar, like what is seen in the video. For the sprite flashing, I don't know - it could be the Famicom, it could be the flash cart being used, or it could be some undiscovered behavior that no emulator emulates properly. EDIT: Didn't notice your second post until I posted my reply, thanks for the test's description!
Ti_
Former player
Joined: 12/4/2008
Posts: 27
Location: Russia
Sour wrote:
For the sprite flashing, I don't know - it could be the Famicom, it could be the flash cart being used, or it could be some undiscovered behavior that no emulator emulates properly.
I've looked - they did timed code change in a new rom , LDX #$D2 was changed to LDX #$C3 at adress:
BANKF:C343                 LDX     #$C3
After that most sprites flashing disappers , except part of a star on status bar, and status bar glitchy very slightly. https://www.youtube.com/watch?v=GQIzHOZyU2M&feature=youtu.be Also I did other rom, without timing fix, but wheres BIT $2004 replaced with NOP's
BANKF:C2C2                 BIT     PPU_SPR_DATA
sprites still flashes, but no glitches after status bar (00:40 in video).
Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
@Sour: is there a frame counter in mesen? I found a lag counter but couldn't find the frame counter. Also , what do think of powering on nes from CPU reset routine ? This is what biz hawk currently does and it would make it easier to compare trace logs.
Joined: 6/29/2016
Posts: 53
Ti_ wrote:
I've looked - they did timed code change in a new rom , LDX #$D2 was changed to LDX #$C3
That change turns off rendering 1 scanline earlier (scanline 228 vs 229) and causes it to be re-enabled on scanline 239, instead of scanline 240. I'm not too sure why that would change anything though. Maybe something with enabling rendering during scanline 240. I also spent some time looking at the $2007 behavior - Nintendulator is emulating the PPU's behavior at a pretty low level, it seems right, but getting the precise timing for this thing to work will probably be relatively hard + there might be some edge cases that might exist, too. Messing around with it, I managed to make Mesen pass like 3 out of the 13 scenarios, but nothing better than that.
Alyosha wrote:
is there a frame counter in mesen? I found a lag counter but couldn't find the frame counter. Also , what do think of powering on nes from CPU reset routine ? This is what biz hawk currently does and it would make it easier to compare trace logs.
Surprisingly, there wasn't - I've never had a need for one, and nobody ever asked for one until now. I just added one though - it's in Options->Emulation->Advanced Not sure what you mean by powering from a reset routine? Both power & reset in Mesen call CPU::Reset which runs the PPU/APU for a number of clocks while the CPU is starting and fetching the reset vector's value, etc.
Ti_
Former player
Joined: 12/4/2008
Posts: 27
Location: Russia
Also I see you have an option for "disable emulation OAMADDR bug". I see its effect if you do something like:
	LDA	#$80
	STA	$2003
	LDA	#2
	STA	$4014 ; SPR_DMA	
What happens on hardware, seems how it's works based on time/ chips temperature: When you just turn on console, you will see picture same as bug emulation on. after 1,5-2min bug disappear and you will see picture same as without bug emulation. If you turn off console for 10-20 second, turn on again - bug will disapper much faster - after 20-30 seconds. https://youtu.be/frDP-IdCWo4 I don't know whether it was known , but options is ok.
Ti_
Former player
Joined: 12/4/2008
Posts: 27
Location: Russia
Sour wrote:
getting the precise timing for this thing to work will probably be relatively hard + there might be some edge cases that might exist, too.
Sure. Because of that it's near impossible to do 100% accurate emulation, only 99% :).
Sour wrote:
=Messing around with it, I managed to make Mesen pass like 3 out of the 13 scenarios, but nothing better than that.
Very good, because other emulators do nothing.
Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
Sour wrote:
Surprisingly, there wasn't - I've never had a need for one, and nobody ever asked for one until now. I just added one though - it's in Options->Emulation->Advanced Not sure what you mean by powering from a reset routine? Both power & reset in Mesen call CPU::Reset which runs the PPU/APU for a number of clocks while the CPU is starting and fetching the reset vector's value, etc.
Thanks! I mean counting the CPU cycles that takes in the trace logger so it starts at 8 (9?) instead of 0. EDIT: Also I'm noticing that Mesen seems to not be resetting everything completely on reloading a game / doing a power cycle. I am working with the game Bible Buffet (USA) (Unl) (v6.0) and when I first open Mesen and load the game, it takes 9 lag frames to get to the title screen. But, if I subsequently load the game or do a power cycle, it takes 10 lag frames, any ideas what is causing that?
Joined: 6/29/2016
Posts: 53
Ti_ wrote:
What happens on hardware, seems how it's works based on time/ chips temperature: When you just turn on console, you will see picture same as bug emulation on. after 1,5-2min bug disappear and you will see picture same as without bug emulation.
I wasn't aware of this at all - and I don't think it's been documented, either. The NesDev wiki mentions the bug, but says nothing about it eventually disappearing due to warming up..
Alyosha wrote:
I mean counting the CPU cycles that takes in the trace logger so it starts at 8 (9?) instead of 0. EDIT: Also I'm noticing that Mesen seems to not be resetting everything completely on reloading a game / doing a power cycle. I am working with the game Bible Buffet (USA) (Unl) (v6.0) and when I first open Mesen and load the game, it takes 9 lag frames to get to the title screen. But, if I subsequently load the game or do a power cycle, it takes 10 lag frames, any ideas what is causing that?
Ah, I see - I could probably change it to 8 with no other impact (might make it harder to compare to other emulators, though). For now, you can change it on your end by changing "_cycleCount = -1;" to "_cycleCount = 7;" in the CPU::Reset function. The hard reset mechanism in Mesen is pretty thorough - it deletes every component and recreates a new copy of them. This looks like it's a bug with the lag counter - it will always (incorrectly) count a lag frame when starting a game, except the very first game you load in the emulator after opening Mesen (looks like an easy fix)
Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
Thanks for the fixes! One other thing I noticed is that when I pause the emulator, the counters disappear. This makes it a bit difficult to compare frames, is there a way they can be made to stay visible?
Joined: 6/29/2016
Posts: 53
Enable the "Hide the pause screen" option in the Preferences - it'll remove the pause+overlay and keep the fps/lag/frame counters visible on the screen
Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
Oh that was fast, thank you! I didn't see that option. One other thing, is it possible to add an option to have the VBlank flag set at power on? Most games are pretty good about checking first but there are important examples where the game just blindly waits for 2 vblanks and obvious differences appear when it is set at power on.
Joined: 6/29/2016
Posts: 53
Shouldn't be too hard - and this also made me realize that the flag is not supposed to be cleared on reset, according to the wiki. (it currently does get cleared on reset though) For now, you can edit PPU::Reset() to change it on your end if you need to: _statusFlags = {}; _statusFlags.VerticalBlank = true; // <<-- add this line below the previous one
Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
I've been comparing trace logs between Mesen and BizHawk but things don't seem to be working out. I believe the problem is that Mesen emulates the system like CPU_tick->PPU_tickx3 while BizHawk does the opposite. This is a big problem for games, and for tests that rely on absolutely timed code, since the issues that crop up are related to reading the VBlank flag which is sensitive to 1 ppu tick. We rely heavily on relatively timed test ROMs, which sync themselves to a singal PPU tick, but the syncing process masks out how the emulator deals with it. Both ways are almost certainly wrong and mistiming certain situations. I believe this is a significant roadblock to getting many more games to sync on hardware. Probably what we really need is an emulator in between visualNES and Mesen/BizHawk that runs at the master clock tick level without being a chip simulator.
Joined: 6/29/2016
Posts: 53
Actually, Mesen does 3x PPU followed by 1x CPU (with the APU running before the CPU cycle) too. I think most emulators I've checked also do it this way. A problem with hardware vs emulators that can't be solved are the CPU/PPU alignments - even with a TAS movie that syncs up on a NES, there is no guarantee that it would sync up on every single possible alignment. I had already tried splitting up the ticks into master clock ticks in Mesen a few months back, but as far as I could tell, it didn't really have much of an impact (I didn't exactly go into depth with it - it didn't help me solve the problems I was trying to solve at the time, and broke a few timing roms, so I just abandoned the idea). Most effects happen at the same time as the rising/falling edges of their own frequency (e.g flags in the PPU tend to be set on the rising edge, but some do so on the falling edge of the PPU's clock). So splitting the CPU/PPU clock processing into rising/falling edges could potentially lead to more exact timings, but at the end of the day, the random CPU/PPU alignments are also always going to cause trouble (and there's nothing to be done there) when you're talking about running TAS movies on actual hardware. As far as Bizhawk vs Mesen goes, it's pretty likely that there might be minor timing differences in the PPU implementation, or stuff like DMC or DMA stalling, etc.
Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
Really? I must be reading the code wrong then. So far I've only been looking at read2004.nes start up, and I can't get the first 5 frames to sync up (which only involves reading the VBlank flag) so there must be something else different happening there. Back to the drawing board I guess.
Joined: 6/29/2016
Posts: 53
The CPU calls MemoryRead()/MemoryWrite() for every CPU cycle - the first thing done in either of them is to call IncCycleCount(), which runs the PPU/APU. So this is done before actually reading/writing to memory. If what you're checking is vblank-based, it might be worth mentioning that Mesen sets vblank on cycle 0 (scanline 241) and clears it at cycle 1 (scanline -1) - whereas the timing sheet on the wiki says it should be set on cycle 1, iirc. I think Nintendulator also does this, but unsure. I feel like I took a look at this in the simulators before, but can't recall the result.
Alyosha
He/Him
Editor, Emulator Coder, Expert player (3823)
Joined: 11/30/2014
Posts: 2832
Location: US
Sour wrote:
The CPU calls MemoryRead()/MemoryWrite() for every CPU cycle - the first thing done in either of them is to call IncCycleCount(), which runs the PPU/APU. So this is done before actually reading/writing to memory. If what you're checking is vblank-based, it might be worth mentioning that Mesen sets vblank on cycle 0 (scanline 241) and clears it at cycle 1 (scanline -1) - whereas the timing sheet on the wiki says it should be set on cycle 1, iirc. I think Nintendulator also does this, but unsure. I feel like I took a look at this in the simulators before, but can't recall the result.
Oh, I see yeah that is definitely a difference. I'll have to see if changing that will help things sync up.