Important thing to remember: decimate does not work with threads > 1 or bframes > 0. Using decimate with either threads of bframes will cause choppiness and generally bad things. At least, this is the case in mencoder.. it's possible threads and decimate might work in another encoder somewhere.
Also, I strongly advise you not to use b-frames in AVIs.
Now to actually be helpful. The following is (probably) what I used for the last SM64 encode I did. (save as something.bat):
title N64 TAS Encoding
REM Audio Encoding
start /wait mencoder.exe -priority idle -oac mp3lame -mc 0 -lameopts preset=71:aq=0:mode=0 -ovc copy -o mixed-a.avi mixed.avi
REM Video Pass 1
start /wait mencoder.exe -priority idle -vf decimate=30:0:0:1 -aspect 4:3 -oac copy -mc 0 -ovc x264 -x264encopts qp=20:pass=1 -o Video-comp-1stpass.avi mixed-a.avi -sub subtitle.sub -subfont-text-scale 4 -sub-bg-alpha 15 -font ./mplayer/subfont.ttf
REM Video Pass 2
start /wait mencoder.exe -aspect 4:3 -vf decimate=30:0:0:1 -priority idle -oac copy -mc 0 -ovc x264 -x264encopts bitrate=700:pass=3:frameref=15:qcomp=1:ratetol=100:me=umh:me_range=64:subq=7:8x8dct:mixed_refs:fast_pskip=no -o Video-comp.avi Video-comp-1stpass.avi -sub subtitle.sub -subfont-text-scale 4 -sub-bg-alpha 15 -font ./mplayer/subfont.ttf
start /wait mencoder.exe -aspect 4:3 -vf decimate=30:0:0:1 -priority idle -oac copy -mc 0 -ovc x264 -x264encopts bitrate=700:pass=3:frameref=15:qcomp=1:ratetol=100:me=umh:me_range=64:subq=7:8x8dct:mixed_refs:fast_pskip=no -o Video-comp.avi Video-comp-1stpass.avi -sub subtitle.sub -subfont-text-scale 4 -sub-bg-alpha 15 -font ./mplayer/subfont.ttf
start /wait mencoder.exe -aspect 4:3 -vf decimate=30:0:0:1 -priority idle -oac copy -mc 0 -ovc x264 -x264encopts bitrate=700:pass=3:frameref=15:qcomp=1:ratetol=100:me=umh:me_range=64:subq=7:8x8dct:mixed_refs:fast_pskip=no -o Video-comp.avi Video-comp-1stpass.avi -sub subtitle.sub -subfont-text-scale 4 -sub-bg-alpha 15 -font ./mplayer/subfont.ttf
I think qcomp=1 and qp=20 on the first pass is what I did to fix my initial-blurring problem. At the very least qcomp is overwritten by 1st pass data, so I believe it makes very little difference whether it's in the command line or not. Also take note at the abnormally high bitrate. This is because decimate forces duplicate frames to 0 bytes, which x264 does not take into account for. As a result, for a game that runs at 20fps, the specified bitrate should be ~3 times what you're aiming for.
Where games are almost fully 60fps with no lag and very little blank transition screens, b-frames will probably be better than decimate. Here's what I used for Ecco 2:
title Genesis TAS Encoding
REM Audio Encoding
start /wait mencoder.exe -priority idle -oac mp3lame -mc 0 -lameopts preset=80:aq=0:mode=0 -ovc copy -o mixed-a.avi mixed.avi
REM Video Pass 1
start /wait mencoder.exe -priority idle -aspect 4:3 -oac copy -mc 0 -ovc x264 -x264encopts threads=4:qp=20:bframes=6:b_pyramid:weight_b:brdo:bime:pass=1:me=umh:me_range=64:subq=7:8x8dct -o Video-comp-1stpass.avi mixed-a.avi -sub subtitle.sub -subfont-text-scale 4 -sub-bg-alpha 15 -font ./mplayer/subfont.ttf
REM Video Pass 2
start /wait mencoder.exe -aspect 4:3 -priority idle -oac copy -mc 0 -ovc x264 -x264encopts threads=4:bitrate=420:pass=3:frameref=15:bframes=6:weight_b:b_pyramid:brdo:bime:qcomp=1:ratetol=150:me=umh:me_range=64:subq=7:8x8dct:mixed_refs:fast_pskip=no:trellis=2 -o Video-comp.avi Video-comp-1stpass.avi -sub subtitle.sub -subfont-text-scale 4 -sub-bg-alpha 15 -font ./mplayer/subfont.ttf
start /wait mencoder.exe -aspect 4:3 -priority idle -oac copy -mc 0 -ovc x264 -x264encopts threads=4:bitrate=420:pass=3:frameref=15:bframes=6:weight_b:b_pyramid:brdo:bime:qcomp=1:ratetol=150:me=umh:me_range=64:subq=7:8x8dct:mixed_refs:fast_pskip=no:trellis=2 -o Video-comp.avi Video-comp-1stpass.avi -sub subtitle.sub -subfont-text-scale 4 -sub-bg-alpha 15 -font ./mplayer/subfont.ttf
Since decimate isn't used, I can afford to use as many threads as required to maximise speed :)
Both of the above batch files are based off of the ones in the
MakingAVI Resource.