Select Graphics -> Frame Skip -> 0.
Make sure Graphics -> VSync is unchecked.
Make sure Sound -> PSG High Quality is unchecked.[9]
If the game predominantly uses the low resolution mode, uncheck Graphics -> Proper Aspect Ratio in low-resolution mode.
Check Tools -> AVI -> Clean AVI screen.
Uncheck Tools -> AVI -> Sync AVI with movie.
Open the ROM.
Pause emulation (Pause key).
Start movie playback.
Select Tools -> AVI -> Start AVI dump. Choose a file name (such as raw.avi), and choose a lossless RGB codec of your choice (see Common settings for suggestions). Click OK.
Unpause emulation. You are now capturing audio/video.
When you reach the desired end point, select Tools -> AVI -> Stop AVI dump.
I can not say the resulting AVI in my computer was amazing quality but it is much better than after uploading to youtube.
To compress it, I have used h264. To configure the codec, I have fallowed step by step this tutorial of youtube:
http://www.youtube.com/watch?feature=player_detailpage&v=QjTe42AXZQs
Also the fps reducing from 60 to 29.97, has some anoying effects, just to put an example, in min 13:08-13:09, Guy is supose to be walking around the edges of a fire but there is no fire in the video because the fire effect of the game is to show the fire 1 frame yes 1 frame no, and by removing half of the frames this fire is completely removed so Guy looks stupid just walking around for no reason.
Hope you can help me. Thanks
PS: I have read lots of the rules of the page (not all), hard for a non native english speaker (I'm Spanish). My apologies if I am breaking any rules of the page by linking my videos here, I didn't get this clear from the rules. Thanks again.
If you have questions about encoding of movies rather than how to TAS a game, I recommend you ask in the Encoder forum:
http://tasvideos.org/forum/viewforum.php?f=52
They can probably help you with flickering and how to fix it for youtube!