Compare x264 encoding with 8 bit per channel RGB to 10 bit per channel YUV 4:4:4.
For lossless (--qp 0), RGB generally makes smaller files from sources that were originally RGB.
For lossy, RGB doesn't stand up to YUV.
A short example test:
encoding commands (here x264_10 is a 10 bit executable)
x264 --bitrate 500 --pass 1 --preset placebo --rc-lookahead 120 --keyint 400 --me umh --subme 10 --merange 32 --output-csp rgb -o foo.mp4 testin.avi
x264 --bitrate 500 --pass 2 --preset placebo --rc-lookahead 120 --keyint 400 --me umh --subme 10 --merange 32 --output-csp rgb -o newtest_rgb.mp4 testin.avi
x264_10 --bitrate 500 --pass 1 --preset placebo --rc-lookahead 120 --keyint 400 --me umh --subme 10 --merange 32 --output-csp i444 --range pc -o foo2.mp4 testin.avi
x264_10 --bitrate 500 --pass 2 --preset placebo --rc-lookahead 120 --keyint 400 --me umh --subme 10 --merange 32 --output-csp i444 --range pc -o newtest_10bit444.mp4 testin.avi
The original source is two minutes of GBA footage. The output files are here:
http://www.mediafire.com/?vbjvitb1j4io22r
You can see by watching the two directly, that RGB doesn't even come close to YUV in image quality at the same bitrate.
Note: If the colors&ranges in the two videos look different, it's your player's fault. Both encodes are properly set as full range.
If anyone has any comments or criticisms on this test, I would love to hear it.