Posted: 7/14/2011 7:59 PM
(Edited: 7/18/2011 9:04 PM)
Post subject: AVISynth colour conversion matrices
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
As [1840] SGB Pokémon: Blue Version "Gotta Catch 'Em All!" by p4wn3r & Mukki in 3:20:46.17 was being published, it was noted that one of the candidate streaming encodes had noticeably different colour relative to the original input. After some experimentation, I have reason to believe this is due to the use of the PC.601 colour conversion matrix in the most widely used AVISynth scripts we are using. The following is the methodology I used to deduce this. First, I created a simple (RGB) colour bars image using the following AVISynth one-liner, and used VirtualDub to capture one frame of the image: 
ColorBars().Trim(0,60).PointResize(1920,1440)
(Note that I'm using the large resolution to ensure that when I upload these to YouTube for casual viewing that it processes them to the best of its ability.) Here's the resulting file. I then losslessly (qp/crf 0) encoded five test files using x264:
  • One was using an mencoder chain using -sws 9 and -vf format=yv12,scale to create a raw yv12 input to x264.
  • The remaining four were done using the following AVISynth script as an input, with only the colourspace settings changing: 
    ColorBars().Trim(0,60).PointResize(1920,1440)
ConvertToYV24(chromaresample="point",matrix="xxx")
ConvertToYV12(chromaresample="point",matrix="xxx")
The four respectively used Rec601 (AVISynth's default), PC.601 (what we currently use), Rec709, and PC.709 as settings. In the latter two cases, I also specified --colorprim bt709 --transfer bt709 --colormatrix bt709 to x264. The results are here, or you can see them uploaded to YouTube: At a glance, it seems that YouTube (and the media players I'm using) are expecting Rec601 input for colour to be displayed properly. One of the best points of comparison that I can see is the set of three grey bars below the large red bar - the contrast is much more visible using Rec vs. PC. (The mencoder chain I'm using also seems to be outputting rec.264.) As such, I think the setting should be changed to get videos to display with colours as close as possible to the original input. (I should note that the header settings I passed to x264 appeared to have no effect on en-/decoding in the case of 709; neither does specifying --fullrange, as far as I can tell.) EDIT: Apparently, after further processing at this resolution, Rec709 turns out to appear correct for YouTube purposes. It would probably be best that Rec709 be used for YouTube large resolution encodes, Rec601 used for downloadable encodes (until we can test for greater player compatibility), and PC.601 for 512s.
Posted: 7/14/2011 9:00 PM
Quote
Toothache
Active player (279)
Joined: 4/30/2009
Posts: 791
Lucky TAS of 2013
From my understanding then, the script needs modifying to use Rec.601 instead of PC.601. What changes need to be made on the Flygon/Brandon script in order for this?
Posted: 7/14/2011 11:06 PM
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
I've made the requisite adjustment to the script.
Posted: 7/15/2011 11:38 PM
(Edited: 7/15/2011 11:42 PM)
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
This is a bit wrong. Flash Player actually does read the color information correctly. So our encodes sent to archive to be streamed in JWPlayer has the color CORRECT. The fact that most media player doesn't read the flag, is well, the player's fault. The reason why YouTube has the color wrong despite using flash player is because YouTube reconverts. Edit: If you want to see an example, you can checkout the GoldenEye encode and compare to the Archive streaming. The Archive streaming has the correct brightness. Technically so does the downloaded video, only if your player is setup correctly.
Posted: 7/15/2011 11:40 PM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
If you are going to cap it onto TV colors anyway, you should remove the --fullrange parameter in x264 command line also.
Posted: 7/16/2011 12:17 AM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
Another note to add is, for YouTube, I don't know if YouTube assumes a different matrix for SD resolutions vs HD resolutions. Usually, for SD, the 601 profile is used, while for HD, it is the 709. While the majority of encodes going to YouTube is HD nowadays, this may not be a problem. [offtopic]Meh to HD YouTube! =p[/offtopic] Edit: Grunt pointed out I switch which matrix was used for SD/HD. Corrected.
Posted: 7/16/2011 12:31 AM
(Edited: 7/16/2011 12:49 AM)
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
Aktan wrote:
Flash Player actually does read the color information correctly. So our encodes sent to archive to be streamed in JWPlayer has the color CORRECT.
This may be true of Flash Player, but as we've both noted on IRC, most media players don't handle reading the flag correctly. So, it should be possible to maintain the flag for encodes strictly intended for streaming purposes where we know they'll play back correctly (read: 512s), but at the moment it's probably a bad idea to use it for other encodes until there's more media player support for it. EDIT: VLC apparently interprets the flag correctly (assuming the right colour space / fullrange combination is used). mplayer (and mplayer2) do not, and neither do most of the other media players I have on hand here. Testing these with more media players would be helpful.
Posted: 7/16/2011 12:34 AM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
Like I said in the IRC channel, it's a bit sad that we need to make compromises because we use too cutting edge things. This problem will also make 4:4:4 not really completely full color still.
Posted: 7/18/2011 8:52 PM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
So I've done some testing as to what matrix YouTube expects for "Original" mode. Here are the results: Original: http://img823.imageshack.us/img823/8608/originalyw.png Rec601: http://img845.imageshack.us/img845/7549/rec601.png http://www.youtube.com/watch?v=UhD-DbWDOPg PC.601: http://img21.imageshack.us/img21/5349/pc601.png http://www.youtube.com/watch?v=sDQliswj7gc Rec709: http://img607.imageshack.us/img607/6866/rec709.png http://www.youtube.com/watch?v=JVisE3hnIqI PC.709: http://img98.imageshack.us/img98/1596/pc709.png http://www.youtube.com/watch?v=o06saTi3e5k Note: The screenshots besides original were from flash player. I'd say YouTube expects Rec709 as that's the only one with the color correct. Next I'll do a test for resolutions 640x480 and under. Basically SD. I'll probably do a screenshot of 320x240.
Posted: 8/11/2011 8:56 PM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
Some more tests! This time on SD. The resolution is 640x480: Original: http://img13.imageshack.us/img13/3921/original480.png Rec601: http://img580.imageshack.us/img580/2633/rec601480.png http://www.youtube.com/watch?v=AV_yQw8uvaU PC.601: http://img560.imageshack.us/img560/4316/pc601480.png http://www.youtube.com/watch?v=5aiMzeY_QLg Rec709: http://img594.imageshack.us/img594/4134/rec709480.png http://www.youtube.com/watch?v=pRThhWje_4U PC.709: http://img607.imageshack.us/img607/4858/pc709480.png http://www.youtube.com/watch?v=43F21GwDgHs Note: The screenshots besides original were from flash player at my desktop resolution of 1600x1200. I'd say YouTube expects Rec601 for SD resolution as that is the only one with the correct color and brightness. On a side note, I noticed loss of detail in the YouTube screenshot. I don't think it's due to the resizing to 1600x1200, but more like YouTube processing... Next is 320x240!
Posted: 8/11/2011 10:02 PM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
Finally SD 320x240: Original: http://img88.imageshack.us/img88/3707/original240.png Rec601: http://img825.imageshack.us/img825/7858/rec601240.png http://www.youtube.com/watch?v=UUMazn5CA1Y PC.601: http://img833.imageshack.us/img833/9880/pc601240.png http://www.youtube.com/watch?v=Ru9Z6dgV21I Rec709: http://img705.imageshack.us/img705/3793/rec709240.png http://www.youtube.com/watch?v=rKdEzZzCZE0 PC.709: http://img593.imageshack.us/img593/5791/pc709240.png http://www.youtube.com/watch?v=wL8WhkBE-oE Note: The screenshots besides original were from flash player at my desktop resolution of 1600x1200. Again I'd say YouTube expects Rec601 for SD resolution as that is the only one with the correct color and brightness. On a side note, again I noticed loss of detail in the YouTube screenshot. I don't think it's due to the resizing to 1600x1200, but more like YouTube processing...
Posted: 1/19/2012 10:55 PM
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
More tests of youtube upload! I'm working on gba encodes, which get "HD" processing at 1920x1280. My goal is to give an appropriate matrix for them, so the test uploads are at 1920x1280. Step 1: The test pattern. Redirect output to pattern.y4m; this takes about 1GB of HDD space. http://pastebin.com/YDjjXPST Step 2: encode (8 bit x264) 
x264                                                                    --qp 0 -o lossless_undef.h264 pattern.y4m
x264 --colorprim smpte170m --transfer smpte170m --colormatrix smpte170m --qp 0 -o lossless_bt601.h264 pattern.y4m
x264 --colorprim bt709     --transfer bt709     --colormatrix bt709     --qp 0 -o lossless_bt709.h264 pattern.y4m
Step 3: Verify that x264 has not touched the colors AT ALL and all three samples are identical except for VUI 
ffmpeg -i pattern.y4m         -f rawvideo md5:
ffmpeg -i lossless_undef.h264 -f rawvideo md5:
ffmpeg -i lossless_bt601.h264 -f rawvideo md5:
ffmpeg -i lossless_bt709.h264 -f rawvideo md5:
I got md5=4d0b4a461f189366e7c4152b8257b70c for all four cases Step 4: mux. h264 in mkv is the same way i've been muxing for yt upload 
mkvmerge -o undef.mkv --default-duration 0:30fps lossless_undef.h264
mkvmerge -o bt601.mkv --default-duration 0:30fps lossless_bt601.h264
mkvmerge -o bt709.mkv --default-duration 0:30fps lossless_bt709.h264
Step 5: upload. http://www.youtube.com/watch?v=I0A4wNhAAUg http://www.youtube.com/watch?v=Ksn55lGtpx0 http://www.youtube.com/watch?v=HRWgLlKnrcQ Step 6: Avisynth script to compare to. 
# requires ffms2.dll

FFVideoSource (source="undef.mkv", cache=false)

#convert to RGB internally; hopefully player leaves it unmolested
clip601 = ConvertToRGB24 (matrix="rec601").Subtitle ("601",
\ y=640, align=5, first_frame=0, last_frame=299, size=96.0, text_color=$20FFFFFF, halo_color=$20000000)

clip709 = ConvertToRGB24 (matrix="rec709").Subtitle ("709",
\ y=640, align=5, first_frame=0, last_frame=299, size=96.0, text_color=$20FFFFFF, halo_color=$20000000)

clip601 + clip709
Step 7: Watch the avisynth script in your favorite video player and the youtube videos in a youtube window. Conclusion: All 3 uploaded videos are understood by youtube as rec601 at resolutions 240p, 360p, 480p, 720p, and 1080p, and are understood by youtube as rec709 at resolution "original". The VUI hints in the h264 stream are completely ignored in all cases.
Posted: 1/19/2012 11:35 PM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
Great info, thanks natt! So apaprently I should have tested 720p/1080p also to see that it is Rec.601. Edit: Question: The source is in what matrix?
Posted: 1/20/2012 12:20 AM
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
Aktan wrote:
Great info, thanks natt! So apaprently I should have tested 720p/1080p also to see that it is Rec.601. Edit: Question: The source is in what matrix?
y4m is a simple format which stores raw yuv data. the "original test source" in this case IS yuv data, with no matrix or interpretation given. the three different encodes and uploads correspond to different sets of VUI data, but the original stream is exactly the same. Edit: Ilari brought up another issue in IRC: what if different flash players play it differently? My test was done on Windows 7 64bit with firefox 32bit and flash player 32bit 11.1.102.55. I got out my laptop, which ruins ubuntu 11.04 32bit with firefox 32bit and flash player 32bit 11.1.102.55, and it played the video DIFFERENTLY. It looks like one thing at 240p and a different thing at 360p, 480p, 720p, 1080p, and original.
Posted: 1/20/2012 9:31 PM
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
This topic was discussed in more detail yesterday on IRC chat, so I'm going to sum that up for anyone who should happen to read this but wasn't there. Flash seems to behave differently depending on a number of factors. Different video card (in the case where hardware accelerations are used), different "resolution" selection on the youtube video (can change the format and decoder flash uses internally), even hitting the fullscreen button (can switch to a different accelerated output path) can cause the color interpretation to change. Not to mention mobile devices, or the new "HTML5 webM" technology that's on the way. I can only say the following things for sure: 1) Don't send "PC level" YUV data to youtube. Always use "TV level" (16-235). 2) Sending RGB data to youtube doesn't hurt, but it doesn't help either; even if the data is transformed once RGB->YUV internally by youtube's transcoder in a predictable way, the different flash players can still handle the resulting YUV differently. Beyond that, I can't say whether 601 or 709 is appropriate for HD uploads, since different youtube players will handle it differently; there is no right answer.