Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
What command do you use to make those encodes?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
I've set up a tech demo site https://rgbscaler.com which is capable of playing back video in web browsers with nearest neighbor scaling. This is significant because there's no straightforward way to do this with just a video element and css like there is for images.
Looks magnificent!
TiKevin83 wrote:
If an original resolution lossless encode for a TAS was available with 444 chroma and no PAR correction, it could be downloaded and played on rgbscaler.com with the PAR entered there, essentially getting the same quality as a custom encode at the user's window/screen resolution would have.
Why not watching it directly in streaming?
TiKevin83 wrote:
Some drawbacks: the source video has to be in a web compatible format like av1 + opus in the webm container, I'm not sure how fast the canvas can keep up if you try scaling beyond 1080/1440 (I experienced noticeable frame skipping at 2880p), and the demo doesn't solve the issue of baked subtitles scaling badly.
Ambassador, Moderator, Site Developer, Player
(155)
Joined: 3/17/2018
Posts: 358
Location: Holland, MI
ffmpeg -i source.avi -c:v libaom-av1 -crf 0 -c:a libopus -b:a 320k -pix_fmt yuv444p output.webm
Listed in the readme on the source in gitlab:
https://gitlab.com/TiKevin83/rgbscaler
In theory it should also work with eg h264 lossless 444 in the mp4 container which is way faster to encode than av1.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
The goal is having a YT alternative that's easy to use and high in quality. Archive can only stream 4:2:0 h264 inside mp4, and from my tests, it's not perfect at seeking, and may even be too slow with playback. Honestly 4:2:0 makes this option look shittier than on YT.
I tried out the DASH format today. Even without the .mpd file, dash stream loads much faster on Archive than regular mp4. So I wonder if your tool will be able to support dash mp4+mpd taken from another site like Archive. That would probably solve a bunch of problems.
Overall, due to problems with 1x subtitles, I'm not sure if we want to use the soft-scaling functionality to resize 1x encodes. But I like how much better it looks compared to the best format Archive can stream.
And yeah if it can only work with files I have locally, it hardly compares to just watching it on YT, because as the poll shows, almost nobody watches SD encodes anymore. They are slow to download and don't really look that good if chroma is subsampled (at least below 4:2:2).
One of the reasons we're switching to 480p is to fully control resizing. Some games have weirdly wide or narrow resolutions that need to be normalized for the TV look. Some games have scrolling pixelated backgrounds that look jumpy if we don't upscale them evenly or with some amount of blur. Could it be possible to stack different scale filters with this player? Like point to 2x and then lanczos to 4:3?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Ambassador, Moderator, Site Developer, Player
(155)
Joined: 3/17/2018
Posts: 358
Location: Holland, MI
It should definitely be possible to stack filters on the canvas with webGL, just a matter of porting things. I agree if people aren't using the downloadable encodes this doesn't help much, streaming video is just too convenient for the slight quality bump/bandwidth savings.
Ambassador, Moderator, Site Developer, Player
(155)
Joined: 3/17/2018
Posts: 358
Location: Holland, MI
https://rgbscaler.com/
With Inverted's help I was able to implement OBS's "Area" effect as the upscaling algorithm for the player. The area effect helps preserve the shape of upscaled objects without sacrificing edge sharpness when scaling by non-integer factors or introducing PAR correction. The below linked image shows the utility of the concept.
https://obsproject.com/forum/attachments/link_scaling-png.31965/
I'm looking into next whether the player can stream from resources on archive.org. It should actually be possible to use this for cross domain resources if they send the correct CORS response, so I might have options to play back streaming video if from sites that supply the right CORS info.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
I'm getting this with latest firefox:
And I can't open it in Opera or Chrome at all.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Ambassador, Moderator, Site Developer, Player
(155)
Joined: 3/17/2018
Posts: 358
Location: Holland, MI
You have to make extra requests to validate CORS to use the video data on a canvas but it should be possible if the third party passes the headers to allow it.
The new shader requires OpenGL ES 3.0 hardware support, I should probably implement a fallback to nearest neighbor, it's not that new of a spec but it's still kinda new.
The new shader requires OpenGL ES 3.0 hardware support, I should probably implement a fallback to nearest neighbor, it's not that new of a spec but it's still kinda new.
Isn't using a 3D graphics library a bit overkill? Wouldn't be possible to implement the area resize by simply making an algorithm that checks if a pixel falls in between, and perform alpha blending for these pixels only?
Note that my concern is mainly about performance. Would my idea lead to higher CPU usage, or even frame drops on older machines?
Ambassador, Moderator, Site Developer, Player
(155)
Joined: 3/17/2018
Posts: 358
Location: Holland, MI
The technical details are a bit complicated but in short the Integer modes can be done using only 2D canvas, however to support the Max modes that guarantee filling the window/screen the 3D webgl canvas is required to use gpu shaders (doing blending on cpu side would indeed be too cpu intensive). The blending also accounts for video sRGB colorspace which is why it needs the ES 3.0 hardware level, older webgl spec doesn't support using internal texture formats in sRGB.
Ambassador, Moderator, Site Developer, Player
(155)
Joined: 3/17/2018
Posts: 358
Location: Holland, MI
https://rgbscaler.com/?videoName=4490M.mp4
I did a custom encode of SM64 1-key for rgbscaler to demonstrate the capability to stream N64 sources losslessly at only 4.8 mbps in this case. The player for this will currently only work in chrome due to the reliance on 444 10 bit support.
Some more examples:
DragsterSMB1
I also implemented a system to link back to tasvideos for these custom encodes to maintain attribution.
I have a fallback system in the works to do Nearest Neighbor on Firefox when either the video desired is cross-domain or when the user's system doesn't support webgl2 (feos was able to confirm a prototype worked for this), but I need to improve my process for making the required AV1 encodes as the libaom-av1 encoder in ffmpeg is too slow and the much faster libsvtav1 intel encoder doesn't have 444 support enabled by default.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
For those vids my browser says the files are corrupted.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
I'm looking into next whether the player can stream from resources on archive.org. It should actually be possible to use this for cross domain resources if they send the correct CORS response, so I might have options to play back streaming video if from sites that supply the right CORS info.
I have found that archive.org sends CORS responses for video files if you request from cors.archive.org and use a /cors/ path prefix.
Start from the item's details page:
I don't remember how I figured this out. I cannot now find where it is documented. Strangely, in my experience, for audio files, you get access-control-allow-origin: * even with a normal archive.org/download URL; it's only video files that need cors.archive.org/cors.
Ambassador, Moderator, Site Developer, Player
(155)
Joined: 3/17/2018
Posts: 358
Location: Holland, MI
Woah thanks a ton, that seems to be working when putting the cors URL into the URL input on rgbscaler! I have a couple more adjustments to make around that now and I've found some new things to get my custom encodes pre aspect-ratio correction working on more platforms.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
For the record, a lot of subtitle problems with A2600 and other tiny resolution consoles can be helped by simply prescaling the needed dimension by some integer factor. For example 2x for A2600 width. If we're going back to 1x encodes in the future for whatever reason, this workaround should be used to not cause subtitle placement hell.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
And I can't open it in Opera or Chrome at all.