HD Video Encoding Tutorial: Optimizing h.264/AVC output settings for YouTubeHD using Sony Vegas Pro 9.0
In this tutorial, I will be showing how to optimize your HD video output for uploading to YouTube, or any other video hosting. And by optimize, I mean getting the best quality out of the smallest file size possible. Sure, anyone can dump a 20 Mbps h.264 video on YouTube and get good results. But seriously, why upload a 500 MB file when you can get the same end quality with just 100+ MB, right?
And although I use Sony Vegas 9, the same settings can be applied on your video editing/encoding software of choice, given the settings are customizable.
As examples, I rendered two(2) 15-second, 1080p videos to show the difference (or lack of) between my optimized settings and a 20 Mbps variable bit rate (VBR) h.264/AVC/MP4 video.
Let’s begin.

The Settings

Assuming you’re done editing your video and you’re ready to render your output, first check to see your project settings.
Here I set the project properties for 1080p progressive video. Note that setting the pixel format to 32-bit floating point (full range) is important to get accurate color representation. Using 8-bit usually results in a darker output, which leads to further darkening when uploaded and played on YouTube.
After checking your project properties, go to ‘File’ on the menu, then select ‘Render As’ to select and set your final output settings.
Select ‘Mainconcept AVC/AAC’ (which is the same as h.264), then click on ‘Custom…’ to customize the settings.
Select the ‘Main’ Profile, as well as your desired frame rate. The frame rate setting would depend on your source video. I used a video captured with Fraps at 29.970 fps so here I selected 29.970 fps. And of course, I selected progressive scan, hence the ‘p’ in 1080p, lol.

I set the reference frames to 3 since the video is mostly full motion, meaning there is little benefit in using lots of reference frames, which will only greatly lengthen the encoding time (though Sony Vegas may sometimes change this automatically). If your source video is animation, however, (or any video with repetitive motion) using more reference frames may help reduce your output file size and/or improve quality.

And as seen on the image, I set my average bit rate to 3.8 Mbps (which I will later explain why). The maximum 135 Mbps is just a ceiling I set based on the maximum Main profile bit rate for level 5 AVC/h.264. It doesn’t have to be this high, but it doesn’t really impact the output file size so I set it to that value. You can adjust this if you want, but don’t set this too close to the average value, as the encoder will limit the bit rate in a certain scene that may require a higher value.

Above I set the sample rate to 48 KHz and the bit rate to 128 Kbps. Later, I will also explain why.

Project settings, set to Best. Simple as that. lol.

Then simply proceed with your render and you’re done!

The above settings should be all you need to render your video with no visible loss in quality compared to a much higher bit rate output. But if you’re interested in learning more about the reasons behind the chosen settings, or if you simply wish a preview of the results, then read on. 😀

The Initial Output

As I’ve mentioned, I rendered two videos. One, using my optimized settings,  resulted in a 7 MB file, while the other, which is basically the same settings except for the average bps set to 20 Mbps, resulted in a 19.1 MB file.

Here is a side by side comparison of screen captures that I did on a specific frame.

The full-sized frames can be seen here:

3.8 Mbps VBR

20 Mbps VBR

YouTube HD

But of course, the real test is on YouTube itself. As can be seen when comparing both clips here:
3.8 Mbps Source uploaded – http://www.youtube.com/watch?v=nUmkw9d48VA
20 Mbps Source uploaded – http://www.youtube.com/watch?v=DWz-f7DC1H4
Wondering how is that possible?
The answer is simple. Since YouTube resamples all videos uploaded, there comes a point where no matter how high your original video quality is, it always gets re-encoded to a certain level. That is why the sample high bit rate source becomes indistinguishable from a lower bit rate source once it gets uploaded to YouTube.
Taking a Closer Look
I downloaded the 1080p videos back from YouTube to see exactly what happened when the videos were re-encoded. I took still frames from the videos for comparison and also used MediaInfo to analyze the video properties.
Here’s a side by side comparison of the still frames from the downloaded YouTube videos.
The left is from the 3.8 Mbps source video, while the right is from the 20 Mbps source. It can be seen quite clearly that the re-encoding process after uploading the videos to YouTube pretty much leveled the quality between the two.
The full sized frames can be seen here:

From the 3.8 Mbps source video

From the 20 Mbps source video

I also loaded the videos on MediaInfo to see the encoding details. Here’s a side by side comparison:
Video Comparison
Left: 3.8 Mbps Source, Right: 20 Mbps Source
Audio Comparison
Left: 3.8 Mbps Source, Right: 20 Mbps Source
The examples I provided clearly show my basis for selecting the settings for both Video and Audio. I also did a lot of tests with different 1080p videos, and the videos re-encoded by YouTube would always range from 3.5 to 3.8 Mbps; the audio averages out at 125 Kbps. The tests also show that YouTube produces the same level of output quality when re-encoding videos, despite a big difference in source video bit rates. And with no visible difference in resulting quality between an optimized bit rate source and a high bit rate source, there really isn’t any reason to go overboard with the bit rates on your videos for uploading, is there? 🙂


Latest Posts