See this page for info on generic video configuration settings.
Deblocking
Enable Deblocking: This feature enables the inloop deblocker for use during decoding.
Deblocking strength/threshold: Also known as alpha deblock and beta deblock, respectively. The two variables can range between -6 and 6. There is a good explanation of what these settings do on doom9.
Misc
Enable PSNR calculation: Checking this box will tell x264 to output PSNR (a quality metric) information into the log at the end of every job. It slightly slows down encoding.
Enable SSIM calculation: Similar to the PSNR calculation, SSIM is simply a different measure of frame quality (most people think it is more accurate). Again, it slightly slows encoding.
Number of threads: Controls the number of threads used by the encoder, to increase performance on multi-core systems. This will negligibly affect quality. The recommended value is 0, which will tell x264 to use (1.5 * number of cores) threads. Note that if you have enabled the MeGUI option 'automatically set number of threads', this will override the setting here.
fourCC: Sets the 4CC which is used as a codec identifier in AVI files).
AVC profiles: This setting is used to help you encode to a specific profile of AVC. Settings above the profile selected will not be settable. AVC Level: This setting specifies the level your encode is created to. Be aware it is only a nominal value written into the bitstream, and x264 does not pay any heed to this setting. MeGUI includes a level checker that will attempt to tell you when your options are not compliant with the chosen level, but some settings cannot be checked this way (for example, internal x264 buffer settings).
VBV Buffer Size: Maximum size of the VBV (Variable Bitrate Video) Buffer.
VBV Maximum Bitrate: Max local bit rate of VBV.
VBV Initial Size: The initial occupancy of the VBV buffer. 1.0 = full, 0.0 = empty.
Bitrate Variance: Also known as rate tolerance. Allowed deviation from the target bitrate or filesize. 1.0 = +/- 1%. This setting only applies to 1 pass ABR encodes.
Quantizer Compression: Occasionally seen as bitrate variability (not to be confused with above setting). Allowed variability of quantizer. 1.0 = pure CQ, 0.0 = Pure CBR. See this thread for more information.
Temp. Blur of est. Frame complexity: Level of temporal blurring in the quantizer curve before curve compression. Temporal blurring increases the consistency of the quantizer levels. Increasing this value moves the video closer to pure CQ. The value specifies the radius (in frames) of the gaussian blur applied. See this doom9 thread for more information.
'Temp. Blur of Quant after CC: Level of temporal blurring in the quantizer curve after curve compression. Increasing this value moves the video closer to CQ. Again, this setting is the radius (in frames) of the gaussian blur applied. See this doom9 thread for more information.
M.E.
Chroma ME: x264 by default searches both luma and chroma for motion information in the sub-pixel search. Disabling this setting will give a slight speed boost at the cost of quality.
ME Range: The maximum range (in pixels) that the motion estimator searches beyond the initial predictor. Does not affect Diamond or Hexagon beyond range 16.
Scene Change Sensitivity: The change required between frames for a scene change to be detected. Can range from 0 to 100, but note this is not a simple % figure. -1 means no scene changes will be detected. See this doom9 thread for more information about the option.
ME Algorithm: The Motion Estimation search algorithm used. Diamond is the fastest, exhaustive is per-pixel and extremely slow.
Subpixel refinement: Controls both subpixel estimation algorithms used and partition decision method. RDO (Rate Distortion Optimisation) enables RDO for I and P-frames. There's some more information in this doom9 post.
Misc
Keyframe Interval: The maximum interval between IDR-frames.
Min GOP size: The minimum interval between IDR-frames.
Noise reduction: Level of pre-processing noise reduction. 0 = disabled.
Encode Interlaced: Check this box to encode the input stream as interlaced. You should use this with interlaced input only. Using this option with progressive input (or vice versa) will produce broken output.
Quant Options
Trellis: Enables Trellis RDO either for final macroblock, or always. It requires CABAC.
Number of Reference Frames: Number of reference frames available for motion compensation.
Mixed Reference Frames:Allows references to be selected individually per partition.
CABAC: A method of encoding stream syntax which results in high compression.
No DCT Decimation: Disables DCT block decimation. Usually not required, but can help in some cases. See this doom9 post for more info.
No Fast P-Skip: Disables p-frame skip detection - slightly increases quality, particularly in low detail areas, such as dark walls and sky. There is some information about how exactly this setting works in this doom9 post.
Minimum Quantizer: Sets the minimum quantizer that x264 can encode with.
Maximum Quantizer: Sets the maximum quantizer that x264 can encode with.
Maximum Quantizer Delta: Set the maximum change in quantizer between frames.
Credits Quantizer: For use if you set the credit starting point in the preview window.
Factor Between I and P frame Quants: Boosts the bitrate of I frames compared to P frames by this amount.
Factor between P and B frame Quants: Reduces the bitrate of B frames compared to P frames by this amount.
Chroma QP Offset: Offset of chroma quantization values from luma.
Inter / Intra luma quantization deadzone: Also known as inter/intra deadzone. Specifies the level of detail to be discarded and not taken into account by the encoder. Lower values discard less information, and therefore hand the encoder a more complex stream. The main use of these settings is to set a detail threshold. Lower settings can help grain retention, while higher settings may reduce noise. Use of deadzone settings requires trellis to be turned off.
Adaptive Quantizers
Mode: AQ reallocates bits based biased towards lower detail sections that need more than what is normally specified. You can choose for this reallocation to end up averaging out to the same number of bits over the entire video (default), or over each individual frame. You can also completely disable AQ here.
Strength: Set the strength of AQ compensation. Lower values reduce the compensation strength.
Quantization matrix: Set a custom quantization matrix to be used. You must specify the custom matrix. The default matrix (none) is a flat matrix. x264 also includes the JVT matrix.
Custom Command Line: Add arbitrary text to the x264 commandline. This is added to the very end of the entire line.
Macroblock Options Enable and disable different macroblocks that can be used during the encoding process. The letter refers to the frame type the macroblock can be used in (either i, p or b), and the numbers refer to the partition size (either 8x8 or 4x4 pixels). Adaptive DCT allows an 8x8 transform in addition to the default 4x4 transform.
B-Frames:
Number of B-Frames: Sets the number of consecutive B-Frames (B-VOPs)
Adaptive B-Frames: Allows x264 to use a dynamic number of consecutive B-Frames
B-Pyramid: Allows use of B-Frames as references
RDO for B-Frames: Uses RDO algorithms on B-Frames (requires subpixel motion estimation set to level 6 or higher).
Weighted Bi-Prediction: When combined with multiple consecutive B-frames, this improves the prediction of fades.
Bidirectional ME: Motion estimation is optimised in both forwards and backwards temporal directions
B-Frame Mode: Motion vector prediction method for B-Frames. Auto selects the best method per-frame
B-Frame Bias: Modify the bias for use of Adaptive B-Frames. Higher values increase use, lower values decrease.
Along the bottom of the dialog there is a section called profiles. Profiles have already been created for the most common applications. Sharktooth has made these profiles available at doom9. The above page provides a very brief explanation of what each profile should be used for as well as a link to download the profiles. Note that you can also modify existing profiles and create your own.
Profiles dropdown: Lets you view all available profiles for x264. If you select one it will load over the top of your current settings.
Delete: Delete the currently selected profile.
New: Create a new profile.
Update: Save the current settings into the currently selected profile.
Load Defaults: Just load the dialog's defaults. These are the settings which try and be as close to x264 defaults as possible. They don't correspond to any profile.