Block-based video codecs choose coding parameters by optimizing a rate-distortion trade-off. The conventional distortion measure, sum of squared errors (SSE), decomposes block-wise, which lets rate-distortion optimization (RDO) treat blocks independently. Full-reference image quality assessment (FR-IQA) metrics such as MS-SSIM or LPIPS often align better with human perception, but they cannot be used in-loop because they do not decompose block-wise and typically need the fully decoded image.
A new paper on Hugging Face addresses this gap by approximating a broad class of FR-IQA metrics with an input-dependent quadratic distortion (IDQD). The quadratic form matrix is derived from the Hessian of the metric evaluated at the source video, building on prior work in metric quadratization.
To make the distortion computable block-wise, the authors propose two Hessian approximations: keeping the block-diagonal, and keeping only the diagonal. Estimators for both require only matrix-vector products with the Hessian obtained via automatic differentiation.
Across five metrics on Kodak and CLIC in VVC, the proposed IDQD-RDO achieves 14.2-36.7% BD-rate savings under the target metric, with no decoder changes and 10-30% encoding complexity overhead.