A Comprehensive Performance Evaluation of Image Quality Assessment Algorithms

Abstract

Image quality assessment (IQA) algorithms aim to predict perceived image quality by human observers. Over the last two decades, a large amount of work has been carried out in the field. New algorithms are being developed at a rapid rate in different areas of IQA, but are often tested and compared with limited existing models using out-of-date test data. There is a significant gap when it comes to large-scale performance evaluation studies that include a wide variety of test data and competing algorithms. In this work we aim to fill this gap by carrying out the largest performance evaluation study so far. We test the performance of 43 full-reference (FR), seven fused FR (22 versions), and 14 no-reference (NR) methods on nine subject-rated IQA datasets, of which five contain singly distorted images and four contain multiply distorted content. We use a variety of performance evaluation and statistical significance testing criteria. Our findings not only point to the top performing FR and NR IQA methods, but also highlight the performance gap between them. In addition, we have also conducted a comparative study on FR fusion methods, and an important discovery is that rank aggregation based FR fusion is able to outperform not only other FR fusion approaches but also the top performing FR methods. It may be used to annotate IQA datasets as a possible alternative to subjective ratings, especially in situations where it is not possible to obtain human opinions, such as in the case of large-scale datasets composed of thousands or even millions of images.

Full Reference Image Quality Assessment Based on Visual Salience With Color Appearance and Gradient Similarity

“…In order to evaluate whether a model is able to predict the perception of human observers, the comparisons are made between the calculated scores using the proposed model and the values rated by the observers. Four commonly evaluation criteria for IQA model are employed: Spearman rank-order correlation coefficient (SROCC), Pearson linear correlation coefficient (PLCC), Kendall rank-order correlation coefficient (KROCC) and root mean squared error (RMSE) [3], [40].…”

“…Their accuracy, however, is not as good as their efficiency. Hence, numerous IQA models have been proposed with better performance based on HVS [3]. Generally, these better models characterize the structural information, luminance information, contrast information and color information in the spatial and frequency domains.…”

“…Furthermore, the results of statistical significance tests to evaluate the performances of the competing models are presented and illustrated in Fig.7, which is achieved by performing a series of hypothesis tests based on the prediction residuals of each model after nonlinear regression [3], [20], [34]. Specifically, the left-tailed F-test is employed to compare every two competing models.…”

“…IQA methods can generally be divided into two categories: human-based subjective assessment and algorithm-based objective assessment [2]. In the past few decades, multiple objective IQA models have been developed to evaluate image quality [3]. There are three main well-established frameworks according to the availability of a reference image: full reference (FR) [4]- [13], reduced reference (RR), and no reference (NR) [14] or blind IQA.…”

Objective Image Quality Measures for Disparity Maps Evaluation

“…Although there are a number of techniques for assessing the similarity between two images [18], simple pixel-based objective measures are generally used to assess the similarity between the ground-truth and the estimated disparity maps.…”