Review of subjective quality assessment methodologies and standards for compressed images evaluation

Abstract

Acquiring and exchanging a large number of pictures has become nowadays a common practice. Therefore, new image compression solutions to optimize the storage resources are in constant demand. In this context, it is essential to have a solid methodology to evaluate the performance of compression techniques. Such performance is usually measured through objective quality metrics, which are fast and inexpensive but not always reliable. However, performance is best assessed through subjective image quality assessment experiments, which are expensive and time-consuming, but reliable as based on the subjective opinion of a large number of subjects. These experiments are usually conducted in a controlled laboratory environment, with high-quality monitors and controlled lighting conditions. Recently, encouraged by the COVID-19 pandemic, crowdsourcing-based subjective image quality experiments are gaining popularity, and have demonstrated to be a faster and cheaper alternative to traditional approaches in subjective quality assessment. In this paper, different methodologies for subjective image quality assessment experiments are examined, including a review of the released standards as well as a list of publicly available tools. Moreover, the analysis is extended to novel plenoptic imaging techniques, i.e. point clouds and light fields, and to visually lossless quality assessment approaches. Authors hope that this work will help researchers interested in conducting subjective experiments for assessing the quality of compressed media, to better select the appropriate methodology for their use cases.

Subjective Assessment of Objective Image Quality Metrics Range Guaranteeing Visually Lossless Compression

“…Recently, high-performance image compression methods have become capable of reconstructing lossless compressed images. Further, with the advancements in storage devices and visual sensor networks, the storage and transportation of a huge amount of data is not a big deal [ 13 ]. This leads to a demand for effective image compression algorithms that can provide lossless reconstruction of image data.…”

“…Further, subjective methods are dependent on the physical conditions and emotions of the viewers, which makes them impractical in real-life applications. However, subjective methods of evaluation are considered more reliable and robust because they mainly rely on the opinions of human subjects, who represent the ultimate users of digital media applications [ 13 ].…”

“…In the current era, the availability of advanced image-capturing and display devices has increased the interest of researchers to design lossless image compression techniques [ 13 ]. Human eyes cannot perceive the very tiny artifacts that appear in compressed images when reconstructed up to a specific compression level.…”

Recent progress in image denoising: A training strategy perspective

“…We employed the double stimulus categorical rating [135] to evaluate the quality of these images. According to the double stimulus impairment scale (DSIS) [136], five levels are set to rate image quality, which are "very annoying," "annoying," "slightly annoying," "perceptible but not annoying" and "imperceptible." The corresponding scores of these five levels are set to 1, 2, 3, 4 and 5.…”

A crowdsourced implementation of ITU-T P.910

“…A recent review of subjective quality assessment tools is given in [12]. P.910 [1] provides a general subjective video quality assessment standard for multimedia applications.…”