Comparison of subjective methods, with and without explicit reference, for quality assessment of 3D graphics

Abstract

Numerous methodologies for subjective quality assessment exist in the field of image processing. In particular, the Absolute Category Rating with Hidden Reference (ACR-HR) and the Double Stimulus Impairment Scale (DSIS) are considered two of the most prominent methods for assessing the visual quality of 2D images and videos. Are these methods valid/accurate to evaluate the perceived quality of 3D graphics data? Is the presence of an explicit reference necessary, due to the lack of human prior knowledge on 3D graphics data compared to natural images/videos? To answer these questions, we compare these two subjective methods (ACR-HR and DSIS) on a dataset of high-quality colored 3D models, impaired with various distortions. These subjective experiments were conducted in a virtual reality (VR) environment. Our results show differences in the performance of the methods depending on the 3D contents and the types of distortions. We show that DSIS outperforms ACR-HR in term of accuracy and points out a stable performance. Results also yield interesting conclusions on the importance of a reference for judging the quality of 3D graphics. We finally provide recommendations regarding the influence of the number of observers on the accuracy.

Visual Quality of 3D Meshes With Diffuse Colors in Virtual Reality: Subjective and Objective Evaluation

“…As stated in the introduction, datasets of human perceptual similarity judgments are of primary importance for understanding human behavior in evaluating perceived quality, as well as for training and benchmarking objective metrics. Many authors have conducted subjective quality assessment tests involving 3D meshes [2], [5], [8], [9], [10], [11], [12], [13], [14], [15], [16] or 3D point clouds [17], [18], [19], [20], [21]. They considered a variety of methods: Absolute Category Rating (ACR) [11], [13], [22], Double-Stimulus Impairment Scale (DSIS) [2], [8], [9], [18], [19], [21] and Pairwise Comparison (PC) [5], [14], [15], [23].…”

“…They considered a variety of methods: Absolute Category Rating (ACR) [11], [13], [22], Double-Stimulus Impairment Scale (DSIS) [2], [8], [9], [18], [19], [21] and Pairwise Comparison (PC) [5], [14], [15], [23]. Very recently, a study [16] attempted to compare these subjective methods and showed that, for the particular case of 3D graphical content, the DSIS method tends to produce more accurate results than ACR (i.e., MOS with smaller confidence intervals). Existing subjective experiments also considered different ways of presenting 3D content: static images [8], animated content without interaction (usually low-speed rotation) [5], [9], [10], [16], [19], [20], [21], [22], [23] or interactive content [2], [11], [13], [14], [15], [17], [18].…”

“…Very recently, a study [16] attempted to compare these subjective methods and showed that, for the particular case of 3D graphical content, the DSIS method tends to produce more accurate results than ACR (i.e., MOS with smaller confidence intervals). Existing subjective experiments also considered different ways of presenting 3D content: static images [8], animated content without interaction (usually low-speed rotation) [5], [9], [10], [16], [19], [20], [21], [22], [23] or interactive content [2], [11], [13], [14], [15], [17], [18]. We denote that only in [15], [16], the experiments were conducted in a VR environment.…”

“…Our dataset allows us to provide an initial investigation on the influence of movement and viewpoint on the quality evaluation of 3D content. As in [15], [16] we considered a VR context and we used a DSIS method with 24 observers per stimulus, as recommended in [16].…”

Comparison of ACR Methods for 360° Video Quality Assessment Subject to Participants’ Experience with Immersive Media

“…In [23], the ACR-HR and DSIS methods were compared to address the question if an explicit reference is needed in subjective experiments for 3D graphics when shown in a virtual environment on an HTC Vive Pro HMD. A motivation of this work was to reveal if these subjective methods are valid and accurate in view of participants' lack of prior knowledge of 3D graphics data.…”