Comparison of Subjective Methods for Quality Assessment of 3D Graphics in Virtual Reality

Abstract

Numerous methodologies for subjective quality assessment exist in the field of image processing. In particular, the Absolute Category Rating with Hidden Reference (ACR-HR), the Double Stimulus Impairment Scale (DSIS), and the Subjective Assessment Methodology for Video Quality (SAMVIQ) are considered three of the most prominent methods for assessing the visual quality of 2D images and videos. Are these methods valid/accurate to evaluate the perceived quality of 3D graphics data? Is the presence of an explicit reference necessary, due to the lack of human prior knowledge on 3D graphics data compared to natural images/videos? To answer these questions, we compare these three subjective methods (ACR-HR, DSIS, and SAMVIQ) on a dataset of high-quality colored 3D models, impaired with various distortions. These subjective experiments were conducted in a virtual reality environment. Our results show differences in the performance of the methods depending on the 3D contents and the types of distortions. We show that DSIS and SAMVIQ outperform ACR-HR in terms of accuracy and point out a stable performance. In regard to the time-effort, DSIS achieves the highest accuracy in the shortest assessment time. Results also yield interesting conclusions on the importance of a reference for judging the quality of 3D graphics. We finally provide recommendations regarding the influence of the number of observers on the accuracy.

Exploring Crowdsourcing for Subjective Quality Assessment of 3D Graphics

“…A crowdsourcing (CS) experiment has been carried out in which subjects were recruited to assess the quality of an animated 3D graphics dataset. Since our goal is to compare the results of this experiment to previously collected Lab results [14], we replicated the lab experiment as much as possible, using the same instructions, dataset and subjective evaluation methodology. This section provides details on the CS experiment.…”

“…The 3D graphic stimuli used in this study were previously used in the laboratory based subjective experiment, reported in [14]. This dataset contains 80 animated meshes with diffuse color information.…”

“…Each distortion was applied with 4 different strengths adjusted manually in order to cover the whole range of visual quality from imperceptible to high levels of impairment. The source models as well as examples of distortions are illustrated in [14].…”

“…In order to be able to compare the results of the CS and lab studies, we opted for the same setup as implemented in the lab experiment [14]: the stimuli were animated with a slow rotation (of 15 • around the vertical axis in clockwise and then in counterclockwise directions) and displayed in a neutral virtual room (light gray walls) under the same viewpoints as those used in the lab experiment. Their material type complied with the lambertian reflectance model, and they were visualized without shadows and under a directional light.…”

“…Thus, we conducted a CS experiment based on the Double Stimulus Impairment Scale (DSIS) method, which is, to the best of our knowledge, one of its first uses in CS studies. We used a dataset of animated 3D models [14]. The results of this experiment were compared to those obtained from a previous lab experiment conducted in a Virtual Reality (VR) environment.…”