Comparison of ACR Methods for 360° Video Quality Assessment Subject to Participants’ Experience with Immersive Media

Abstract

Watching 360 • videos on head-mounted displays (HMDs) has become more popular in recent years. As such, subjective quality assessment of 360 • videos on HMDs is needed to obtain a ground truth on the quality as perceived by users. Due to the lack of standardized methods, the absolute category rating (ACR) with hidden reference (ACR-HR) method for conventional videos has also been used for subjective quality assessment of 360 • videos on HMDs. A modified ACR (M-ACR) method tailored for assessing 360 • videos on HMDs has recently been proposed. In this paper, we compare the ACR-HR and M-ACR with hidden reference (M-ACR-HR) method regarding subjective quality assessment of 360 • videos on HMDs. The comparison is conducted subject to participants' experience with watching immersive media on HMDs. The findings include: 1) Average rating times to cast opinion scores is lower for the ACR-HR method irrespective of participants' experience, 2) Mean opinion scores are similar for both methods irrespective of participants' experience, 3) The M-ACR-HR method is more reliable, especially with experts and participants with no experience, 4) Discomfort is significantly lower for the ACR-HR method, especially, for participants with no experience.

ACR360: A Dataset on Subjective 360° Video Quality Assessment Using ACR Methods

“…In this section, we illustrate the recorded data for both the ACR and MACR method. A detailed comparison of the ACR and MACR methods are available in [35]. For the purpose of illustrating the use of the ACR360 dataset, we selected the frst four participants P1-P4 for the ACR method and participants P20-P23 for the MACR method.…”

On the Number of Subjects Needed for 360° Video Quality Experiments: An SOS Based Analysis

“…The ground truth in terms of opinion scores used for the SOS based approach is taken from the dataset [29], [30] and is described here to the extent needed for supporting reproducibility. Four 360 • video scenes with different contents and dynamics were selected from the publicly available VQA-ODV database [16], [31].…”

Comparison of Conditions for Omnidirectional Video with Spatial Audio in Terms of Subjective Quality and Impacts on Objective Metrics Resolving Power

“…Methodologies for 360°video quality experiments have recently been standardized in ITU-T Rec. P.919 [1], and previous works [2][3][4][5] focusing exclusively on the video component. Spatial audio, e.g., ambisonics, brings a better sense of immersion in content by allowing users to look in a specific direction and hear audio that reflects their head movements and positions.…”