Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies

Abstract

Abstract-We introduce a new video quality database that models video distortions in heavily-trafficked wireless networks and that contains measurements of human subjective impressions of the quality of videos. The new LIVE Mobile Video Quality Assessment (VQA) database consists of 200 distorted videos created from 10 RAW HD reference videos, obtained using a RED ONE digital cinematographic camera. While the LIVE Mobile VQA database includes distortions that have been previously studied such as compression and wireless packet-loss, it also incorporates dynamically varying distortions that change as a function of time, such as frame-freezes and temporally varying compression rates. In this article, we describe the construction of the database and detail the human study that was performed on mobile phones and tablets in order to gauge the human perception of quality on mobile devices. The subjective study portion of the database includes both the differential mean opinion scores (DMOS) computed from the ratings that the subjects provided at the end of each video clip, as well as the continuous temporal scores that the subjects recorded as they viewed the video. The study involved over 50 subjects and resulted in 5,300 summary subjective scores and time-sampled subjective traces of quality. In the behavioral portion of the article we analyze human opinion using statistical techniques, and also study a variety of models of temporal pooling that may reflect strategies that the subjects used to make the final decision on video quality. Further, we compare the quality ratings obtained from the tablet and the mobile phone studies in order to study the impact of these different display modes on quality. We also evaluate several objective image and video quality assessment (IQA/VQA) algorithms with regards to their efficacy in predicting visual quality. A detailed correlation analysis and statistical hypothesis testing is carried out. Our general conclusion is that existing VQA algorithms are not well-equipped to handle distortions that vary over time. The LIVE Mobile VQA database, along with the subject DMOS and the continuous temporal scores is being made available to researchers in the field of VQA at no cost in order to further research in the area of video quality assessment. Index Terms-Mobile video quality, objective algorithm evaluations, subjective quality, video quality assessment, video quality database.

Quality of Experience of adaptive video streaming: Investigation in service parameters and subjective quality assessment methodology

“…P.910. After presentation of each PVS, the subjects were asked to evaluate the PVS by answering two questions: the overall quality of the PVS (rating on the five graded ACR scale Bad{\), Poor[2\ Fair(3), Good (4) and Excellent (5)), and if they perceived any change in the quality (options: Increasing, No change, Decreasing).…”

“…About the switching frequency, results presented in [2][3][4][5][6] show that frequency of the adaptation should be kept as low as possible. In the same time, the study presented in [3] indicates that if the duration spent on the high quality level is sufficiently long, higher switching frequencies do not significantly degrade the QoE.…”

“…Considering the switching amplitude, most of the previous studies [3][4][5][6][7] conclude that gradual multiple variations are preferred over rapid variations. Nevertheless, as highlighted in [4], this conclusion may not be applied to the scenarios where the quality levels' difference is very small.…”

A novel methodology for mapping objective video quality metrics to the subjective MOS scale

“…The LIVE Mobile Video Quality Assessment database [24], consists of 10 progressive reference sequences in the raw YUV 4:2:0 format, each of them having a HD 720p (1280×720 pixels) resolution, a 30 fps framerate, and a 15 seconds duration. The video quality subjective data corresponding to 40 test sequences compressed using the H.264 codec at bitrates between 0.7 Mbps to 6 Mbps, was used for evaluating the performance of various objective metrics.…”

“…The difference scores for the ECVQ and EVVQ databases were computed as in (5) and could take values between 0 to 100, with 100 being the highest quality [23]. As opposed, the difference scores for the LIVE Mobile database were computed as in (6) and could range between 0 and 5, with 0 being the highest quality [24].…”