Challenges in crowd-based video quality assessment

Keimel, Christian; Habigt, Julian; Diepold, Klaus

doi:10.1109/qomex.2012.6263866

Cited by 25 publications

(22 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The migration to crowdsourcing invokes some conceptual challenges on how to assess QoE and how to design the user tests [29]. In laboratory studies user tests may take up to 90 minutes [30] which allow, for example, to investigate memory effects [31].…”

Section: B Conceptual Challenges For Qoe Crowdtestingmentioning

confidence: 99%

Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing

Hoßfeld

Keimel

Hirth

et al. 2014

IEEE Trans. Multimedia

Self Cite

254

177

View full text Add to dashboard Cite

Quality of Experience (QoE) in multimedia applications is closely linked to the end users' perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.

show abstract

Section: B Conceptual Challenges For Qoe Crowdtestingmentioning

confidence: 99%

Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing

Hoßfeld

Keimel

Hirth

et al. 2014

IEEE Trans. Multimedia

Self Cite

254

177

View full text Add to dashboard Cite

show abstract

“…Also, in Table 2 the row "λ" depicts the λ values for each model, used in Eq. (2). From the provided results of these tables we can see that LASSO is a sparser approach compared to OLS.…”

mentioning

confidence: 79%

A no-reference bitstream-based perceptual model for video quality estimation of videos affected by coding artifacts and packet losses

et al. 2015

View full text Add to dashboard Cite

In this work, we propose a No-Reference (NR) bitstream-based model for predicting the quality of H.264/AVC video sequences, affected by both compression artifacts and transmission impairments. The proposed model is based on a feature extraction procedure, where a large number of features are calculated from the packet-loss impaired bitstream. Many of the features are firstly proposed in this work, and the specific set of the features as a whole is applied for the first time for making NR video quality predictions. All feature observations are taken as input to the Least Absolute Shrinkage and Selection Operator (LASSO) regression method. LASSO indicates the most important features, and using only them, it is possible to estimate the Mean Opinion Score (MOS) with high accuracy. Indicatively, we point out that only 13 features are able to produce a Pearson Correlation Coefficient of 0.92 with the MOS. Interestingly, the performance statistics we computed in order to assess our method for predicting the Structural Similarity Index and the Video Quality Metric are equally good. Thus, the obtained experimental results verified the suitability of the features selected by LASSO as well as the ability of LASSO in making accurate predictions through sparse modeling.

show abstract

“…First studies by Keimel et al [140,141] have shown that crowdtesting can indeed provide results similar to the results from formal subjective testing in a laboratory environment. But as also discussed by Keimel et al [138], many challenges still remain before crowdtesting is a universally acceptable replacement for formal subjective testing.…”

Section: Roommentioning

confidence: 99%

Design of Video Quality Metrics with Multi-Way Data Analysis

Keimel¹

2016

T-Labs Series in Telecommunication Services

Self Cite

View full text Add to dashboard Cite

Video quality metrics determine the visual quality of distorted video sequences by using prediction models based on objectively measurable features and are therefore an alternative to the time-consuming and costly subjective video quality assessment. In the conventional design approach to video quality metrics, however, the temporal nature of video is often considered only inadequately due to the use of temporal pooling in the prediction process. Moreover, this approach also often requires knowledge about the human visual system that is not readily or only partly available. In this thesis, I therefore propose a data driven design methodology using multi-way data analysis for the design of video quality metrics. This data driven design approach not only requires no detailed knowledge of the human visual system, but also allows for a proper consideration of the temporal nature of video by using a three-way prediction model, corresponding to the three-way structure of video. Using two simple example metrics, I demonstrate that this purely data driven approach not only outperforms video quality metrics in the state-of-the-art that are often highly optimised towards specific properties of the human visual system, but also that multi-way data analysis methods outperform the combination of two-way data analysis methods and temporal pooling.iii

show abstract

Challenges in crowd-based video quality assessment

Cited by 25 publications

References 19 publications

Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing

Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing

A no-reference bitstream-based perceptual model for video quality estimation of videos affected by coding artifacts and packet losses

Design of Video Quality Metrics with Multi-Way Data Analysis

Contact Info

Product

Resources

About