Active Learning for Crowdsourced QoE Modeling

Chang, Haw-Shiuan; Hsu, Chih-Fan; Hosfeld, Tobias; Chen, Kuan-Ta

doi:10.1109/tmm.2018.2831639

Cited by 17 publications

(9 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interestingly, the presence of the exponential form to represent the saturation effect of the QoE on a continuous scale was also suggested in the literature ( [18] and references therein).…”

Section: Appendix 1 Second Case Study: Resultsmentioning

confidence: 89%

“…We report in the results section a comparison of our proposed model with this model, where we observe a more coherent pattern for increasing bitrate for our model. Subject bias is also considered in the model developed in [18]. We highlight, however, that our goal is different since our model only considers the results of the subjective tests and does not attempt to establish a relationship between QoS and QoE as in [18].…”

Section: Modeling Subject Bias and Influence Of Content/pvsmentioning

confidence: 99%

See 1 more Smart Citation

Estimation of Quality Scores From Subjective Tests-Beyond Subjects’ MOS

Pezzulli

Martini

Barman

2021

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Subjective tests for the assessment of the quality of experience (QoE) are typically run with a pool of subjects providing their opinion scores using a 5-point scale. The subjects' mean opinion score (MOS) is generally assumed as the best estimation of the average score in the target population. Indeed, for a large enough sample, we may assume that the mean of the variations across the subjects approaches zero, but this is not the case for the limited number of subjects typically considered in subjective tests. In this paper, we propose an approach based on generalized linear models (GLMs) for estimation of the population average QoE. The motivating dataset is composed of the individual scores assigned by 25 subjects to a set of gaming videos evaluated under different resolutions and compression ratios. The approach recognizes the multinomial nature of the data and allows for correlation between scores of the same subject. The resulting estimated average QoE is shown to follow more credible patterns than the MOS, particularly for higher bitrates, for which the model estimates present more coherent behavior. Similar convincing results are found on a second dataset, showing the validity of the approach.

show abstract

Section: Appendix 1 Second Case Study: Resultsmentioning

confidence: 89%

Section: Modeling Subject Bias and Influence Of Content/pvsmentioning

confidence: 99%

Estimation of Quality Scores From Subjective Tests-Beyond Subjects’ MOS

Pezzulli

Martini

Barman

2021

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…In [29], authors use active learning to target crowdsourced QoE modeling scenarios whereas we consider controlled experimentation. They highlight the problem of subjectivity in the QoE scores which leads to degraded performance with active learning.…”

Section: Related Work and Discussionmentioning

confidence: 99%

An intelligent sampling framework for controlled experimentation and QoE modeling

Khokhar¹,

Saber²,

Spetebroot³

et al. 2018

Computer Networks

View full text Add to dashboard Cite

For internet applications, measuring, modeling and predicting the quality experienced by end users as a function of network conditions is challenging. A common approach for building application specific Quality of Experience (QoE) models is to rely on controlled experimentation. For accurate QoE modeling, this approach can result in a large number of experiments to carry out because of the multiplicity of the network features, their large span (e.g., bandwidth, delay) and the time needed to setup the experiments themselves. However, most often, the space of network features in which experimentations are carried out shows a high degree of similarity in the training labels of QoE. This similarity, difficult to predict beforehand, amplifies the training cost with little or no improvement in QoE modeling accuracy. So, in this paper, we aim to exploit this similarity, and propose a methodology based on active learning, to sample the experimental space intelligently, so that the training cost of experimentation is reduced. We validate our approach for the case of YouTube video streaming QoE modeling from out-of-band network performance measurements, and perform a rigorous analysis of our approach to quantify the gain of active sampling over uniform sampling.

show abstract

“…This sampling problem is interesting and investigated in recent literature. As a starting point, the interested reader may review for example [21][22][23][24] developing active learning algorithms for multidimensional QoE models.…”

Section: Fundamental Multidimensional Qoe Relationshipsmentioning

confidence: 99%

“…In general, we may have m different mapping functions f * j , ( j = 1, ⋯ , m ) which are defined on a k-dimensional parameter space , where m and k might be different. For instance, (21)…”

Section: Additive Mos Model and Expected Qoe In The Systemmentioning

confidence: 99%

Deriving QoE in systems: from fundamental relationships to a QoE-based Service-level Quality Index

et al. 2020

Self Cite

View full text Add to dashboard Cite

With Quality of Experience (QoE) research having made significant advances over the years, service and network providers aim at user-centric evaluation of the services provided in their system. The question arises how to derive QoE in systems. In the context of subjective user studies conducted to derive relationships between influence factors and QoE, user diversity leads to varying distributions of user rating scores for different test conditions. Such models are commonly exploited by providers to derive various QoE metrics in their system, such as expected QoE, or the percentage of users rating above a certain threshold. The question then becomes how to combine (a) user rating distributions obtained from subjective studies, and (b) system parameter distributions, so as to obtain the actual observed QoE distribution in the system? Moreover, how can various QoE metrics of interest in the system be derived? We prove fundamental relationships for the derivation of QoE in systems, thus providing an important link between the QoE community and the systems community. In our numerical examples, we focus mainly on QoE metrics. We furthermore provide a more generalized view on quantifying the quality of systems by defining a QoE-based Service-level Quality Index. This index exploits the fact that quality can be seen as a proxy measure for utility. Following the assumption that not all user sessions should be weighted equally, we aim to provide a generic framework that can be utilized to quantify the overall utility of a service delivered by a system.

show abstract

Active Learning for Crowdsourced QoE Modeling

Cited by 17 publications

References 26 publications

Estimation of Quality Scores From Subjective Tests-Beyond Subjects’ MOS

Estimation of Quality Scores From Subjective Tests-Beyond Subjects’ MOS

An intelligent sampling framework for controlled experimentation and QoE modeling

Deriving QoE in systems: from fundamental relationships to a QoE-based Service-level Quality Index

Contact Info

Product

Resources

About