Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario

Volk, Thomas; Keimel, Christian; Moosmeier, Michael; Diepold, Klaus

doi:10.1016/j.comnet.2015.05.021

Cited by 10 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has to be noted that each worker provided 13.2 ratings on average, while each in-lab listener provided 29 ratings. Lower participant reliability in CS compared to in-lab tests was also reported in [20,18].…”

Section: Cs-sca Test Resultsmentioning

confidence: 67%

“…Other studies using CS for collecting rates on a Likert scale concentrate on audio quality assessments [16,7] employing the discrete 5-point absolute-category-rating (ACR) for subjective Mean Opinion Scores), on voice naturalness [17], and on perceived Quality of Experience (QoE) in a teleconference system [18]. Different to using a Likert scale, the approach in [19] was to ask CS participants to rate correct or incorrect realizations of the /r/ sound in words.…”

Section: Previous Workmentioning

confidence: 99%

“…In laboratory, the most reliable ratings were obtained for the 5-point ACR scale and for the continuous scale with anchors, and in CS, the highest user agreement was found for the discrete ACR scale, yet lower reliability compared to laboratory was detected for all scales. Also the analyses in [18] revealed lower participant reliability in CS compared to laboratory, and [7] and [19] showed strong agreement between CS and in-lab test participants.…”

Section: Previous Workmentioning

confidence: 99%

See 2 more Smart Citations

Perceptual Ratings of Voice Likability Collected Through In-Lab Listening Tests vs. Mobile-Based Crowdsourcing

2017

View full text Add to dashboard Cite

Human perceptions of speaker characteristics, needed to perform automatic predictions from speech features, have generally been collected by conducting demanding in-lab listening tests under controlled conditions. Concurrently, crowdsourcing has emerged as a valuable approach for running user studies through surveys or quantitative ratings. Micro-task crowdsourcing markets enable the completion of small tasks (commonly of minutes or seconds), rewarding users with micro-payments. This paradigm permits effortless collection of user input from a large and diverse pool of participants at low cost. This paper presents different auditory tests for collecting perceptual voice likability ratings employing a common set of 30 male and female voices. These tests are based on direct scaling and on paired-comparisons, and were conducted in the laboratory and via crowdsourcing using micro-tasks. Design considerations are proposed for adapting the laboratory listening tests to a mobile-based crowdsourcing platform to obtain trustworthy listeners' answers. Our likability scores obtained by the different test approaches are highly correlated. This outcome motivates the use of crowdsourcing for future listening tests investigating e.g. speaker characterization, reducing the efforts involved in engaging participants and administering the tests on-site.

show abstract

Section: Cs-sca Test Resultsmentioning

confidence: 67%

Section: Previous Workmentioning

confidence: 99%

Section: Previous Workmentioning

confidence: 99%

See 1 more Smart Citation

Perceptual Ratings of Voice Likability Collected Through In-Lab Listening Tests vs. Mobile-Based Crowdsourcing

2017

View full text Add to dashboard Cite

show abstract

“…T. Hoßfeld et al has stated [17] that in general, every crowdsourcing task suffers from bad quality results. Though, even if the task is designed effectively, subjects might still submit unreliable and misleading survey results [41]. Establishing a trustworthy cluster of subjects (either paid or voluntarily registered) across distributed geographical locations that access the service via different network operators will establish a good understanding of the QoE of the service.…”

Section: Experiments Methodologymentioning

confidence: 99%

Predicting quality of experience for online video service provisioning

Bulkan

Dagiuklas

2019

Multimed Tools Appl

View full text Add to dashboard Cite

The expansion of the online video content continues in every area of the modern connected world and the need for measuring and predicting the Quality of Experience (QoE) for online video systems has never been this important. This paper has designed and developed a machine learning based methodology to derive QoE for online video systems. For this purpose, a platform has been developed where video content is unicasted to users so that objective video metrics are collected into a database. At the end of each video session, users are queried with a subjective survey about their experience. Both quantitative statistics and qualitative user survey information are used as training data to a variety of machine learning techniques including Artificial Neural Network (ANN), K-nearest Neighbours Algorithm (KNN) and Support Vector Machine (SVM) with a collection of cross-validation strategies. This methodology can efficiently answer the problem of predicting user experience for any online video service provider, while overcoming the problematic interpretation of subjective consumer experience in terms of quantitative system capacity metrics.

show abstract

“…[9,10,11,12]) as well as audio and (synthetic) voice quality testing (e.g. [13,14,15]). In contrast, Crowdsourcing has been much less used for Web QoE research (e.g.…”

Section: Related Workmentioning

confidence: 99%

Size does matter. Comparing the results of a lab and a crowdsourcing file download QoE study

Sackl¹,

Gardlo²,

Schatz³

2016

5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS 2016)

View full text Add to dashboard Cite

Over the last couple of years, crowdsourcing has become a widely used method for conducting subjective QoE experiments over the Internet. However, the scope of crowdsourced QoE experiments so far has been mostly limited to video and image quality testing, despite the existence of many other relevant application categories. In this paper we demonstrate the applicability of crowdsourced QoE testing to the case of file downloads. We conducted several campaigns in which participants had to download large (10-50MB) media files (with defined waiting times) and subsequently rate their QoE. The results are compared with those of a lab-based file download QoE study featuring an equivalent design. Our results show that crowdsourced QoE testing can also be applied to file downloads with a size of 10 MB as rating results are very similar to the lab. However, beyond user reliability checks and filtering, we found the study design to be a highly critical element as it exerted strong influence on overall participant behavior. For this reason we also present a discussion of valuable lessons learned in terms of test design and participant behavior.

show abstract

Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario

Cited by 10 publications

References 20 publications

Perceptual Ratings of Voice Likability Collected Through In-Lab Listening Tests vs. Mobile-Based Crowdsourcing

Perceptual Ratings of Voice Likability Collected Through In-Lab Listening Tests vs. Mobile-Based Crowdsourcing

Predicting quality of experience for online video service provisioning

Size does matter. Comparing the results of a lab and a crowdsourcing file download QoE study

Contact Info

Product

Resources

About