On subjective quality assessment of adaptive video streaming via crowdsourcing and laboratory based experiments

Sogaard, Jacob; Shahid, Muhammad; Pokhrel, Jeevan; Brunnström, Kjell

doi:10.1007/s11042-016-3948-3

Cited by 11 publications

(10 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[6] used CS as well to collect ground truth ratings of the aesthetic appeal of images. Work in [7] employed CS to evaluate users' perceived quality in video streaming applications, and [8] carried out perceptual audio evaluation tasks. Research in [9] addressed the QoE of a teleconference system via a CS and lab test.…”

Section: Related Workmentioning

confidence: 99%

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

et al. 2020

View full text Add to dashboard Cite

Subjective speech quality assessment has traditionally been carried out in laboratory environments under controlled conditions. With the advent of crowdsourcing platforms tasks, which need human intelligence, can be resolved by crowd workers over the Internet. Crowdsourcing also offers a new paradigm for speech quality assessment, promising higher ecological validity of the quality judgments at the expense of potentially lower reliability. This paper compares laboratory-based and crowdsourcing-based speech quality assessments in terms of comparability of results and efficiency. For this purpose, three pairs of listening-only tests have been carried out using three different crowdsourcing platforms and following the ITU-T Recommendation P.808. In each test, listeners judge the overall quality of the speech sample following the Absolute Category Rating procedure. We compare the results of the crowdsourcing approach with the results of standard laboratory tests performed according to the ITU-T Recommendation P.800. Results show that in most cases, both paradigms lead to comparable results. Notable differences are discussed with respect to their sources, and conclusions are drawn that establish practical guidelines for crowdsourcing-based speech quality assessment.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Paired-comparison approaches have been widely adopted in CS for quality assessment of image [21], video [22], audio [23] and synthetic speech [24,17]. The study in [23] introduces a pair-comparison framework for quantifying QoE of multimedia content as a more convenient approach compared to 5-point scale ratings.…”

Section: Previous Workmentioning

confidence: 99%

Perceptual Ratings of Voice Likability Collected Through In-Lab Listening Tests vs. Mobile-Based Crowdsourcing

2017

View full text Add to dashboard Cite

Human perceptions of speaker characteristics, needed to perform automatic predictions from speech features, have generally been collected by conducting demanding in-lab listening tests under controlled conditions. Concurrently, crowdsourcing has emerged as a valuable approach for running user studies through surveys or quantitative ratings. Micro-task crowdsourcing markets enable the completion of small tasks (commonly of minutes or seconds), rewarding users with micro-payments. This paradigm permits effortless collection of user input from a large and diverse pool of participants at low cost. This paper presents different auditory tests for collecting perceptual voice likability ratings employing a common set of 30 male and female voices. These tests are based on direct scaling and on paired-comparisons, and were conducted in the laboratory and via crowdsourcing using micro-tasks. Design considerations are proposed for adapting the laboratory listening tests to a mobile-based crowdsourcing platform to obtain trustworthy listeners' answers. Our likability scores obtained by the different test approaches are highly correlated. This outcome motivates the use of crowdsourcing for future listening tests investigating e.g. speaker characterization, reducing the efforts involved in engaging participants and administering the tests on-site.

show abstract

“…On the other hand, tons of streaming data collected from sensing devices are uploaded to DCs, which results in significant traffic flow between DCs and backbone network. (2) Traditional model can lead to poor Quality of Service (QoS) for users [ 47 ]. To obtain the first-hand information such as emergency events, public hot issues, or interesting news, users have to go through some prolonged routing processes.…”

Section: Introductionmentioning

confidence: 99%

“…In those caching schemes, the major idea is that, when a user requests data, the requested data would be cached in routers along the transmission path of that data. When other users request the same data, they can directly fetch the data from routers, and the traffic burden of delivering data would hence be released and QoS would be ameliorated [ 35 , 47 ].…”

Section: Introductionmentioning

confidence: 99%

Caching Joint Shortcut Routing to Improve Quality of Service for Information-Centric Networking

Huang

Liu

Zhang

et al. 2018

Sensors

View full text Add to dashboard Cite

Hundreds of thousands of ubiquitous sensing (US) devices have provided an enormous number of data for Information-Centric Networking (ICN), which is an emerging network architecture that has the potential to solve a great variety of issues faced by the traditional network. A Caching Joint Shortcut Routing (CJSR) scheme is proposed in this paper to improve the Quality of service (QoS) for ICN. The CJSR scheme mainly has two innovations which are different from other in-network caching schemes: (1) Two routing shortcuts are set up to reduce the length of routing paths. Because of some inconvenient transmission processes, the routing paths of previous schemes are prolonged, and users can only request data from Data Centers (DCs) until the data have been uploaded from Data Producers (DPs) to DCs. Hence, the first kind of shortcut is built from DPs to users directly. This shortcut could release the burden of whole network and reduce delay. Moreover, in the second shortcut routing method, a Content Router (CR) which could yield shorter length of uploading routing path from DPs to DCs is chosen, and then data packets are uploaded through this chosen CR. In this method, the uploading path shares some segments with the pre-caching path, thus the overall length of routing paths is reduced. (2) The second innovation of the CJSR scheme is that a cooperative pre-caching mechanism is proposed so that QoS could have a further increase. Besides being used in downloading routing, the pre-caching mechanism can also be used when data packets are uploaded towards DCs. Combining uploading and downloading pre-caching, the cooperative pre-caching mechanism exhibits high performance in different situations. Furthermore, to address the scarcity of storage size, an algorithm that could make use of storage from idle CRs is proposed. After comparing the proposed scheme with five existing schemes via simulations, experiments results reveal that the CJSR scheme could reduce the total number of processed interest packets by 54.8%, enhance the cache hits of each CR and reduce the number of total hop counts by 51.6% and cut down the length of routing path for users to obtain their interested data by 28.6–85.7% compared with the traditional NDN scheme. Moreover, the length of uploading routing path could be decreased by 8.3–33.3%.

show abstract

On subjective quality assessment of adaptive video streaming via crowdsourcing and laboratory based experiments

Cited by 11 publications

References 38 publications

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Perceptual Ratings of Voice Likability Collected Through In-Lab Listening Tests vs. Mobile-Based Crowdsourcing

Caching Joint Shortcut Routing to Improve Quality of Service for Information-Centric Networking

Contact Info

Product

Resources

About