Crowdsourcing for usability testing

Liu, Di; Bias, Randolph G.; Lease, Matthew; Kuipers, Rebecca

doi:10.1002/meet.14504901100

Cited by 68 publications

(40 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We carefully designed our study to remove privacy biases by making tasks uniform for all MTurkers and assessing only usability. Moreover, while the quality of results from tests on MTurk are not as good as lab-based testing, Liu et al [35] found that MTurk was a viable platform for usability testing. A general concern with using platforms such as MTurk is gaming by the workers.…”

Section: Limitationsmentioning

confidence: 99%

To Permit or Not to Permit, That is the Usability Question: Crowdsourcing Mobile Apps’ Privacy Permission Settings

Ismail

Ahmed

Caine

et al. 2017

Proceedings on Privacy Enhancing Technologies

View full text Add to dashboard Cite

Abstract:Millions of apps available to smartphone owners request various permissions to resources on the devices including sensitive data such as location and contact information. Disabling permissions for sensitive resources could improve privacy but can also impact the usability of apps in ways users may not be able to predict. We study an efficient approach that ascertains the impact of disabling permissions on the usability of apps through large-scale, crowdsourced user testing with the ultimate goal of making recommendations to users about which permissions can be disabled for improved privacy without sacrificing usability. We replicate and significantly extend previous analysis that showed the promise of a crowdsourcing approach where crowd workers test and report back on various configurations of an app. Through a large, between-subjects user experiment, our work provides insight into the impact of removing permissions within and across different apps (our participants tested three apps: Facebook Messenger (N=218), Instagram (N=227), and Twitter (N=110)). We study the impact of removing various permissions within and across apps, and we discover that it is possible to increase user privacy by disabling app permissions while also maintaining app usability.

show abstract

Section: Limitationsmentioning

confidence: 99%

To Permit or Not to Permit, That is the Usability Question: Crowdsourcing Mobile Apps’ Privacy Permission Settings

Ismail

Ahmed

Caine

et al. 2017

Proceedings on Privacy Enhancing Technologies

View full text Add to dashboard Cite

show abstract

“…By contrast, the intent of our approach is to solve a newly encountered problem in crowdsourced testing. Liu et al [36] investigated both methodological differences and empirical contrasts between crowdsourced usability testing and traditional face-to-face usability testing. To solve the oracle problem, Pastore et al [37] applied the crowdsourcing technique to generate test inputs depending on a test oracle that required human input in one form or another.…”

Section: Crowdsourced Software Testingmentioning

confidence: 99%

Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining

Guo

Chen

2017

Symmetry

View full text Add to dashboard Cite

Abstract:Crowdsourcing is an appealing and economic solution to software application testing because of its ability to reach a large international audience. Meanwhile, crowdsourced testing could have brought a lot of bug reports. Thus, in crowdsourced software testing, the inspection of a large number of test reports is an enormous but essential software maintenance task. Therefore, automatic prediction of the severity of crowdsourced test reports is important because of their high numbers and large proportion of noise. Most existing approaches to this problem utilize supervised machine learning techniques, which often require users to manually label a large number of training data. However, Android test reports are not labeled with their severity level, and manual labeling is time-consuming and labor-intensive. To address the above problems, we propose a Knowledge Transfer Classification (KTC) approach based on text mining and machine learning methods to predict the severity of test reports. Our approach obtains training data from bug repositories and uses knowledge transfer to predict the severity of Android test reports. In addition, our approach uses an Importance Degree Reduction (IDR) strategy based on rough set to extract characteristic keywords to obtain more accurate reduction results. The results of several experiments indicate that our approach is beneficial for predicting the severity of android test reports.

show abstract

“…They use Amazon's Mechanical Turk (MTurk) as a tool that allows them to easily manage crowdsourced studies, perform prerequisite qualification tests for filtering participants, ensure privacy, manage payments, and collect results. The authors in [16], used crowdsourcing and MTurk platform in evaluating the usability of a school website. The advantages are claimed to be more participants' involvement, low cost, high speed, and various users' backgrounds, while the disadvantages include lower quality feedback, less interactions, more spammers, less focused user groups.…”

Section: Related Workmentioning

confidence: 99%

Crowdsourcing software evaluation

Sherief

Jiang

Hosseini

et al. 2014

Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

Crowdsourcing is an emerging online paradigm for problem solving which involves a large number of people often recruited on a voluntary basis and given, as a reward, some tangible or intangible incentives. It harnesses the power of the crowd for minimizing costs and, also, to solve problems which inherently require a large, decentralized and diverse crowd. In this paper, we advocate the potential of crowdsourcing for software evaluation. This is especially true in the case of complex and highly variable software systems, which work in diverse, even unpredictable, contexts. The crowd can enrich and keep the timeliness of the developers' knowledge about software evaluation via their iterative feedback. Although this seems promising, crowdsourcing evaluation introduces a new range of challenges mainly on how to organize the crowd and provide the right platforms to obtain and process their input. We focus on the activity of obtaining evaluation feedback from the crowd and conduct two focus groups to understand the various aspects of such an activity. We finally report on a set of challenges to address and realize correct and efficient crowdsourcing mechanisms for software evaluation.

show abstract

Crowdsourcing for usability testing

Cited by 68 publications

References 15 publications

To Permit or Not to Permit, That is the Usability Question: Crowdsourcing Mobile Apps’ Privacy Permission Settings

To Permit or Not to Permit, That is the Usability Question: Crowdsourcing Mobile Apps’ Privacy Permission Settings

Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining

Crowdsourcing software evaluation

Contact Info

Product

Resources

About