Suggestions for Online User Studies

Salminen, Joni; Jung, Sin‐Ho; Jansen, Bernard J.

doi:10.1007/978-3-030-90238-4_11

Cited by 5 publications

(4 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based primarily on the experience of the authors, some of these techniques are: Crowdworker Platforms : Unlike MTurk, which at the time of this study provides limited sample selection features, some other crowdworker platforms offer rather sophisticated sampling methods and access to populations of real users from various domains, demographics, and experiences, mitigating access issues to real users and offering the payment of real users at a reasonable cost. Additionally, some of these platforms offer advance quality control features, although, of course, there are both pros and cons of using these crowdworker platforms ( Peer et al, 2022 ; Salminen, Jung & Jansen, 2021 ). Partnering with Commercial Companies, Non-profits, and Governmental Organizations : Paying real users , especially in some specialty domains, can be rather expensive.…”

Section: Discussionmentioning

confidence: 99%

Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields

Salminen

Jung

Kamel

et al. 2022

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Background Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users or surrogate users is unknown. Therefore, it is uncertain whether or not the use of surrogate users in place of real users is a widespread problem within user study practice. Objective To determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants. Method We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users (i.e., from the study population) or surrogate users (i.e., other than real users). Results Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users. However, of the targeted studies, only around half (58.4%) sampled real users. Of the targeted studies sampling surrogate users, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were. Conclusions Key findings are as follows: (a) the state of sampling real users in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers. Implications The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.

show abstract

Section: Discussionmentioning

confidence: 99%

Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields

Salminen

Jung

Kamel

et al. 2022

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…The pre-screening of study subjects is often regarded as a best practice when conducting online studies [41]. Hence, we used Prolific's built-in qualification features to recruit participants based on their gender (i.e., to ensure balance within study groups), social media usage (Instagram), and language skills (German).…”

Section: Study Limitationsmentioning

confidence: 99%

“…Hence, we used Prolific's built-in qualification features to recruit participants based on their gender (i.e., to ensure balance within study groups), social media usage (Instagram), and language skills (German). In addition, we targeted users who already took part in at least 10 other studies as these are usually more committed and less likely to drop from experiments [41]. Last but not least, we also tried to keep the length of both the individual surveys and the full experiment as short as possible to reduce participants' fatigue and minimize the chances of attrition.…”

Section: Study Limitationsmentioning

confidence: 99%

ENAGRAM: An App to Evaluate Preventative Nudges for Instagram

Ferreyra,

Ostendorf,

Aïmeur

et al. 2022

Preprint

View full text Add to dashboard Cite

Online self-disclosure is perhaps one of the last decade's most studied communication processes, thanks to the introduction of Online Social Networks (OSNs) like Facebook. Self-disclosure research has contributed significantly to the design of preventative nudges seeking to support and guide users when revealing private information in OSNs. Still, assessing the effectiveness of these solutions is often challenging since changing or modifying the choice architecture of OSN platforms is practically unfeasible. In turn, the effectiveness of numerous nudging designs is supported primarily by self-reported data instead of actual behavioral information. Objective: This work presents ENAGRAM, an app for evaluating preventative nudges, and reports the first results of an empirical study conducted with it. Such a study aims to showcase how the app (and the data collected with it) can be leveraged to assess the effectiveness of a particular nudging approach. Method: We used ENAGRAM as a vehicle to test a risk-based strategy for nudging the self-disclosure decisions of Instagram users. For this, we created two variations of the same nudge (i.e., with and without risk information) and tested it in a between-subjects experimental setting. Study participants (N=22) were recruited via Prolific and asked to use the app regularly for 7 days. An online survey was distributed at the end of the experiment to measure some privacy-related constructs. Results: From the data collected with ENAGRAM, we observed lower (though non-significant) self-disclosure levels when applying risk-based interventions. The constructs measured with the survey were not significant either, except for participants' External Information Privacy Concerns (EIPC). Implications: Our results suggest that (i) ENAGRAM is a suitable alternative for conducting longitudinal experiments in a privacy-friendly way, and (ii) it provides a flexible framework for the evaluation of a broad spectrum of nudging solutions.

show abstract

“…We opted for an international sample experienced in different regulations to investigate privacy requirements relevant to a global context. Thus, based on previous research recommendations exploring different recruitment samples, methods, and platforms [39,69], we recruited a cross-section of participants from various locations with expertise in software development and privacy on Upwork.com [93]. Theoretical sampling indicated that employees with team coordinator positions (e.g., team leads, product owners) in software development teams play a crucial role in the communication process, henceforth referred to as team coordinators.…”

Section: Introductionmentioning

confidence: 99%

“Those things are written by lawyers, and programmers are reading that.” Mapping the Communication Gap Between Software Developers and Privacy Experts

Horstmann,

Domiks,

Gutfleisch

et al. 2024

PoPETs

View full text Add to dashboard Cite

To ensure data-privacy compliance, it is common for companies to consult privacy experts for the identification and communication of privacy requirements to software developers. However, developers often fail to fulfill those requirements resulting in companies regularly being fined for violations due to non-compliance with privacy data regulations. To investigate why software developers struggle with the implementation of privacy requirements and explore their communication modality, we conducted a qualitative semi-structured interview study with 30 participants involving 10 software developers, 10 privacy experts, and 10 team coordinators with an average experience of nine years in the privacy communication and implementation process within a company context. We found a communication gap between software developers and privacy experts, suggesting a lack of proper procedural steps during the software development process to guarantee that the privacy requirements have been adequately addressed. We also uncovered that since privacy requirements were mostly communicated in a uni-directional manner, they were often perceived as a hindrance during software development, thus fostering an adversarial relationship between privacy experts and developers. Therefore, in order to fulfill the experts' requirements, software developers requested concrete steps to take during the software development process, as observed in the security field. However, privacy experts often lacked the technical knowledge to provide such instructions. This work contributes an explanatory theory on the communication gap between software developers and privacy experts. We discuss common obstacles in the communication of privacy experts and software developers and provide guidance on how to address them.

show abstract

Suggestions for Online User Studies

Cited by 5 publications

References 50 publications

Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields

Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields

ENAGRAM: An App to Evaluate Preventative Nudges for Instagram

“Those things are written by lawyers, and programmers are reading that.” Mapping the Communication Gap Between Software Developers and Privacy Experts

Contact Info

Product

Resources

About