Using Anonymity and Communal Efforts to Improve Quality of Crowdsourced Feedback

Hui, Julie; Glenn, Amos; Jue, Rachel; Gerber, Elizabeth M.; Dow, Steven P.

doi:10.1609/hcomp.v3i1.13229

Cited by 14 publications

(2 citation statements)

References 40 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There may also be specific benefits to providing anonymity for students within crowdsourcing tools. There is emerging evidence that anonymity can result in more useful peer reviews of learnersourced content [67] and expose fewer biases related to gender and race [37].…”

Section: Discussionmentioning

confidence: 99%

“…There is also some evidence that allowing anonymity in crowdsourced environments may help lead to more useful peer feedback and avoid biases during review phases. For example, Hui et al [67] report that anonymous peer reviews provide more specific criticism and praise than reviews from identifiable reviewers, and students find the specificity of the reviews more helpful. Similar findings are reported by Lu et al [68] and Howard et al [69] who note that anonymous reviews are more critical than nonanonymous ones.…”

Section: ) Student Anonymitymentioning

confidence: 99%

See 1 more Smart Citation

Lessons Learned From Four Computing Education Crowdsourcing Systems

et al. 2023

View full text Add to dashboard Cite

Crowdsourcing is a general term that describes the practice of many individuals working collectively to achieve a common goal or complete a task, often involving the generation of content. In an educational context, crowdsourcing of learning materials -where students create resources that can be used by other learners -offers several benefits. Students benefit from the act of producing resources as well as from using the resources. Despite benefits, instructors may be hesitant to adopt crowdsourcing for several reasons, such as concerns around the quality of content produced by students and the perceptions students may have of creating resources for their peers. While prior work has explored crowdsourcing concerns within the context of individual tools, lessons that are generalisable across multiple platforms and derived from practical use can provide considerably more robust insights. In this perspective article, we present four crowdsourcing tools that we have developed and used in computing classrooms. From our previous studies and experience, we derive lessons which shed new light on some of the concerns that are typical for instructors looking to adopt such tools. We find that across multiple contexts, students are capable of generating high quality learning content which provides good coverage of key concepts. Although students do appear hesitant to engage with new kinds of activities, various types of incentives have proven effective. Finally, although studies on learning effects have shown mixed results, no negative outcomes have been observed. In light of these lessons, we hope to see a greater uptake and use of crowdsourcing in computing education.INDEX TERMS contributing student pedagogy, crowdsourcing, crowdsourcing systems, learnersourcing

show abstract

Section: Discussionmentioning

confidence: 99%

Section: ) Student Anonymitymentioning

confidence: 99%

Lessons Learned From Four Computing Education Crowdsourcing Systems

et al. 2023

View full text Add to dashboard Cite

show abstract

Understanding crowdsourcing in science

et al. 2022

View full text Add to dashboard Cite

Over the past 16 years, the concept of crowdsourcing has rapidly gained traction across many research fields. While related debates focused mainly on its importance for business, the public and non-governmental sectors, its relevance for generating scientific knowledge is increasingly emphasized. This rising interest remains in contradiction with its feeble recognition, and excessive simplifications reducing crowdsourcing in science to citizen science. Conceptual clarity and a coherent framework would help integrate the various research streams. The aim of this paper is to extend reflection on crowdsourcing in science by analyzing the characteristics of the phenomenon. We synthesize a consensual definition from the literature, and structure key characteristics into a coherent framework, useful in guiding further research. We use a systematic literature review procedure to generate a pool of 42 definitions from a comprehensive set of 62 articles spanning different literatures, including: business and economics, education, psychology, biology, and communication studies. We follow a mixed-method approach that combines bibliometric and frequency analyses with deductive coding and thematic analysis. Based on triangulated results we develop an integrative definition: crowdsourcing in science is a collaborative online process through which scientists involve a group of self-selected individuals of varying, diverse knowledge and skills, via an open call to the Internet and/or online platforms, to undertake a specified research task or set of tasks. We also provide a conceptual framework that covers four key characteristics: initiator, crowd, process, and technology.

show abstract