The potential for scientific outreach and learning in mechanical turk experiments

Jun, Eunice; Arian, Morelle; Reinecke, Katharina

doi:10.1145/3231644.3231666

Cited by 3 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings suggest a potential direction for LA research in human-computation. In rethinking the design of crowdsourcing tasks to be person-oriented, designers may explore the pedagogical value of the tasks to provide meaningful learning opportunities for all individuals involved [22]. From an ethical perspective, integrating learning into crowdsourcing platforms positions workers as more than rote elements of data processing pipelines [4].…”

Section: Discussionmentioning

confidence: 99%

Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks

Ahn

Nguyen

Campos

et al. 2021

LAK21: 11th International Learning Analytics and Knowledge Conference

View full text Add to dashboard Cite

Educators use a wide variety of data to inform their practices. Examples of these data include forms of information that are commonplace in schools, such as student work and paper-based artifacts. One limitation in these situations is that there are less efficient ways to process such everyday varieties of information into analytics that are more usable and practical for educators. To explore how to address this constraint, we describe two sets of design experiments that utilize crowdsourced tasks for scoring open-ended assessments. Developing crowdsourced systems and their resulting analytics introduced a variety of challenges, such as attending to the expertise and learning of the crowd. In this paper, we describe the potential efficacy of design decisions such as screening the crowd, providing multimedia instruction, and asking the crowd to explain their answers. We also explore the potential of crowdsourcing as a learning opportunity for those participating in the collective tasks. Our work offers key design implications for leveraging crowdsourcing to process educational data in ways that are relevant to educators, while offering learning experiences for the crowd.

show abstract

Section: Discussionmentioning

confidence: 99%

Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks

Ahn

Nguyen

Campos

et al. 2021

LAK21: 11th International Learning Analytics and Knowledge Conference

View full text Add to dashboard Cite

show abstract

“…There are a number of studies that have looked into various ways of training crowdworkers, in addition to the ones mentioned above (Oleson et al 2011;Dow et al 2012;Singla et al 2014;Mamykina et al 2016;Streuer et al 2017;Wang, Hicks, and Luther 2018). Beyond simply training crowdworkers to do better on particular tasks, recent work has looked into understanding how crowdsourcing platforms can support learning as part of crowdwork and to foster the longer term development of worker skills (Krause et al 2016;Dontcheva et al 2014;Suzuki et al 2016;Jun, Arian, and Reinecke 2018). For example, Jun, Arian, and Reinecke (2018) showed that workers value learning about scientific studies that they participate in.…”

Section: Crowdsourcing As Learning At Scalementioning

confidence: 99%

“…Beyond simply training crowdworkers to do better on particular tasks, recent work has looked into understanding how crowdsourcing platforms can support learning as part of crowdwork and to foster the longer term development of worker skills (Krause et al 2016;Dontcheva et al 2014;Suzuki et al 2016;Jun, Arian, and Reinecke 2018). For example, Jun, Arian, and Reinecke (2018) showed that workers value learning about scientific studies that they participate in. Our work fits into this narrative of crowdsourcing platforms as not just platforms to test learning at scale ideas, but to enact and support learning at scale.…”

Section: Crowdsourcing As Learning At Scalementioning

confidence: 99%

Not Everyone Writes Good Examples but Good Examples Can Come from Anywhere

Doroudi

Kamar

Brunskill

2019

HCOMP

View full text Add to dashboard Cite

In many online environments, such as massive open online courses and crowdsourcing platforms, many people solve similar complex tasks. As a byproduct of solving these tasks, a pool of artifacts are created that may be able to help others perform better on similar tasks. In this paper, we explore whether work that is naturally done by crowdworkers can be used as examples to help future crowdworkers perform better on similar tasks. We explore this in the context of a product comparison review task, where workers must compare and contrast pairs of similar products. We first show that randomly presenting one or two peer-generated examples does not significantly improve performance on future tasks. In a second experiment, we show that presenting examples that are of sufficiently high quality leads to a statistically significant improvement in performance of future workers on a near transfer task. Moreover, our results suggest that even among high quality examples, there are differences in how effective the examples are, indicating that quality is not a perfect proxy for pedagogical value.

show abstract

Digestif

Jun

Oliveira

et al. 2018

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Online experiments allow researchers to collect data from large, demographically diverse global populations. Unlike in-lab studies, however, online experiments often fail to inform participants about the research to which they contribute. This paper is the first to investigate barriers that prevent researchers from providing such science communication in online experiments. We found that the main obstacles preventing researchers from including such information are assumptions about participant disinterest, limited time, concerns about losing anonymity, and concerns about experimental bias. Researchers also noted the dearth of tools to help them close the information loop with their study participants. Based on these findings, we formulated design requirements and implemented Digestif, a new web-based tool that supports researchers in providing their participants with science communication pages. Our evaluation shows that Digestif's scaffolding, examples, and nudges to focus on participants make researchers more aware of their participants' curiosity about research and more likely to disclose pertinent research information.

show abstract

The potential for scientific outreach and learning in mechanical turk experiments

Cited by 3 publications

References 25 publications

Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks

Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks

Not Everyone Writes Good Examples but Good Examples Can Come from Anywhere

Digestif

Contact Info

Product

Resources

About