Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks

Abstract

Educators use a wide variety of data to inform their practices. Examples of these data include forms of information that are commonplace in schools, such as student work and paper-based artifacts. One limitation in these situations is that there are less efficient ways to process such everyday varieties of information into analytics that are more usable and practical for educators. To explore how to address this constraint, we describe two sets of design experiments that utilize crowdsourced tasks for scoring open-ended assessments. Developing crowdsourced systems and their resulting analytics introduced a variety of challenges, such as attending to the expertise and learning of the crowd. In this paper, we describe the potential efficacy of design decisions such as screening the crowd, providing multimedia instruction, and asking the crowd to explain their answers. We also explore the potential of crowdsourcing as a learning opportunity for those participating in the collective tasks. Our work offers key design implications for leveraging crowdsourcing to process educational data in ways that are relevant to educators, while offering learning experiences for the crowd.

LAK of Direction

“…These performance measures, and many more, had they been observed, would have been coded as measuring learning. A study that examined different methods for assessing middle school students' use of evidence and systems thinking was coded as measuring learning (Ahn et al, 2021). Two studies that examined children's dialogue during collaborative problem solving were both coded as measuring learning (Emara et al, 2021;Ma et al, 2022).…”

Practical Sentiment Analysis for Education: The Power of Student Crowdsourcing

“…Crowdsourcing has proven useful to cheaply label large SA datasets used in applications outside of education (Heidari and Shamsinejad 2020). In the educational context crowdsourcing has been utilized for the design and use of crowdsourced learning analytics tasks (Ahn et al 2021), as well as, to interpret learners' reviews of MOOCs (Li et al 2022), but neither to gather sentiment labels nor to serve as a hands-on learning activity for students themselves.…”

Artificial Intelligence in Today’s Education Landscape: Understanding and Managing Ethical Issues for Educational Assessment

“…• construct assessments collaboratively through e.g., crowdsourcing of assessment tasks (Ahn et al, 2021); and…”

“…Research by Ahn et al (2021) also showed that automated grading of learners' work, which contains complex data, rich semantic meaning and idiosyncratic and local nuances, may not be well graded by present computational approaches that utilize metrics such as counts of parts of speech and essay length as proxies for writing complexity and quality.…”