Crowdsourcing Content Creation for SQL Practice

Leinonen, Juho; Pirttinen, Nea; Hellas, Arto

doi:10.1145/3341525.3387385

Cited by 17 publications

(8 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One instructional approach that has become increasingly common in computing education is learnersourcing Kim (2015) where students participate in the creation and evaluation of course materials such as questions and exercises (see e.g. Denny et al (2015Denny et al ( , 2017; Pirttinen et al (2018); Leinonen et al (2020)).…”

Section: Future Workmentioning

confidence: 99%

Automatic Generation of Programming Exercises and Code Explanations with Large Language Models

Sarsa,

Denny,

Hellas

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.

show abstract

Section: Future Workmentioning

confidence: 99%

Automatic Generation of Programming Exercises and Code Explanations with Large Language Models

Sarsa,

Denny,

Hellas

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This act of using students as a crowd in a crowdsourcing activity is sometimes referred to as learnersourcing [3] -in this article, we include learnersourcing under the more commonly used banner of crowdsourcing. In computing education research, crowdsourcing has been used, for example, to create multiple choice questions [1], [4], introductory programming assignments [2], [5], and SQL exercises [6].…”

Section: Introductionmentioning

confidence: 99%

Lessons Learned From Four Computing Education Crowdsourcing Systems

et al. 2023

Self Cite

View full text Add to dashboard Cite

Crowdsourcing is a general term that describes the practice of many individuals working collectively to achieve a common goal or complete a task, often involving the generation of content. In an educational context, crowdsourcing of learning materials -where students create resources that can be used by other learners -offers several benefits. Students benefit from the act of producing resources as well as from using the resources. Despite benefits, instructors may be hesitant to adopt crowdsourcing for several reasons, such as concerns around the quality of content produced by students and the perceptions students may have of creating resources for their peers. While prior work has explored crowdsourcing concerns within the context of individual tools, lessons that are generalisable across multiple platforms and derived from practical use can provide considerably more robust insights. In this perspective article, we present four crowdsourcing tools that we have developed and used in computing classrooms. From our previous studies and experience, we derive lessons which shed new light on some of the concerns that are typical for instructors looking to adopt such tools. We find that across multiple contexts, students are capable of generating high quality learning content which provides good coverage of key concepts. Although students do appear hesitant to engage with new kinds of activities, various types of incentives have proven effective. Finally, although studies on learning effects have shown mixed results, no negative outcomes have been observed. In light of these lessons, we hope to see a greater uptake and use of crowdsourcing in computing education.INDEX TERMS contributing student pedagogy, crowdsourcing, crowdsourcing systems, learnersourcing

show abstract

“…Current learnersourcing tools support a wide variety of artefact types, including hints, subgoal-labels, programming problems and complex assignments (Mitros 2015;Kim, Miller, and Gajos 2013;Leinonen, Pirttinen, and Hellas 2020;Pirttinen et al 2018;Denny et al 2011). Multiplechoice questions (MCQs) are a very popular format in learnersourcing platforms, appearing in tools such as RiPPLE (Khosravi, Kitto, and Williams 2019), Quizzical (Riggs, Kang, and Rennie 2020), UpGrade (Wang et al 2019) and PeerWise (Denny, Luxton-Reilly, and Hamer 2008).…”

Section: Introductionmentioning

confidence: 99%

DeepQR: Neural-Based Quality Ratings for Learnersourced Multiple-Choice Questions

Qiming

et al. 2022

AAAI

View full text Add to dashboard Cite

Automated question quality rating (AQQR) aims to evaluate question quality through computational means, thereby addressing emerging challenges in online learnersourced question repositories. Existing methods for AQQR rely solely on explicitly-defined criteria such as readability and word count, while not fully utilising the power of state-of-the-art deep-learning techniques. We propose DeepQR, a novel neural-network model for AQQR that is trained using multiple-choice-question (MCQ) datasets collected from PeerWise, a widely-used learnersourcing platform. Along with designing DeepQR, we investigate models based on explicitly-defined features, or semantic features, or both. We also introduce a self-attention mechanism to capture semantic correlations between MCQ components, and a contrastive-learning approach to acquire question representations using quality ratings. Extensive experiments on datasets collected from eight university-level courses illustrate that DeepQR has superior performance over six comparative models.

show abstract

Crowdsourcing Content Creation for SQL Practice

Cited by 17 publications

References 33 publications

Automatic Generation of Programming Exercises and Code Explanations with Large Language Models

Automatic Generation of Programming Exercises and Code Explanations with Large Language Models

Lessons Learned From Four Computing Education Crowdsourcing Systems

DeepQR: Neural-Based Quality Ratings for Learnersourced Multiple-Choice Questions

Contact Info

Product

Resources

About