Crowdsourcing programming assignments with CrowdSorcerer

Abstract

Small automatically assessed programming assignments are an often used resource for learning programming. Creating sufficiently large amounts of such assignments is, however, time consuming. As a consequence, offering large quantities of practice assignments to students is not always possible. CrowdSorcerer is an embeddable open-source system that students and teachers alike can use for creating and evaluating small automatically assessed programming assignments. While creating programming assignments, the students also write simple input-output-tests, and are gently introduced to the basics of testing. Students can also evaluate the assignments of others and provide feedback on them, which exposes them to code written by others early in their education. In this article we both describe the CrowdSorcerer system and our experiences in using the system in a large undergraduate programming course. Moreover, we discuss the motivation for crowdsourcing course assignments and present some usage statistics. CCS CONCEPTS • Information systems → Crowdsourcing; • Human-centered computing → Collaborative content creation; • Social and professional topics → Computing education;

Crowdsourcing Content Creation for SQL Practice

“…Similarly, CrowdSorcerer [31] is an embeddable tool that is used for programming assignment creation. Students come up with an assignment description according to some specifics given by an instructor.…”

“…Another way to alleviate quality issues is peer reviewing, as with the multiple-choice questions in PeerWise [12] and programming assignments in CrowdSorcerer [31]. Studies have shown that the quality of peer reviewed exercises is good [13,19], even when both the creators and the reviewers are from the same crowd of students.…”

Analysis of Students' Peer Reviews to Crowdsourced Programming Assignments

“…In addition to weekly programming assignments, the courses used a crowdsourcing tool called CrowdSourcerer [15] that allows the students to create programming assignments related to a certain topic of the course, e.g. loops.…”

Crowdsourcing in Computer Science Education

“…My research extends the previous work regarding the use of CrowdSorcerer [6], and the quality and usability of the programming assignments created with the tool [7]. My research also strives to generalize the results and working solutions of existing educational tools that utilize crowdsourcing in computer science education.…”