Building a Corpus of Task-Based Grading and Feedback Systems for Learning and Teaching Programming

Abstract

Using grading and feedback systems in the context of learning and teaching programming is quite common. During the last 20 to 40 years research results on several hundred systems and approaches have been published. Existing papers may tell researchers what works well in terms of educational support and how to make a grading and feedback system stable, extensible, secure, or sustainable. However, finding a solid basis for such kind of research is hard due to the vast amount of publications from a very diverse community. Hardly any recent systematic review includes data from more than 100 systems (most include less than 30). Hence, the authors started an endeavor to build a corpus of all task-based grading and feedback systems for learning and teaching programming that deal with source code and have been published in recent years. The intention is to provide the community with a solid basis for their research. The corpus is also designed to be updated and extended by the community with future systems. This paper describes the process of building the corpus and presents some meta-analysis that shed light on the involved research communities.

Towards Giving Timely Formative Feedback and Hints to Novice Programmers

“…A number of systematic literature reviews from the past six years were used as a starting point for the identification of adequate learning environments and their features [3,12,14,15,31,35,51,62,63,66]. We briefly introduce the main points of this (non-exhaustive) list of references in chronological order starting in 2016.…”

“…The 39 papers from 2016 to 2021 distinguish six different feedback approaches of relevant tools, but the report does not provide information on their availability and accessibility. Strickroth and Striewe [66] provide the most comprehensive and recent corpus of task-based grading and feedback systems for learning and teaching programming, although feedback types are not part of the overview.…”

AI-Enhanced Auto-Correction of Programming Exercises: How Effective is GPT-3.5?

“…Crowdsourcing feedback through peer review has limitations, relying on the motivation and expertise of participating students [4][5][6]. Many e-assessment systems have been developed to (semi-)automatically assess students' submissions and provide feedback [7]. However, a common issue is that these systems often focus solely on functional correctness and employ standard compilers and test cases.…”