Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis

Abstract

In large introductory programming classes, teacher feedback on individual incorrect student submissions is often infeasible. Program synthesis techniques are capable of fixing student bugs and generating hints automatically, but they lack the deep domain knowledge of a teacher and can generate functionally correct but stylistically poor fixes. We introduce a mixedinitiative approach which combines teacher expertise with data-driven program synthesis techniques. We demonstrate our novel approach in two systems that use different interaction mechanisms. Our systems use program synthesis to learn bug-fixing code transformations and then cluster incorrect submissions by the transformations that correct them. The MISTAKEBROWSER system learns transformations from examples of students fixing bugs in their own submissions. The FIXPROPAGATOR system learns transformations from teachers fixing bugs in incorrect student submissions. Teachers can write feedback about a single submission Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored.

Automated clustering and program repair for introductory programming assignments

“…The extension of our approach to larger programming problems, as found in more advanced courses, is left for future work. Focusing on small to medium size programs is in line with related work on automated feedback generation for introductory programming (e.g., D'Antoni et al [9], Singh et al [32], Head et al [19]). We stress that the state-of-the-art in teaching is manual feedback (as well as failing test cases); thus, automation, even for small to medium size programs, promises huge benefits.…”

“…To avoid the threat that a student's attempt is repaired by her own future correct solution, we split the data into two sets. 19.7s (6.3s) From the first (chronologically earlier) set we take only the correct solutions: these solutions are then clustered, and the obtained clusters are used during the repair of the incorrect attempts. From the second (chronologically later) set we take only the incorrect attempts: on these attempts we perform repair.…”

“…A more guided feedback can be generated from modifications that make a student's program correct, using a program repair technique as pioneered by the AutoGrader tool [32]. Generating feedback from program repair is an active area of research: One line of work focuses on improving the technical capabilites of program repair in introductory education [9,30,31], while another line of research focuses on pedagogical questions such as how to best provide repairbased feedback to the students [19,36]. In this paper we propose a new completely automated approach for repairing introductory programming assignments, while sidelining the pedagogical questions for future work.…”

Investigating the Essential of Meaningful Automated Formative Feedback for Programming Assignments

“…A focus shift from automated grading to automated feedback was witnessed in the most recent decade. Specifically, the focus was on feedback generation through data-driven approaches [17], [18]. Massive Open Online Programming Courses provided large datasets of programming assignments, which is critical to make such approaches possible.…”

TraceDiff: Debugging unexpected code behavior using trace divergences

“…[0, 2, 4, 6, 8, 10] and [0, 1, 3, 6, 10, 15]). The other variables have the same update sequences (e.g., i is the same sequence of updates [1,2,3,4,5,6] for both incorrect and fixed code trace). This technique allows us to extract only traces for the key variables the student should pay attention to.…”