Re-Factoring Based Program Repair Applied to Programming Assignments

Abstract

Automated program repair has been used to provide feedback for incorrect student programming assignments, since program repair captures the code modification needed to make a given buggy program pass a given test-suite. Existing student feedback generation techniques are limited because they either require manual effort in the form of providing an error model, or require a large number of correct student submissions to learn from, or suffer from lack of scalability and accuracy. In this work, we propose a fully automated approach for generating student program repairs in real-time. This is achieved by first re-factoring all available correct solutions to semantically equivalent solutions. Given an incorrect program, we match the program with the closest matching refactored program based on its control flow structure. Subsequently, we infer the input-output specifications of the incorrect program's basic blocks from the executions of the correct program's aligned basic blocks. Finally, these specifications are used to modify the blocks of the incorrect program via search-based synthesis. Our dataset consists of almost 1,800 real-life incorrect Python program submissions from 361 students for an introductory programming course at a large public university. Our experimental results suggest that our method is more effective and efficient than recently proposed feedback generation approaches. About 30% of the patches produced by our tool Refactory are smaller than those produced by the state-of-art tool Clara, and can be produced given fewer correct solutions (often a single correct solution) and in a shorter time. We opine that our method is applicable not only to programming assignments, and could be seen as a general-purpose program repair method that can achieve good results with just a single correct reference solution.

FAPR: Fast and Accurate Program Repair for Introductory Programming Courses

“…Data-Based Feedback Generation: Some tools were proposed to generate feedback based on the comparison between correct and incorrect solutions [32], [33], [34], [35], [36], [4], [3], [37], [38], [39], [40]. For instance, Pex4Fun [33] and CodeHunt [34] use dynamic symbolic execution to generate test cases that reveal behavioral discrepancies between student solutions and the teacher's hidden program specification.…”

Automated Repair of Heap-Manipulating Programs Using Deductive Synthesis

“…7) are produced using a bug injection tool. Therefore, we plan to evaluate our approach on student submissions in programming courses, similar to previous approaches [14,18,45,51].…”

Verifix: Verified Repair of Programming Assignments

“…Completely Beyond Identical Verified Automated Reference CFG Repair Clara [9] ✓ ✗ ✗ SarfGen [28] ✓ ✗ ✗ ITSP [29] ✓ ✓ ✗ Refactory [12] ✓ ✓ ✗ CoderAssist [13] ✗ ✓ ✓ Verifix ✓ ✓ ✓ Table 1. Programming assignment repair tools comparison.…”

“…Several programming assignments are typically attempted by the students as a part of this course, which are evaluated and graded against pre-defined test-cases. Given the importance of programming education and the difficulty of providing relevant feedback for the massive number of students, there has been increasing interest in automated program repair techniques for providing automated feedback to student assignments (APR) [9,12,13,25,28,29].…”

“…Table 1 provides a summary of state-of-the-art APR works, and compares them with our approach Verifix. The repair rate of the state-of-the-art APR techniques [9,12,28] is astonishingly high, around 90%. However, these works make certain assumptions such as the presence of tests.…”