An Effective Approach to Enhancing Compiler Error Messages

Abstract

Publication information SIGCSE '16 Proceedings of the 47th ACM Technical Symposium on Computing Science EducationConference details ABSTRACTOne of the many challenges novice programmers face from the time they write their first program is inadequate compiler error messages. These messages report details on errors the programmer has made and are the only feedback the programmer gets from the compiler. For students they play a particularly essential role as students often have little experience to draw upon, leaving compiler error messages as their primary guidance on error correction. However these messages are frequently inadequate, presenting a barrier to progress and are often a source of discouragement. We have designed and implemented an editor that provides enhanced compiler error messages and conducted a controlled empirical study with CS1 students learning Java. We find a reduced frequency of overall errors and errors per student. We also identify eight frequent compiler error messages for which enhancement has a statistically significant effect. Finally we find a reduced number of repeated errors. These findings indicate fewer students struggling with compiler error messages.

A New Metric to Quantify Repeated Compiler Errors for Novice Programmers

“…We show that RED is reduced in an empirical control/intervention study, providing evidence that this metric is valid. Reductions in EQ and RED also support the results of [3] and [4], providing further evidence that enhancing error messages is effective. We also discuss some threats to validity.…”

“…The control group experienced the normal JDK compiler errors while the intervention group experienced enhanced errors provided by the editor based on the JDK error and the student code. This editor was found, with the same data set, to significantly reduce the overall number of student errors, the number of errors per student, and the amount of repeated student errors per compiler error message [4]. Of course RED is just one of many possible metrics that could be developed to measure repeated errors.…”

“…To compare and contrast the Error Quotient and the Repeated Error Density, and to see how the RED behaves on real data, we applied each to a data set obtained with the methodology described in [4]. This data consists of 29,019 error events from two cohorts of programming novices (approximately 100 students each) who had compiler data logged for four weeks while using a custom Java editor.…”

Effective compiler error message enhancement for novice programming students

“…In previous work (Becker, 2016a) we showed that there was a significant reduction in the number of errors encountered by an intervention group that received ECEMs compared to a control group that did not. We also reported preliminary evidence that the number of repeated errors was significantly reduced.…”

“…This is evidence that students in both groups were behaving similarly in the absence of ECEMs. In (Becker, 2016a) we investigated the 15 most frequent CEMs and found that of these, nine had a statistically significant reduction in the number of errors per student. These are presented in Table 5.…”