The purpose of this study is to analyze which behaviors are or are not helpful for debugging when a novice is in a state of unproductive persistence. Further, this project will exploratorily use a variety of analytical techniques -- including association rule mining, process mining, frequent sequence mining, and machine learning-- in order to determine which approaches are useful for data analysis. For the study, programming process data from hundreds of novice programmers were analyzed to determine which behaviors were more or less likely to be correlated with escaping a state of unproductive persistence. Of these events, only three had a statistically significant difference in their rates of occurrence and large effect sizes: file, edit, and compile events. While the data set cannot reveal a user's motivation for a file event, the most logical explanation of these events is that the user is tracing the code. Thus, a higher rate of file events suggests that code tracing (with the goal of code comprehension) is a key behavior correlated with a student's ability to escape a state of unproductive persistence. On the other hand, editing events are far more common in unproductive states that are not escaped. A content analysis suggests that there are more trivial edits for users in an unescaped state of unproductive persistence. An important finding of this study is that an unproductive persistence is not just a phenomenon of the worst-performing students; rather, a third of users who completed the assignment had at least one unproductive state. This study also lends support to the idea that tinkering combined with code tracing is correlated with positive outcomes, but that less systematic tinkering is not effective behavior. Further, association rule mining and frequent sequence mining were effective tools for data analysis in this study. The findings from this study have two main practical implications for curriculum designers and instructors: (1) the need to normalize struggle and (2) possibilities for curriculum and tool development. This work is particularly important given that debugging is not normally a process evident to instructors, curriculum designers, tool developers, and computer science education researchers, either because it happens outside of class time and/or because it is a process and these stakeholders usually only see the end result; this project attempts to make the process of debugging more transparent.
