Intention-based scoring

Abstract

Traditional methods of evaluating student programs are not always appropriate for assessment of different instructional interventions. They tend to focus on the final product rather than on the process that led to it. This paper presents intention-based scoring (IBS), an approach to measuring programming ability that requires inspection of intermediate programs produced over the course of an implementation rather than just the one at the end. The intent is to assess a student's ability to produce algorithmically correct code on the first attempt at achieving each program goal. In other words, the goal is to answer question "How close was the student to being initially correct?" rather than the the ability to ultimately produce a working program. To produce an IBS, it is necessary to inspect a student's online protocol, which is defined as the collection of all programs submitted to a compiler. IBS involves a three-phase process of (1) identification of the subset of all programs in a protocol that represent the initial attempts at achieving programming goals, (2) analysis of the bugs in those programs, and (3) rubric-based scoring of the resulting tagged programs. We conclude with an example application of IBS in the evaluation of a tutoring system for beginning programmers and also show how an IBS can be broken down by the underlying bug categories to reveal more subtle differences.

Affective and behavioral predictors of novice programmer achievement

“…The saved data included (but were not limited to) the computer number, time stamp, error message (if any), file name, and source code in compilation A single student record for a particular lab was composed of many compilations. This collection of all programs submitted to the compiler is known as an online protocol [15]. Data was only retained for students who consented to participate in the study.…”

Affective and behavioral predictors of novice programmer achievement

“…[10] proposes that it may be possible to detect moods from programmer uses of the keyboard or mouse but has not published experimental results as of the time of this writing. [13,15] have shown that there is a measurable relationship between students' compilation behaviors and their achievement levels, which are discussed in greater detail in the following section.…”