Towards Computer-based Exams in CS1

Abstract

Abstract:Even though IDEs are often a central tool when learning to program in CS1, many teachers still lean on paperbased exams. In this study, we examine the "test mode effect" in CS1 exams using the Rainfall problem. The test mode was two-phased. Half of the participants started working on the problem with pen and paper, while the other half had access to an IDE. After submitting their solution, all students could rework their solution on an IDE. The experiment was repeated twice during subsequent course instances. The results were mixed. From the marking perspective, there was no statistically significant difference resulting from the mode. However, the students starting with the paper-based part tended to make more errors in their code, but after the computer-based reworking phase, they matched or exceeded the level of the students who started with the computer-based phase. We also discuss the reliability of automatic assessment that is based on a unit test suite that was developed for the purposes of this study.

Introductory programming: a systematic literature review

“…Students and staff perceive the assessment to be fair and an accurate assessment of programming skills [63,236,715] and to improve student motivation [52]. Students working on computers produce more accurate code than on paper [360]. In a study by Haghighi and Sheard [236] involving both paperbased and computer-based assessment, students found the computer test environment to be more stressful; however, ten years later, Matthíasdóttir and Arnalds [420] found that students were more comfortable taking exams on a computer than on paper.…”

Exploring ‘reverse-tracing’ Questions as a Means of Assessing the Tracing Skill on Computer-based CS 1 Exams

“…It is also noteworthy that past studies that evaluated the performance difference between computerized and paper assessments only evaluated the product of student work and not the process in which it was generated [11,20,30], in contrast to the current work. Also, these previous studies evaluated the differences only on code writing problems, while our study focuses on code tracing (and reverse-tracing) problems.…”

“…Corley et al found that CS 1 students' problem solving ability was largely unaffected by the different assessment formats in code writing tasks, and that student ability to correctly write syntax was significantly worse in the paper-and-pencil version [11]. Similarly, Lappalainen et al's study found that CS1 students in both paper and computer groups had similar performances in terms of understanding on a Rainfall code writing problem, but the paper group had more syntax and minor cosmetic errors (e.g., misspelled variable name) [30]. In contrast, Grissom et al found that CS 2 students' solutions demonstrated higher levels of understanding in the computerized formats when writing implementations of recursive binary search trees [20].…”

“…In contrast, Grissom et al found that CS 2 students' solutions demonstrated higher levels of understanding in the computerized formats when writing implementations of recursive binary search trees [20]. As students in the computerformat groups had access to Integrated Development Environments (IDEs) [11,30], it is not surprising that students had fewer syntax errors, because IDEs can highlight and auto-correct incorrect syntax, but the mixed results related to understanding/problem solving are harder to explain. Computer-based exams can also benefit instructors with auto-grading and automatic analysis of students' learning progress [44].…”

“…In most disciplines, the effect of shifting exams from paper to computer is minimal [8,38], but previous work has noted significant format effects in computing courses when students can compile and execute code in the computer-format. Corley et al and Lappalainen et al found that CS1 student performance on code writing tasks did not significantly differ between the two formats in terms of understanding, but did differ significantly in syntax error frequency [11,30]. In contrast, Grissom et al found that CS2 students in the paper-format group of their study performed significantly worse in terms of understanding [20].…”