Automated assessment of students' testing skills for improving correctness of their code

Abstract

Although software testing is included as a regular part of many programming courses, current assessment techniques used in automated grading tools for evaluating student-written software tests are imperfect. Code coverage measures are typically used in practice, but that approach does not assess how much of the expected behavior is checked by the tests and sometimes, overestimates the true quality of the tests. Two robust and thorough measures for evaluating student-written tests are running each students' tests against others' solutions(known as all-pairs testing) and injecting artificial bugs to determine if tests can detect them (also known as mutation analysis). Even though they are better indicators of test quality, both of them posed a number of practical obstacles to classroom use. This proposal describes technical obstacles behind using these two approaches in automated grading. We propose novel and practical solutions to apply all-pairs testing and mutation analysis of student-written tests, especially in the context of classroom grading tools. Experimental results of applying our techniques in eight CS1 and CS2 assignments submitted by 147 students show the feasibility of our solution. Finally, we discuss our plan to combine the approaches to evaluate tests of assignments having variable amounts of design freedom and explain their evaluation method.

A Quantitative Study of Faculty Views on the Goals of an Undergraduate CS Program and Preparing Students for Industry

“…We already see some evidence of this because the academia-industry gap persists despite the many papers that have been published on how to effectively teach industry-relevant content. For instance, there is a plethora of studies on how to teach software testing as well as on how to automatically grade it; some examples are [9,29,31]. Unfortunately, these practices are not yet commonplace at many institutions (among which our own) which suggests these barriers may be inhibiting adoption.…”