An Empirical Study to Determine if Mutants Can Effectively Simulate Students' Programming Mistakes to Increase Tutors' Confidence in Autograding

Clegg, Benjamin; McMinn, Phil

doi:10.1145/3408877.3432411

Cited by 3 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results revealed that these varying suites had a significant variation in grades, with a mean standard deviation of 2.94%, despite the mean grade only being ∼96.5%. While we later found that the detection of mutants and students' faults are correlated [5], there are still some differences between mutants and real faults. In particular, mutants typically contain only one fault, while students' faults can contain several; they would likely produce different test results and grades.…”

Section: Existing Workmentioning

confidence: 80%

“…We previously investigated this, by measuring grades generated for artificial faulty program variants, called mutants [4]. However, these mutants do not always perfectly reflect students' faults [5]. As such, we build upon our previous work in this paper, using real students' solution programs in place of artificial mutants, to better reflect the real influence of test suites on grades.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Diagnosability, Adequacy & Size: How Test Suites Impact Autograding

Clegg¹,

McMinn²

2022

Proceedings of the Annual Hawaii International Conference on System Sciences

Self Cite

View full text Add to dashboard Cite

Automated grading is now prevalent in software engineering courses, typically assessing the correctness of students' programs using automated test suites. However, deficiencies in test suites could result in inconsistent grading. As such, we investigate how different test suites impact grades, and the extent to which their observable properties influence these grades. We build upon existing work, using students' solution programs, and test suites that we constructed using a sampling approach. We find that there is a high variation in grades from different test suites, with a standard deviation of ∼10.1%. We further investigate how several properties of test suites influence these grades, including the number of tests, coverage, ability to detect other faults, and uniqueness. We use our findings to provide tutors with strategies for building test suites that evaluate students' software with consistency. These strategies include constructing test suites with high coverage, writing unique and diverse tests which evaluate solutions' correctness in different ways, and to run the tests against artificial faults to determine their quality.

show abstract

Section: Existing Workmentioning

confidence: 80%

Section: Introductionmentioning

confidence: 99%