The Influence of Test Suite Properties on Automated Grading of Programming Exercises

Clegg, Benjamin; McMinn, Phil

doi:10.1109/cseet49119.2020.9206231

Cited by 3 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we have provided empirical evidence that different test suites generate varying grades for students' programs, and that observable properties of suites can influence these generated grades. Our findings differ from those our previous study [4]. We attribute this to both differences in our updated analysis, and differences between students' solutions and mutants, such as the quantity of faults, or their subtlety.…”

Section: Discussioncontrasting

confidence: 99%

“…We previously investigated how different test suites generated varying grades, and how properties of the suites influence this variation [4]. We constructed test suites by randomly sampling from a large set of tests, and executed these test suites on simple artificial faulty variants of their associated subject programs, called mutants.…”

Section: Existing Workmentioning

confidence: 99%

“…Since the relative importance analysis that we employ is robust to correlated variables, by including |τ | as a property we can effectively control for its contribution to the other properties. We exclude the recoverage metric that we used in our original study [4], since diagnosability metrics also evaluate the repeated coverage of a program's lines, and are more well established in other studies [6,9]. With the exception of the detection of other students' solutions, we measure all of the properties using the model solution, to simulate a tutor developing a new grading test suite.…”

Section: Test Suite Propertiesmentioning

confidence: 99%

“…For example, test suites that have a higher coverage tend to have a higher mutation score [11]. This offers a benefit over simply comparing the magnitudes of a linear model's normalised (β) coefficients, which we used in our original study [4]; β coefficients do not accurately capture the contributions for correlated predictors. This measure also provides estimates of the relative importance in terms of the predictors' impacts on the variance of the response variable.…”

Section: Relative Importancementioning

confidence: 99%

“…We seek to understand how such differences in test suites affect students' grades. We previously investigated this, by measuring grades generated for artificial faulty program variants, called mutants [4]. However, these mutants do not always perfectly reflect students' faults [5].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Diagnosability, Adequacy & Size: How Test Suites Impact Autograding

Clegg¹,

McMinn²

2022

Proceedings of the Annual Hawaii International Conference on System Sciences

Self Cite

View full text Add to dashboard Cite

Automated grading is now prevalent in software engineering courses, typically assessing the correctness of students' programs using automated test suites. However, deficiencies in test suites could result in inconsistent grading. As such, we investigate how different test suites impact grades, and the extent to which their observable properties influence these grades. We build upon existing work, using students' solution programs, and test suites that we constructed using a sampling approach. We find that there is a high variation in grades from different test suites, with a standard deviation of ∼10.1%. We further investigate how several properties of test suites influence these grades, including the number of tests, coverage, ability to detect other faults, and uniqueness. We use our findings to provide tutors with strategies for building test suites that evaluate students' software with consistency. These strategies include constructing test suites with high coverage, writing unique and diverse tests which evaluate solutions' correctness in different ways, and to run the tests against artificial faults to determine their quality.

show abstract

Section: Discussioncontrasting

confidence: 99%

Section: Existing Workmentioning

confidence: 99%

Section: Test Suite Propertiesmentioning

confidence: 99%

Section: Relative Importancementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Diagnosability, Adequacy & Size: How Test Suites Impact Autograding

Clegg¹,

McMinn²

2022

Proceedings of the Annual Hawaii International Conference on System Sciences

Self Cite

View full text Add to dashboard Cite

show abstract

Automated Grading and Feedback Tools for Programming Education: A Systematic Review

Messer,

Brown,

Kölling

et al. 2024

ACM Trans. Comput. Educ.

View full text Add to dashboard Cite

We conducted a systematic literature review on automated grading and feedback tools for programming education. We analysed 121 research papers from 2017 to 2021 inclusive and categorised them based on skills assessed, approach, language paradigm, degree of automation and evaluation techniques. Most papers assess the correctness of assignments in object-oriented languages. Typically, these tools use a dynamic technique, primarily unit testing, to provide grades and feedback to the students or static analysis techniques to compare a submission with a reference solution or with a set of correct student submissions. However, these techniques’ feedback is often limited to whether the unit tests have passed or failed, the expected and actual output, or how they differ from the reference solution. Furthermore, few tools assess the maintainability, readability or documentation of the source code, with most using static analysis techniques, such as code quality metrics, in conjunction with grading correctness. Additionally, we found that most tools offered fully automated assessment to allow for near-instantaneous feedback and multiple resubmissions, which can increase student satisfaction and provide them with more opportunities to succeed. In terms of techniques used to evaluate the tools’ performance, most papers primarily use student surveys or compare the automatic assessment tools to grades or feedback provided by human graders. However, because the evaluation dataset is frequently unavailable, it is more difficult to reproduce results and compare tools to a collection of common assignments.

show abstract