Automatic Grading of Programming Exercises using Property-Based Testing

Earle, Clara Benac; Fredlund, Lars–Åke; Hughes, John

doi:10.1145/2899415.2899443

Cited by 14 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The last decade underwent a slight shift in what is considered to be a correct, partially correct, and incorrect solution by most practitioners. First, producing the expected output is not by itself the only factor that weighs in the grade of the program [27]. The adopted strategy, quality of the source code, algorithmic complexity, and…”

Section: Discussionmentioning

confidence: 99%

“…1 Learning progress graphs 2 Code size variation, code save progression, syntactic analytics, compilation error timeline, execution sequence 3 Student behavior and question analytics many other aspects are also relevant indicators and, thus, may afect grades [27] and originate feedback to improve learning and promote good practices [238]. Then, in the limit, a program with a syntax error might be closer to the solution than a program failing a single test case.…”

Section: Discussionmentioning

confidence: 99%

“…Notwithstanding, JavAssess [115] can check, using relection, the cyclomatic complexity, the number of usages of a certain construct, among others. Benac Earle et al [27] used algorithmic complexity (computed using property-based testing), the number of noncommenting source statements, and McCabe's cyclomatic complexity as weights of the grade assigned to a program.…”

Section: Functionality (Source Code)mentioning

confidence: 99%

See 2 more Smart Citations

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Paiva

Leal

Figueira

2022

ACM Trans. Comput. Educ.

124

View full text Add to dashboard Cite

Practical programming competencies are critical to the success in computer science education and go-to-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through a broad range of programming activities that learners must perform themselves. It is not reasonable to consider that teachers could evaluate all attempts that the average learner should develop multiplied by the number of students enrolled in a course, much less in a timely, deeply, and fairly fashion. Unsurprisingly, exploring the formal structure of programs to automate the assessment of certain features has long been a hot topic among CS education practitioners. Assessing a program is considerably more complex than asserting its functional correctness, as the proliferation of tools and techniques in the literature over the past decades indicates. Program efficiency, behavior, readability, among many other features, assessed either statically or dynamically, are now also relevant for automatic evaluation. The outcome of an evaluation evolved from the primordial boolean values to information about errors and tips on how to advance, possibly taking into account similar solutions. This work surveys the state-of-the-art in the automated assessment of CS assignments, focusing on the supported types of exercises, security measures adopted, testing techniques used, type of feedback produced, and the information they offer the teacher to understand and optimize learning. A new era of automated assessment, capitalizing on static analysis techniques and containerization, has been identified. Furthermore, this review presents several other findings from the conducted review, discusses the current challenges of the field, and proposes some future research directions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Functionality (Source Code)mentioning

confidence: 99%

See 1 more Smart Citation

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Paiva

Leal

Figueira

2022

ACM Trans. Comput. Educ.

124

View full text Add to dashboard Cite

show abstract

“…The use of automatic grading tools has become nearly ubiquitous in large undergraduate programming courses to accommodate for increased enrollment in traditional CS programs as well as online courses. Several authors have addressed automatic grading platforms [1,3,4,[7][8][9]12]. Many of the systems used so far have been focused on binary criteria: Does the program run?, does the program produce the right output for problem 1?, does the program produce the right output for problem 2?, etc.…”

Section: Introductionmentioning

confidence: 99%

“…Additionally, much of the current literature assesses grading approaches according to the similarity of grades and feedback to human feedback [1,4,6,12], rather than by student outcomes. Given the extent to which automatic grading has become commonplace, it is natural to ask what (if anything) is lost in the real world by replacing human grading with automatic feedback.…”

Section: Introductionmentioning

confidence: 99%

Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style

Leite

Blanco

2020

Proceedings of the 51st ACM Technical Symposium on Computer Science Education

View full text Add to dashboard Cite

The use of automatic grading tools has become nearly ubiquitous in large undergraduate programming courses, and recent work has focused on improving the quality of automatically generated feedback. However, there is a relative lack of data directly comparing student outcomes when receiving computer-generated feedback and human-written feedback. This paper addresses this gap by splitting one 90-student class into two feedback groups and analyzing differences in the two cohorts' performance. The class is an intro to AI with programming HW assignments. One group of students received detailed computer-generated feedback on their programming assignments describing which parts of the algorithms' logic was missing; the other group additionally received human-written feedback describing how their programs' syntax relates to issues with their logic, and qualitative (style) recommendations for improving their code. Results on quizzes and exam questions suggest that human feedback helps students obtain a better conceptual understanding, but analyses found no difference between the groups' ability to collaborate on the final project. The course grade distribution revealed that students who received human-written feedback performed better overall; this effect was the most pronounced in the middle two quartiles of each group. These results suggest that feedback about the syntax-logic relation may be a primary mechanism by which human feedback improves student outcomes. CCS CONCEPTS • Social and professional topics → Computer science education; Computational thinking; Student assessment.

show abstract