Customizable and scalable automated assessment of C/C++ programming assignments

Delgado‐Pérez, Pedro; Medina‐Bulo, Inmaculada

doi:10.1002/cae.22317

Cited by 19 publications

(9 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…JEWL is a Java library to create GUI with a mock version that can be used for automatic assessment [34]. CAC++ is a C++ library developed to help instructors design their own assessment programs for C and C++ assignments [110]. Some systems have also been implemented as programs to be run as backend services to ease the integration with existing frontend ones.…”

Section: Toolmentioning

confidence: 99%

Automated Code Assessment for Education: Review, Classification and Perspectives on Techniques and Tools

Combéfis

2022

Software

View full text Add to dashboard Cite

Automatically assessing code for learning purposes is a challenging goal to achieve. On-site courses and online ones developed for distance learning both require automated ways to grade learners’ programs to be able to scale and manage a large public with a limited teaching staff. This paper reviews recent automated code assessment systems. It proposes a systematic review of the possible analyses they can perform with the associated techniques, the kinds of produced feedback and the ways they are integrated in the learning process. It then discusses the key challenges for the development of new automated code assessment systems and the interaction with human grading. In conclusion, the paper draws several recommendations for new research directions and for possible improvements for automatic code assessment.

show abstract

Section: Toolmentioning

confidence: 99%

Automated Code Assessment for Education: Review, Classification and Perspectives on Techniques and Tools

Combéfis

2022

Software

View full text Add to dashboard Cite

show abstract

“…Automation of assessment facilitates more extensive practice and self-learning of students, which is essential in novice programming and supports nontraditional modes of classes such as in massive open online courses (MOOCs) [2,22,42]. APASs have been found to be beneficial to both instructors and students in a variety of other ways [1,17,32,43,46], including convenient access and streamlined administration and marking of exercises and assignments [4,6,22], provision of prompt and informative feedback to students for effective learning [24,27,35,42], promotion of students' learning motivation [9,26], collection of data analytics for improvement of teaching, and so on. These benefits, however, are critically dependent on how well APASs can assess students' programs accurately.…”

Section: Introductionmentioning

confidence: 99%

An automated system with a versatile test oracle for assessing student programs

Tang

Poon

2022

Comp Applic In Engineering

View full text Add to dashboard Cite

Automated program assessment systems have been widely adopted in many universities. Many of these systems judge the correctness of student programs by comparing their actual outputs with predefined expected outputs for selected test inputs. A common weakness of such systems is that student programs would be marked as incorrect as long as their outputs deviate from the predefined ones, even if the deviations are only minor, insignificant, and considered acceptable by a human assessor that the programs have satisfied the specifications. This critical weakness caused undue frustration to students and undesirable pedagogical consequences that undermine these systems’ benefits. To address this issue, we developed an improved mechanism for program output comparison to serve as a versatile test oracle that brings the results of automated assessment much closer to those of human assessors. We evaluated the new mechanism in real programming classes using an existing automated program assessment system. We found that the new mechanism achieved zero false‐positive error (did not wrongly accept any incorrect output) and very low (0%–0.02%) false‐negative error (that wrongly rejected correct outputs), with very high accuracy (99.8%–100%) in correctly recognizing outputs deemed acceptable by instructors. This represents a major improvement over an existing assessment mechanism, which had 56.4%–64.1% false‐negative error with an accuracy of 25.4%–40.9%. Moreover, about 67%–96% of students achieved their best results in their first attempt, which could be encouraging to them and reduce their frustration. Furthermore, students generally welcomed the new assessment mechanism and agreed it was beneficial to their learning.

show abstract

“…Output comparison techniques also present another important limitation: because they cannot access the source code and they only check whether the output is correct or wrong, they have only two possible evaluation values: correct or wrong. Nevertheless, the real scenario is often different: teachers usually divide the mark of an exercise into smaller evaluable parts [15,26,9]. Assuming a methodology where 10 points is the maximum grade, 5 the minimum grade to pass the exam, and each appraisable piece not fulfilled decreases the student's final mark, Solution 1 would barely pass the exam.…”

Section: Introductionmentioning

confidence: 99%

“…These properties are specified and verified according to a language based on the abstract syntax tree (AST) of the program, which is unique for a particular Java program. A new similar assessment system is the one introduced by [15] for C/C++ programs. They developed a C library able to inspect the static structure of the code.…”

Section: Introductionmentioning

confidence: 99%

“…Unlike some of the described systems, the exercise does not need to be provided with a set of test cases, our system is able to generate its own test suite maximizing branch coverage. Additionally, it uses an assessment DSL that performs dynamic and static analysis of the source code measuring the number of tests passed and properties fulfilled and associating a recommended mark for them, like Delgado and Medina [15]. The DSL also allows the system to automatically correct errors, so they do not affect the rest of the code.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semiautomatic generation and assessment of Java exercises in engineering education

Insa

Pérez

Silva

et al. 2020

Comp Applic In Engineering

View full text Add to dashboard Cite

Continuous assessment is essential in education. It should be an integral part of education that provides immediate feedback to students. Unfortunately, the assessment of programming source code is still a hand‐operated and error‐prone task, and can take weeks before the student gets feedback. This study presents a semiautomatic code assessment method able to automatically apply black‐box assessment (which relies on the comparison of input‐output pairs) and white‐box assessment (which relies on checking different source code properties). The method proposed is a general‐purpose assessment system that was originally designated to be used in engineering education, but that can be used in other educational contexts to assist the assessment of any Java programming assignments or exams. The main advantage of this system is that the assessment made is quicker, exhaustive, and objective; and it does not produce false positives. After the application of this method over two years in several real university courses, we have released a public and free implementation. An empirical evaluation of this system estimates that the amount of assessment work automatically done by the tool is over 48%. Additionally, the system has been used to measure the average subjective influence (i.e., assessment errors) introduced by teachers when they assess exams manually.

show abstract

Customizable and scalable automated assessment of C/C++ programming assignments

Cited by 19 publications

References 37 publications

Automated Code Assessment for Education: Review, Classification and Perspectives on Techniques and Tools

Automated Code Assessment for Education: Review, Classification and Perspectives on Techniques and Tools

An automated system with a versatile test oracle for assessing student programs

Semiautomatic generation and assessment of Java exercises in engineering education

Contact Info

Product

Resources

About