Finding Anomalies in Scratch Assignments

Abstract

In programming education, teachers need to monitor and assess the progress of their students by investigating the code they write. Code quality of programs written in traditional programming languages can be automatically assessed with automated tests, verification tools, or linters. In many cases these approaches rely on some form of manually written formal specification to analyze the given programs. Writing such specifications, however, is hard for teachers, who are often not adequately trained for this task. Furthermore, automated tool support for popular block-based introductory programming languages like SCRATCH is lacking. Anomaly detection is an approach to automatically identify deviations of common behavior in datasets without any need for writing a specification. In this paper, we use anomaly detection to automatically find deviations of SCRATCH code in a classroom setting, where anomalies can represent erroneous code, alternative solutions, or distinguished work. Evaluation on solutions of different programming tasks demonstrates that anomaly detection can successfully be applied to tightly specified as well as open-ended programming tasks.

Anomaly Detection in Scratch Assignments

“…We mined AS anomalies on one of the tightly specified tasks of our original data set, the cat task. The AA approach found 91 violations and reported 30 of them in our initial experiments [3]. Using the same parameters, the AS approach found 54 violations and reported 20 of them, as the others were below the confidence threshold configured.…”

“…We evaluated our approach on six data sets of student solutions for SCRATCH assignments (799 solutions in total), one of them being an open task where students were free to implement what they wanted to [3]. All of these solutions were created from students aged 8-13 during programming sessions with qualified teachers.…”

“…Figure 1 presents the four essential phases of anomaly detection in SCRATCH [3]. Ot h e r S p r i t e s V i o l a t i o n P a t t e r n Da t a s e t Fig.…”

“…To address this problem, anomaly detection can help teachers assess and evaluate student assignments [3]. Anomaly detection is a static software analysis technique that has been successful at finding bugs in large projects using text-based programming languages [12].…”

“…In prior work, we adapted and extended an approach to anomaly detection from the context of object-oriented programming based on formal concept analysis [12] to the context of SCRATCH. We mined and evaluated anomalies of six data sets consisting of different SCRATCH solutions created by students [3]. To further explore the potential of anomaly detection in SCRATCH, in this paper we propose actor specific anomaly detection, which groups scripts by the actor (sprite or background) they belong to.…”