Building a Comprehensive Automated Programming Assessment System

Mekterović, Igor; Brkić, Ljiljana; Milašinović, Boris; Baranović, Mirta

doi:10.1109/access.2020.2990980

Cited by 51 publications

(23 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to assessing programming and functional programming, OJ systems have recently been used to assess complex exercise-based contents such as circuit design and AI [18]. In [39], an automated programming evaluation tool called Edgar was proposed and used to evaluate and analyze a variety of programming tasks. This system does not provide recommendations or any other type of personalized evaluation.…”

Section: Oj Systemsmentioning

confidence: 99%

Educational Data Mining to Support Programming Learning Using Problem-Solving Data

et al. 2022

View full text Add to dashboard Cite

Computer programming has attracted a lot of attention in the development of information and communication technologies in the real world. Meeting the growing demand for highly skilled programmers in the ICT industry is one of the major challenges. In this point, online judge (OJ) systems enhance programming learning and practice opportunities in addition to classroom-based learning. Consequently, OJ systems have created a large number of problem-solving data (solution codes, logs, and scores) archives that can be valuable raw materials for programming education research. In this paper, we propose an educational data mining framework to support programming learning using unsupervised algorithms. The framework includes the following sequence of steps: (i) problem-solving data collection (logs and scores are collected from the OJ) and preprocessing; (ii) MK-means clustering algorithm is used for data clustering in Euclidean space; (iii) statistical features are extracted from each cluster; (iv) frequent pattern (FP)-growth algorithm is applied to each cluster to mine data patterns and association rules; (v) a set of suggestions are provided on the basis of the extracted features, data patterns, and rules. Different parameters are adjusted to achieve the best results for clustering and association rule mining algorithms. For the experiment, approximately 70,000 realworld problem-solving data from 537 students of a programming course (Algorithm and Data Structures) were used. In addition, synthetic data have leveraged for experiments to demonstrate the performance of MK-means algorithm. The experimental results show that the proposed framework effectively extracts useful features, patterns, and rules from problem-solving data. Moreover, these extracted features, patterns, and rules highlight the weaknesses and the scope of possible improvements in programming learning.

show abstract

Section: Oj Systemsmentioning

confidence: 99%

Educational Data Mining to Support Programming Learning Using Problem-Solving Data

et al. 2022

View full text Add to dashboard Cite

show abstract

“…This study showed that numerous useful hidden features can be extracted by analyzing the M2B system's data. Mekterović et al [12] proposed an APA system for conducting programming courses and created the educational software Edgar to automatically evaluate programming assignments and other programming-related tasks. Edgar provides a variety of services, including content writing, course administration, system monitoring, and troubleshooting.…”

Section: A Online Programming Learning Platformmentioning

confidence: 99%

“…Today, OJs are used by many educational institutions to conduct courses related to programming, computing, and software engineering [10], [11]. Many universities have created their own automated program assessment (APA) systems for programming courses to accelerate students' learning [12]- [14]. As a result, a large number of programmingrelated submission logs are created every day by OJ or APA systems in various organizations worldwide, which can be valuable resources for research and analysis [15], [16].…”

Section: Introductionmentioning

confidence: 99%

Impact of Practical Skills on Academic Performance: A Data-Driven Analysis

Rahman¹,

Watanobe

Kiran

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Most academic courses in information and communication technology (ICT) or engineering disciplines are designed to improve practical skills; however, practical skills and theoretical knowledge are equally important to achieve high academic performance. This research aims to explore how practical skills are influential in improving students' academic performance by collecting real-world data from a computer programming course in the ICT discipline. Today, computer programming has become an indispensable skill for its wide range of applications and significance across the world. In this paper, a novel framework to extract hidden features and related association rules using a real-world dataset is proposed. An unsupervised k-means clustering algorithm is applied for data clustering, and then the frequent pattern-growth algorithm is used for association rule mining. We leverage students' programming logs and academic scores as an experimental dataset. The programming logs are collected from an online judge (OJ) system, as OJs play a key role in conducting programming practices, competitions, assignments, and tests. To explore the correlation between practical (e.g., programming, logical implementations, etc.) skills and overall academic performance, the statistical features of students are analyzed and the related results are presented. A number of useful recommendations are provided for students in each cluster based on the identified hidden features. In addition, the analytical results of this paper can help teachers prepare effective lesson plans, evaluate programs with special arrangements, and identify the academic weaknesses of students. Moreover, a prototype of the proposed approach and data-driven analytical results can be applied to other practical courses in ICT or engineering disciplines.

show abstract

“…Programming skills are developed primarily through repetitive practice, and many universities [2][3][4][5] have created their own programming learning platforms to facilitate such practice by their students. These platforms are often used for programming competitions and serve as automated assessment tools for programming courses [6].…”

Section: Introductionmentioning

confidence: 99%

A Bidirectional LSTM Language Model for Code Evaluation and Repair

2021

View full text Add to dashboard Cite

Programming is a vital skill in computer science and engineering-related disciplines. However, developing source code is an error-prone task. Logical errors in code are particularly hard to identify for both students and professionals, and a single error is unexpected to end-users. At present, conventional compilers have difficulty identifying many of the errors (especially logical errors) that can occur in code. To mitigate this problem, we propose a language model for evaluating source codes using a bidirectional long short-term memory (BiLSTM) neural network. We trained the BiLSTM model with a large number of source codes with tuning various hyperparameters. We then used the model to evaluate incorrect code and assessed the model’s performance in three principal areas: source code error detection, suggestions for incorrect code repair, and erroneous code classification. Experimental results showed that the proposed BiLSTM model achieved 50.88% correctness in identifying errors and providing suggestions. Moreover, the model achieved an F-score of approximately 97%, outperforming other state-of-the-art models (recurrent neural networks (RNNs) and long short-term memory (LSTM)).

show abstract

Building a Comprehensive Automated Programming Assessment System

Cited by 51 publications

References 37 publications

Educational Data Mining to Support Programming Learning Using Problem-Solving Data

Educational Data Mining to Support Programming Learning Using Problem-Solving Data

Impact of Practical Skills on Academic Performance: A Data-Driven Analysis

A Bidirectional LSTM Language Model for Code Evaluation and Repair

Contact Info

Product

Resources

About