Source Code Assessment and Classification Based on Estimated Error Probability Using Attentive LSTM Language Model and Its Application in Programming Education

Abstract

The rate of software development has increased dramatically. Conventional compilers cannot assess and detect all source code errors. Software may thus contain errors, negatively affecting end-users. It is also difficult to assess and detect source code logic errors using traditional compilers, resulting in software that contains errors. A method that utilizes artificial intelligence for assessing and detecting errors and classifying source code as correct (error-free) or incorrect is thus required. Here, we propose a sequential language model that uses an attention-mechanism-based long short-term memory (LSTM) neural network to assess and classify source code based on the estimated error probability. The attentive mechanism enhances the accuracy of the proposed language model for error assessment and classification. We trained the proposed model using correct source code and then evaluated its performance. The experimental results show that the proposed model has logic and syntax error detection accuracies of 92.2% and 94.8%, respectively, outperforming state-of-the-art models. We also applied the proposed model to the classification of source code with logic and syntax errors. The average precision, recall, and F-measure values for such classification are much better than those of benchmark models. To strengthen the proposed model, we combined the attention mechanism with LSTM to enhance the results of error assessment and detection as well as source code classification. Finally, our proposed model can be effective in programming education and software engineering by improving code writing, debugging, error-correction, and reasoning.

A Neural Network Based Intelligent Support Model for Program Code Completion

“…Rahman et al [14] proposed a language model using LSTM for fixing source code errors. e proposed model is a combined attention mechanism with LSTM which increases the effectiveness of standard LSTM.…”

“…If the direction of a hidden state Δh t is moved to v i by any modifications, then the gradient will be λ i Δh t . From equation (14) the product of the Jacobians of the hidden state sequences is λ 1 i…”

“…As of May 2020, about 75,000 users are regularly playing their programming activities on the AOJ platform, with 2100 autonomous problem sets. All problem sets are classified based on different algorithms and branches of computer science [14]. As a result, about 4.5 million massive solution source codes have been archived on the AOJ platform, encouraging better research in the field of software engineering.…”

“…Before we conducted training, raw source codes were filtered by removing unnecessary elements. To accomplish this, we followed the procedure applied to [14] for source code embedding and tokenization. First, we removed all irrelevant elements from codes like lines (n), comments, and tabs (\t).…”

“…Our proposed model created the probability for each word. We considered a word will be an error candidate whose probability is below 0.1 [14]. Additionally, to test the model loss function, we calculated the cross-entropy for each epoch at the softmax layer.…”