A Differential Testing Approach for Evaluating Abstract Syntax Tree Mapping Algorithms

Abstract

syntax tree (AST) mapping algorithms are widely used to analyze changes in source code. Despite the foundational role of AST mapping algorithms, little effort has been made to evaluate the accuracy of AST mapping algorithms, i.e., the extent to which an algorithm captures the evolution of code. We observe that a program element often has only one best-mapped program element. Based on this observation, we propose a hierarchical approach to automatically compare the similarity of mapped statements and tokens by different algorithms. By performing the comparison, we determine if each of the compared algorithms generates inaccurate mappings for a statement or its tokens. We invite 12 external experts to determine if three commonly used AST mapping algorithms generate accurate mappings for a statement and its tokens for 200 statements. Based on the experts' feedback, we observe that our approach achieves a precision of 0.98-1.00 and a recall of 0.65-0.75. Furthermore, we conduct a large-scale study with a dataset of ten Java projects containing a total of 263,165 file revisions. Our approach determines that GumTree, MTDiff and IJM generate inaccurate mappings for 20%-29%, 25%-36% and 21%-30% of the file revisions, respectively. Our experimental results show that state-of-the-art AST mapping algorithms still need improvements.

Regression test prioritization leveraging source code similarity with tree kernels

“…Several proposed techniques in this field leverage different structural models to predict the impact of code changes on an evolving software, often involving graph or tree‐based structures 8,51 . ASTs are among the most widely used models in literature 52,53 and are often used in Software Evolution Understanding and Code Summarization domains. More recently, deep neural network approaches converting the changes in the code to an automated embeddings representation have been proposed 54 and have been applied to task as log message generation and code patch prediction.…”

Software Defect Prediction using Deep Learning by Correlation Clustering of Testing Metrics

“…After then, the Bidirectional Long Short-Term Memory (Bi-LSTM) system will retain the most important features while ignoring the less important features to improve the accuracy of software defect prediction. Fan et al [16] developed a hierarchical method to automatically evaluate the degree of similarity between mapped statements and tokens using a variety of distinct algorithms. The author was able to identify whether or not each of the compared algorithms provides erroneous mappings for a sentence or the tokens that make up the claim by doing the comparison.…”