Logical Segmentation of Source Code

Abstract

Many software analysis methods have come to rely on machine learning approaches. Code segmentation -the process of decomposing source code into meaningful blockscan augment these methods by featurizing code, reducing noise, and limiting the problem space. Traditionally, code segmentation has been done using syntactic cues; current approaches do not intentionally capture logical content. We develop a novel deep learning approach to generate logical code segments regardless of the language or syntactic correctness of the code. Due to the lack of logically segmented source code, we introduce a unique data set construction technique to approximate ground truth for logically segmented code. Logical code segmentation can improve tasks such as automatically commenting code, detecting software vulnerabilities, repairing bugs, labeling code functionality, and synthesizing new code.

Literature survey of deep learning-based vulnerability analysis on source code

“…Among the reviewed works several of them use some form of RNNs. classical RNNs [86 ], long short‐term memory (LSTM) [16, 67, 69, 70, 73, 76, 83 ], contextual long short‐term memory (CLSTM) [72 ], bidirectional recurrent neural networks (BRNNs) [17, 74 ], bidirectional gated recurrent unit (BGRU) [83 ], and bidirectional long short‐term memory (BLSTM) [14, 15, 19, 77, 79, 82–85 ].…”

“…This issue has been highlighted by several reviewed papers as a limitation that can be explored in the future. On several of the surveyed works the programming language of the data set focus ranges from C/C++ [14, 15, 17, 19, 69–72, 77–87 ] and JavaScript [18, 60, 79 ]. Having a data set that consists of multiple programming languages would require further exploration.…”

“…This research direction involves using multiple programming language data sets for a DL model and studying their impact on different metrics for vulnerability detection. This direction of vulnerability detection was studied in the work of [79, 82 ]. The authors explored the viability of this approach by using operational semantics of a data set of C/C++ and Python files.…”

“…The discovery of these lines of code with a higher accuracy allows for the focus on those specific code lines since the probability of its vulnerability is higher. This vulnerability aspect is focused on by several of the surveyed papers [15, 17, 78–87 ] Table 7.…”

“…Examples include Lexed code representation and using the source code files directly. Among the reviewed papers, several rely on the use of text‐based representations of source code [60, 67, 68, 78, 79, 86 ] (d) Mixed features‐based representation: This category includes recent studies that combined the abovementioned three types of feature representations. Among the reviewed papers, several rely on the use of mixed feature‐based representations of source code [19, 69, 81, 84 ].…”