On the "naturalness" of buggy code

Abstract

Real software, the kind working programmers produce by the kLOC to solve real-world problems, tends to be "natural", like speech or natural language; it tends to be highly repetitive and predictable. Researchers have captured this naturalness of software through statistical models and used them to good effect in suggestion engines, porting tools, coding standards checkers, and idiom miners. This suggests that code that appears improbable, or surprising, to a good statistical language model is "unnatural" in some sense, and thus possibly suspicious. In this paper, we investigate this hypothesis. We consider a large corpus of bug fix commits (ca. 7,139), from 10 different Java projects, and focus on its language statistics, evaluating the naturalness of buggy code and the corresponding fixes. We find that code with bugs tends to be more entropic (i.e. unnatural), becoming less so as bugs are fixed. Ordering files for inspection by their average entropy yields cost-effectiveness scores comparable to popular defect prediction methods. At a finer granularity, focusing on highly entropic lines is similar in cost-effectiveness to some well-known static bug finders (PMD, FindBugs) and ordering warnings from these bug finders using an entropy measure improves the cost-effectiveness of inspecting code implicated in warnings. This suggests that entropy may be a valid, simple way to complement the effectiveness of PMD or FindBugs, and that search-based bug-fixing methods may benefit from using entropy both for fault-localization and searching for fixes.

DeepBugs: a learning approach to name-based bug detection

“…Even though it has been noted that buggy code stands out compared to non-buggy code [Ray et al 2016], little work exists on automatically detecting bugs via machine learning. Murali et al train a recurrent neural network that probabilistically models sequences of API calls and then use it for finding incorrect API usages [Murali et al 2017].…”

Easy over hard: a case study on deep learning

“…We study simplicity since it is very useful to replace N methods with M ≪ N methods, especially when the results from the many are no better than the few. A bewildering array of new methods for software quality prediction are reported each year (some of which rely on intimidatingly complex mathematical methods) such as deep belief net learning [50], spectral-based clustering [55], and n-gram language models [46]. Ghotra et al list dozens of different data mining algorithms that might be used for defect predictors [16].…”

Lessons Learned from Using a Deep Tree-Based Model for Software Defect Prediction in Practice

“…This is done by aggregating those outputs from the descendants, i.e. calling t-lstm() recursively on the children nodes (lines [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26]. This function first obtains the embedding w t of the input AST node t (using ast2vec as discussed in Section 4.2).…”

“…[8]) and the line level (e.g. [24]). Since our approach is able to learn features at the code token level, it may work at those finer level of granularity.…”