Data Quality: Some Comments on the NASA Software Defect Datasets

Abstract

BACKGROUND -self evidently empirical analyses rely upon the quality of their data. Likewise replications rely upon accurate reporting and using the same rather than similar versions of data sets. In recent years there has been much interest in using machine learners to classify software modules into defectprone and not defect-prone categories. The publicly available NASA datasets have been extensively used as part of this research.OBJECTIVE -this short note investigates the extent to which published analyses based on the NASA defect data sets are meaningful and comparable. METHOD -we analyse the five studies published in IEEE Transactions on Software Engineering since2007 that have utilised these data sets and compare the two versions of the data sets currently in use.RESULTS -we find important differences between the two versions of the data sets, implausible values in one data set and generally insufficient detail documented on data set pre-processing.CONCLUSIONS -it is recommended that researchers (i) indicate the provenance of the data sets they use (ii) report any pre-processing in sufficient detail to enable meaningful replication and (iii) invest effort in understanding the data prior to applying machine learners.

Researcher Bias: The Use of Machine Learning in Software Defect Prediction

“…There are various mechanisms to facilitate this sharing of data, with the Promise Data Repository [11] being at the forefront of such initiatives. Whilst sharing of data is clearly a good thing it is not without risk, particularly when problems and errors in the data are propagated [12], [13]. However, it does afford us the opportunity to examine the impact of other factors upon defect prediction performance since many different researcher groups have used the same data.…”

Using evidential reasoning to make qualified predictions of software quality

“…One can safely assume that, with more contextual information about the developers and development processes involved etc., a completely different picture of software quality might emerge. However, the important point is that this assessment accurately captures the author's subjective assessment of the system, when limited to reasoning about a relatively restricted set of metrics of potentially questionable provenance [14]. As will be discussed later, the construction of a more systematic validation study is a part of our ongoing and future work.…”

“…Static analysis tools can fail to parse or resolve certain relations in the code, or data that has been collected by hand might apply to a different version of the source code than the one we are assessing. This is a salient point for the CM1 system, where Shepperd et al [14] have highlighted some important inconsistencies in the data-set over different studies that have utilised the data sets.…”

An Empirical Study on the Equivalence and Stability of Feature Selection for Noisy Software Defect Data

“…BENCHMARK DATASET To investigate the equivalence and the stability of the feature selection methods for noisy SDD, we used eight original version projects of NASA dataset and the corresponding clean version preprocessed by Shepperd et al [17] as our experimental dataset. NASA dataset is a method-level software defect dataset that is characterized by static code metrics [5].…”

“…For the generalization of our results, we carefully chose the NASA dataset which is commonly used in previous studies in software engineering domain [4], [17], [18], [19], [20]. Besides, previous work also conducted case studies on NASA dataset to investigate the effect of noise on SDD [8], [27].…”