Quantitative analysis of faults and failures in a complex software system

Abstract

AbstractÐThe dearth of published empirical data on major industrial systems has been one of the reasons that software engineering has failed to establish a proper scientific basis. In this paper, we hope to provide a small contribution to the body of empirical knowledge. We describe a number of results from a quantitative study of faults and failures in two releases of a major commercial system. We tested a range of basic software engineering hypotheses relating to: The Pareto principle of distribution of faults and failures; the use of early fault data to predict later fault and failure data; metrics for fault prediction; and benchmarking fault data. For example, we found strong evidence that a small number of modules contain most of the faults discovered in prerelease testing and that a very small number of modules contain most of the faults discovered in operation. However, in neither case is this explained by the size or complexity of the modules. We found no evidence to support previous claims relating module size to fault density nor did we find evidence that popular complexity metrics are good predictors of either fault-prone or failure-prone modules. We confirmed that the number of faults discovered in prerelease testing is an order of magnitude greater than the number discovered in 12 months of operational use. We also discovered fairly stable numbers of faults discovered at corresponding testing phases. Our most surprising and important result was strong evidence of a counter-intuitive relationship between pre-and postrelease faults: Those modules which are the most fault-prone prerelease are among the least fault-prone postrelease, while conversely, the modules which are most faultprone postrelease are among the least fault-prone prerelease. This observation has serious ramifications for the commonly used fault density measure. Not only is it misleading to use it as a surrogate quality measure, but, its previous extensive use in metrics studies is shown to be flawed. Our results provide data-points in building up an empirical picture of the software development process. However, even the strong results we have observed are not generally valid as software engineering laws because they fail to take account of basic explanatory data, notably testing effort and operational usage. After all, a module which has not been tested or used will reveal no faults, irrespective of its size, complexity, or any other factor.

Using static analysis to determine where to focus dynamic testing effort

“…If n files were modified as the result of a failure, then this was counted as being n distinct faults. This is consistent with the convention used in References [8] and [3]. This implies that each fault was associated with exactly one file.…”

“…Most of the previous research in this area, including our earlier work [10], and that by other authors described in [1,2,3,5,8,9], was aimed at examining software systems to establish characteristics that may be associated with high incidences of faults. In this paper, we go beyond merely identifying characteristics and successfully build a statistical model that can predict the incidence of faults in future versions of a system.…”

ImpactScale: Quantifying change impact to predict faults in large software systems

“…Modules classified as fault-prone tend to be large because of the correlation between faults and size [3]. Arisholm et al reported the effort of testing a module is roughly proportional to the size of the module [20].…”

“…Metrics useful in pre-release phase may often become useless in postrelease phase. For example, Fenton et al [3] reported the case that McCabe's cyclomatic complexity correlated with prerelease faults but little correlated with post-release faults.…”

“…Fault prediction requires metrics correlated with faults as explanatory variables. Several studies have reported the set of faulty modules in pre-release phase differs from the set of ones in post-release phase [3] [4]. Metrics useful in pre-release phase may often become useless in postrelease phase.…”