A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems

Abstract

Background: Software engineering is searching for general principles that apply across contexts, for example, to help guide software quality assurance. Fenton and Ohlsson presented such observations on fault distributions, which have been replicated once. Objectives: We aimed to replicate their study again to assess the robustness of the findings in a new environment, five years later. Method: We conducted a literal replication, collecting defect data from five consecutive releases of a large software system in the telecommunications domain, and conducted the same analysis as in the original study. Results: The replication confirms results on unevenly distributed faults over modules, and that fault proneness distributions persist over test phases. Size measures are not useful as predictors of fault proneness, while fault densities are of the same order of magnitude across releases and contexts. Conclusions: This replication confirms that the uneven distribution of defects motivates uneven distribution of quality assurance efforts, although predictors for such distribution of efforts are not sufficiently precise.

A quantitative analysis of the unit verification perspective on fault distributions in complex software systems: an operational replication

“…Their study was aimed at studying empirical principles that are widely used by software engineering practitioners. The quantitative analysis performed in that study and their two replication studies (Andersson and Runeson 2007;Galinac Grbac et al 2013) provide support for the following:…”

“…This study has been replicated twice by Andersson and Runeson (2007) and by Galinac Grbac et al (2013). The three studies were all performed in a closed proprietary software development within the telecommunications domain.…”

“…In the first replication Andersson and Runeson (2007) published in 2007 and the second replication Galinac Grbac et al (2013) published in 2014, the set of hypotheses is refined. The full list of hypotheses and a summary of results are already discussed in the second replication study (Galinac Grbac et al 2013). However, for the clarity of presentation, we repeat a short version of Table 10 from Galinac Grbac et al (2013) in Table 1.…”

“…Although the initial motivation by Fenton and Ohlsson (2000) was to verify the Pareto assumptions of fault behavior, the study and its replications (Andersson and Runeson 2007;Galinac Grbac et al 2013) focused the analysis only on the faults from testing the complete system. They analyzed only the faults that caused failures in the testing phases running the complete system, such as FT, ST and SI phases (here referred to as later verification (LV) phases), and compared them to the faults that caused failures in operation at the customer site (OP).…”

“…Since these studies hypothesized that fault data from earlier phases may be used as predictors, we have identified a possibility to broaden the quantitative analysis by involving data from the UV phase and compare them to data from LV phases, see Fig.1. We therefore launch a study that uses the same operationalization (i.e., hypotheses, metrics, analyses) as the Fenton and Ohlsson (2000) study and its replications (Andersson and Runeson 2007;Galinac Grbac et al 2013), but change the hypothesis to focus on UV instead.…”