On the effectiveness of early life cycle defect prediction with Bayesian Nets

Abstract

Standard practice in building models in software engineering normally involves three steps: collecting domain knowledge (previous results, expert knowledge); building a skeleton of the model based on step 1 including as yet unknown parameters; estimating the model parameters using historical data. Our experience shows that it is extremely difficult to obtain reliable data of the required granularity, or of the required volume with which we could later generalize our conclusions. Therefore, in searching for a method for building a model we cannot consider methods requiring large volumes of data. This paper discusses an experiment to develop a causal model (Bayesian net) for predicting the number of residual defects that are likely to be found during independent testing or operational usage. The approach supports (1) and (2), does not require (3), yet still makes accurate defect predictions (an R 2 of 0.93 between predicted and actual defects). Since our method does not Empir Software Eng (2008) require detailed domain knowledge it can be applied very early in the process life cycle. The model incorporates a set of quantitative and qualitative factors describing a project and its development process, which are inputs to the model. The model variables, as well as the relationships between them, were identified as part of a major collaborative project. A dataset, elicited from 31 completed software projects in the consumer electronics industry, was gathered using a questionnaire distributed to managers of recent projects. We used this dataset to validate the model by analyzing several popular evaluation measures (R 2 , measures based on the relative error and Pred). The validation results also confirm the need for using the qualitative factors in the model. The dataset may be of interest to other researchers evaluating models with similar aims. Based on some typical scenarios we demonstrate how the model can be used for better decision support in operational environments. We also performed sensitivity analysis in which we identified the most influential variables on the number of residual defects. This showed that the project size, scale of distributed communication and the project complexity cause the most of variation in number of defects in our model. We make both the dataset and causal model available for research use.

The role of Artificial Intelligence in Software Engineering

“…One example of a probabilistic AI technique that has proved to be highly applicable in Software Engineering has been the use of Bayesian probabilistic reasoning to model software reliability [5], one of the earliest [6] examples of the adoption of what might be called, perhaps with hindsight, 'AI for SE'. Another example of the need for probabilistic reasoning comes from the analysis of users, inherently requiring an element of probability because of the stochastic nature of human behaviour [7].…”

Guest editorial: learning to organize testing

“…• Despite much effort on data mining and defects, most of that work achieves similar conclusions (Lessmann et al 2008); • Data mining data is fundamentally less important than discussing those effects with the users (Fenton et al 2008).…”

Rethinking Experiments in a Socio-Technical Perspective: The Case of Software Engineering

“…This endeavor is part of a larger research endeavor in software metrics, whose "ultimate goal (...) is to help project managers make decisions under uncertainty." [13] (p. 500). Finally, the Reinhartz-Berger experiment compared two modeling techniques, namely object-process methodology (OPM) and UML, to establish the level of comprehension and the quality of the constructed models in the context of web applications.…”

“…• The causal model used in the Fenton experiment [13] was comprised of variables and relationships identified by means of questionnaires distributed to a number of managers involved in 31 software projects in the consumer electronics industry.…”

“…Our focus is on those endeavors that adopted the most rigorous methodology from the scientific tradition, namely the controlled experiment, so that we are able to observe the widest range of instruments for scientific investigation put to the test in the context of SE. Some of these papers [9][10][11][12] explicitly refer to the presented work as a "controlled experiment"; others [13][14][15] only speak of an "experiment", but due to the rigorous procedures adopted, can be considered equivalent from a methodological point of view. In particular, we selected these works because they present the results of controlled experiments performed with human subjects, which makes them examples of one of the most challenging methodological endeavors, where the most rigorous (and traditional) experimental procedures are applied in a context where the human factor plays a primary role and brings in its typical unforeseeable and disruptive effects.…”