Comparing detection methods for software requirements inspections: a replicated experiment

Abstract

Software requirements speci cations (SRS) are often validated manually. One such process is inspection, in which several reviewers independently analyze all or part of the speci cation and search for faults. These faults are then collected at a meeting of the reviewers and author(s). Usually, reviewers use Ad Hoc or Checklist methods to uncover faults. These methods force all reviewers to rely on nonsystematic techniques to search for a wide variety of faults. We hypothesize that a Scenario-based method, in which each reviewer uses di erent, systematic techniques to search for di erent, speci c classes of faults, will have a signi cantly higher success rate. We evaluated this hypothesis using a 3 2 4 partial factorial, randomized experimental design. Forty eight graduate students in computer science participated in the experiment. They were assembled into sixteen, three-person teams. Each team inspected two SRS using some combination of Ad Hoc, Checklist or Scenario methods. For each inspection we performed four measurements: (1) individual fault detection rate, (2) team fault detection rate, (3) percentage of faults rst identi ed at the collection meeting (meeting gain rate), and (4) percentage of faults rst identi ed by an individual, but never reported at the collection meeting (meeting loss rate). The experimental results are that (1) the Scenario method had a higher fault detection rate than either Ad Hoc or Checklist methods, (2) Scenario reviewers were more e ective at detecting the faults their scenarios are designed to uncover, and were no less e ective at detecting other faults than both Ad Hoc or Checklist reviewers, (3) Checklist reviewers were no more e ective than Ad Hoc reviewers, and (4) Collection meetings produced no net improvement in the fault detection rate { meeting gains were o set by meeting losses.

Requirements engineering: In search of the dependent variables

“…Failure of V&V can be an indication of a poor requirements engineering process, e.g., the requirements could be too poor to serve as a basis for cre- ating test cases. Inspections of requirements can be good way to ascertain and in fact measure the inherent testability of requirements [37][38][39][40]. (3) Evolving requirements.…”

A Framework for Software Engineering Experimental Replications

“…A large body of studies has demonstrated that reading techniques are effective for improving individual review practices in different domains and types of inspection, e.g. natural language requirements [3], requirements in formal notation [20], high-level designs [14], code [13,15], and user interfaces [32].…”

An experimental comparison of checklist-based reading and perspective-based reading for UML design document inspection

“…Real team meetings [7,12,15,18] as well as virtual teams [1,3,20] are reported in literature. We decided to use virtual teams because we were more concerned with the range of team's defect coverage than with issues of interaction between members.…”

“…Inspections have been extensively investigated through controlled experiments in university environment and industry case studies. However, in most cases software inspections have been used for defect detection in documents of conventional structured development process, such as functional requirement documents or code modules [1,13,15,20]. There is a significant lack of information about how inspections should be applied to Object-Oriented (subsequently denoted OO) artefacts, such as OO code and design diagrams, because inspections were developed when the structured development process was dominant.…”

“…Several variants of scenario-based reading have been proposed: Defect-based reading [15], Reading technique based on function points [5] and Perspective-based reading (subsequently denoted PBR) [1,12,13,14]. The Defect-based reading concentrates on specific defect classes.…”