Learning better IV&V practices

Abstract

After data mining National Aeronautics and Space Administration (NASA) independent verification and validation (IV&V) data, we offer (a) an early life cycle predictor for project issue frequency and severity; (b) an IV&V task selector (that used the predictor to find the appropriate IV&V tasks); and (c) pruning heuristics describing what tasks to ignore, if the budget cannot accommodate all selected tasks. In ten-way cross-validation experiments, the predictor performs very well indeed: the average f -measure for predicting four classes of issue severity was over 0.9. This predictor is built using public-domain data and software. To the best of our knowledge, this is the first reproducible report of a predictor for issue frequency and severity that can be applied early in the life cycle.

Privacy and utility for defect prediction: Experiments with MORPH

“…Hence, those contractors are very reluctant to share data about (say) the effort associated with finding mission-critical errors in manned spacecraft, lest that data is used against them during the bidding process. Consequently, researchers and NASA managers suffer from a severe "data drought" that inhibits their ability to learn effective defect detection models for life-threatening software errors [4].…”

Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle

“…For example, Jones and Bonsignour report 52% of pretest defects removed before entering implementation, for large projects that focus on upfront defect removal techniques [48]. NASA robotics projects had a slightly higher percentage (58%) of defects removed before implementation began, although these had invested in independent verification and validation on top of other forms of defect removal [65]. Figure 13 and Figure 14 show the 50th and 90th percentile (respectively) of the time spent resolving issues (note that, in TSP, when developers see issues, they enter review or inspect or test until that issue is retired).…”

On the shoulders of giants

“…As sharing becomes more commonplace, we will be in the position of having a multitude of possible techniques for independently validating and verifying (V&V) results with shared data, but no clear guidance on when to use one over another. One of us, Tim Menzies et al [14] searched the literature for any study comparatively evaluating V&V techniques in the current IEEE standard V&V without success. Research into this area will help our field converge towards a standard practice for replication.…”