Maximising the information gained from a study of static analysis technologies for concurrent software

Abstract

The results of empirical studies in Software Engineering are limited to particular contexts, difficult to generalise and the studies themselves are expensive to perform. Despite these problems, empirical studies can be made effective and they are important to both researchers and practitioners. The key to their effectiveness lies in the maximisation of the information that can be gained by examining and replicating existing studies and using power analyses for an accurate minimum sample size. This approach was applied in a controlled experiment examining the combination of automated static analysis tools and code inspection in the context of the verification and validation (V&V) of concurrent Java components. The paper presents the results of this controlled experiment and shows that the combination of automated static analysis and code inspection is cost-effective. Throughout the experiment a strategy to maximise the information gained from the experiment was used. As a result, despite the size of the study, conclusive results were obtained, contributing to the research on V&V technology evaluation.

Trends in the Quality of Human-Centric Software Engineering Experiments--A Quasi-Experiment

“…Our search process found six of the eight studies, missing one paper where a human-centric experiment was only a small part of the validation exercise [13] and one paper we judged to be technology-centric [28].…”

Test case generation for the task tree type of architecture

“…The authors of [27] rightly advocate that a combination of different methods should be used for the verification and validation of concurrent programs. Further on, in their paper, they combine two particular methods, namely automated static analysis and manual code inspection to verify and validate concurrent Java components.…”

Experience with a Concurrency Bugs Benchmark

“…Testing concurrent components by combining code inspection and static analysis [2] and static [5] and dynamic [3][4], [28] tools for race detection and finding atomicity violations. Random test generation [13].…”