Making testing product lines more efficient by improving the testability of product line architectures

Abstract

Product line engineering is a recent approach to software development that has shown to enable organizations to achieve significant reductions in development and maintenance cost as well as time-to-market of increasingly complex software systems. Yet, the testing process has not kept up with these reductions and the relative cost for testing product lines is actually becoming higher than in traditional single system development. Also, testing often cannot keep pace with accelerated development in product line engineering due to technical and organizational issues. This paper advocates that testing of product lines can be made more efficient and effective by considering testability already during architectural design. It explores the relationship between testability and product line architecture and discusses the importance of high testability for reducing product line testing effort and achieving required coverage criteria. The paper also outlines a systematic approach that will support product line organizations in improving and evaluating testability of product lines at the architectural level

Improving the Testing and Testability of Software Product Lines

“…Others have argued that variability decreases testability [15], but we believe that there should be a finer grained examination of this argument. Both optional and alternative features can be viewed as points of variability in a software product line, yet we believe they may behave differently from a black box testing approach and provide different opportunities to reduce testing effort, as we explain next.…”

“…A large body of research on SPL engineering has focused on reuse of core program assets [4,15,17], refined feature modeling [8,9,23], and code generation techniques [2,7]. There has also been research on testing software product lines [3,6,10,24].…”

“…Kolb and Muthig [15] point out that testing has not made the same advances as other parts of the SPL lifecyle and remains a bottleneck in SPL development. Their work highlights issues related to testability of SPLs, where testability is viewed as the ease with which one can incorporate testing into the development cycle and increase reuse while retaining a high rate of fault detection.…”

“…Their work highlights issues related to testability of SPLs, where testability is viewed as the ease with which one can incorporate testing into the development cycle and increase reuse while retaining a high rate of fault detection. They comment that the primary strength of SPL development, variability, also has the greatest impact on reducing testability [15], due to the combinatorial explosion of feature combinations that occurs as variability increases [6,17]. In other related work, Jaring et al [13] point out that the testability of a product line can be viewed as a function of the binding time of variability, and that providing early binding can increase the ability to test products early.…”

A systematic mapping study of software product lines testing

“…Regarding dynamic analysis, some studies [29,47] recommend the V-model phases, commonly used in single-systems, to structure a series of dynamic analysis. The V-model gives equal weight to development and testing rather than treating testing as an afterthought [25].…”