Clue: A Clustering-Based Test Reduction Approach for Software Product Lines

Abstract

Nowadays, organizations have increasingly turned to software product lines (SPLs)/configurable systems to meet diverse user needs and market demands. SPLs offer configurability and scalability but pose a formidable challenge in testing. Assuring the quality of variants across numerous configurations demands innovative testing strategies that balance effectiveness and efficiency. To improve testing productivity, test reduction techniques have been widely employed in non-configurable code. However, test reduction for SPL systems remains mostly unexplored. This paper introduces CLUE, a novel test reduction approach to enhance testing productivity in SPLs. Our idea is that to effectively and efficiently reveal failures, tests should be distinctive and cover diverse behaviors of the system. Meanwhile, similar tests covering the same/similar behaviors of an SPL system do not need to be redundantly executed multiple times. CLUE clusters the system’s tests into distinctive groups containing similar tests. Within each cluster, tests are prioritized based on the number of feature interactions, a common root cause of defects in SPLs. CLUE continually selects and executes the top-prioritized test in each cluster until a bug is revealed or the allocated effort budget is exhausted. To evaluate CLUE, we conducted several experiments on a dataset of six widely-used SPL systems. The results show that CLUE enables developers to detect defects earlier, requiring up to 88% less effort than existing approaches. Additionally, using only 50% of the original test suites, CLUE can identify most of the bugs in the buggy SPL systems while maintaining fault localization performance.