Variability testing in the wild: the Drupal case study

Abstract

Variability testing techniques search for effective and manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in realistic settings is a must, but challenging due to the lack of variability intensive systems with available code, automated tests and fault reports. In this article, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability using a feature model. Then, we report on extensive non-functional data extracted from the Drupal Git repository and the Drupal issue tracking system. Among other results, we identified 3,392 faults in single features and 160 faults triggered by the interaction of up to 4 features in Drupal v7.23. We also found positive correlations relating the number of bugs in Drupal features to their size, cyclomatic complexity, number of changes and fault history. To show the feasibility of our work, we evaluated the effectiveness of non-functional data for test case prioritization in Drupal. Results show that non-functional attributes are effective at accelerating the detection of faults, outperforming related prioritization criteria as test case similarity.

Multi-objective test case prioritization in highly configurable systems: A case study

“…Functional prioritization objectives are those based on the functional features of the system and their interactions. Some examples are those based on combinatorial interaction testing [75], configuration dissimilarity [2,33,60] or feature model complexity metrics [59,60]. Non-functional prioritization objectives consider extra-functional information such as user preferences [21,39], cost [75], memory consumption [45] or execution probability [15] to find the best ordering for test cases.…”

“…Non-functional prioritization objectives consider extra-functional information such as user preferences [21,39], cost [75], memory consumption [45] or execution probability [15] to find the best ordering for test cases. In a previous work [59], we performed a preliminary evaluation comparing the effectiveness of several functional and non-functional prioritization objectives in accelerating the detection of faults in an HCS. Results suggested that non-functional properties such as the number of changes or the number of defects in a previous version of the system were among the most effective prioritization criteria.…”

“…Current approaches for test case prioritization in HCSs follow a single objective approach [2,39,15,21,33,45,59], that is, they either aim to maximize or minimize an objective (e.g. feature coverage) or another (e.g.…”

“…Then, we report a comparison of 63 different combinations of up to three objectives in accelerating the detection of faults in the Drupal framework. Drupal is a highly modular open source web content management system for which we have mined a feature model and extracted real data from its issue tracking system and Git repository [59]. Results reveal that non-functional properties, such as the number of defects in previous versions of the system, accelerate the detection of faults more effectively than functional properties extracted from the feature model.…”

“…In this section, we present the Drupal case study fully reported by the authors in a previous work [59]. Drupal is a highly modular open source web content management framework written in PHP [8,67].…”