Multi-objective test case prioritization in highly configurable systems: A case study

Parejo, José Antonio; Sánchez, Ana María Sánchez; Segura, Sergio; Ruiz–Cortés, Antonio; Lopez-Herrejon, Roberto E.; Egyed, Alexander

doi:10.1016/j.jss.2016.09.045

Cited by 79 publications

(54 citation statements)

References 68 publications

(126 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Medeiros et al Parejo et al [60] extend Sánchez et al [70] work by defining multi-objectives test case selection and prioritization. The algorithm starts with a set of configurations samples, each satisfying 2-wise coverage, and evolves them in order to produce one prioritized configuration sample that maximize the defined criteria.…”

Section: Selected Studiesmentioning

confidence: 99%

“…Drupal (PHP modules based Web content management system) Parejo et al [60] Multi-objective Drupal (PHP modules based Web content management system) Souto et al [76] random, one-enabled, one-disabled, most-enableddisabled and pairwise computed from SPLat [44] 8 small SPLs + GCC (50 most used options). Samplings' sizes and number of failures were considered.…”

Section: Pairwisementioning

confidence: 99%

See 1 more Smart Citation

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

et al. 2018

View full text Add to dashboard Cite

Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations. This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them if the effort required to do so remains acceptable. Not only this: we believe there is a lot to be learned by systematically and exhaustively testing a configurable system. In this case study, we report on the first ever endeavour to test all possible configurations of the industry-strength, open source configurable software system: JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time. We find that 35.70% configurations fail and we identify the feature interactions that cause the errors. We show that sampling strategies (like A. Halin, G. Perrouin (FNRS research associate) PReCISE, dissimilarity and 2-wise): (1) are more effective to find faults than the 12 default configurations used in the JHipster continuous integration; (2) can be too costly and exceed the available testing budget. We cross this quantitative analysis with the qualitative assessment of JHipster's lead developers.

show abstract

Section: Selected Studiesmentioning

confidence: 99%

Section: Pairwisementioning

confidence: 99%

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

et al. 2018

View full text Add to dashboard Cite

show abstract

“…The first experiment varies the number of features per MPL, whilst the second experiment varies the density of constraints among product lines. Lastly, the third experiment considers four product lines taken from the current state-of-the-art, namely DecISIonAL [10] , Drupal [30] , E-Shop [15] , and Web Portal [21] models. In all cases we measure the resolution time of product configuration.…”

Section: Rq1mentioning

confidence: 99%

“…The Web Portal 7 FM represents the variability in the implementation of web systems [21] . Finally, the Drupal 8 FM represents the variability of a content management system [30] . Studied FMs with their corresponding number of features, CTCs, CTC density, and number of products are presented in Table 4 .…”

Section: Subjects Design and Variablesmentioning

confidence: 99%

Constraint programming heuristics for configuring optimal products in multi product lines

Ochoa

González-Rojas

Cardozo

et al. 2019

Information Sciences

View full text Add to dashboard Cite

a b s t r a c tNowadays, complex application domains require configuring multi-product lines where product features and constraints among them are specified in several variability models. These variability models are enriched with inter-model constraints representing the existing relations among domain concerns, and with non-functional properties modeled as attributes attached to product features. Multiple techniques use constraint programming to automate the cumbersome task of manually configuring a suitable product. Currently, there are some proposals to improve the performance of constraint solvers when configuring single-model product lines, however configuration scenarios with multiple interrelated and attributed models are not yet targeted. This paper proposes and evaluates three search heuristics used to configure optimal products regarding multi-objective criteria. Results are compared against the default search strategy of the Choco constraint solver. We evaluated the performance for configuring optimal products in four state-of-the-art product lines and 130 generated variability models representing multi-product lines that scale up to 6400 features and 960 constraints. As a result, we observe that the proposed heuristics perform better than the default solver strategy when the variability models scale in terms of features. In contrast, the default strategy and one of the proposed heuristics perform better as the number of interdependencies between variability models increases.

show abstract

“…Thus, the problem is to find a test suite (sequence of valid products) that is likely to be a good solution (in terms of selection and prioritisation). It has been observed that a number of properties of good test suites can be captured by objective functions that map a test suite to a value that represents how 'good' this test suite is according to the properties (see, for example [19,23,29,31,32,48,51,58,61,71,72,74]). For example, a fault might be associated with the interaction of a pair of features and so we might want to test as many such interactions as possible (pairwise coverage).…”

Section: Introductionmentioning

confidence: 99%

Many-Objective Test Suite Generation for Software Product Lines

Hierons

Liu

et al. 2020

ACM Trans. Softw. Eng. Methodol.

Self Cite

View full text Add to dashboard Cite

Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document. When citing, please reference the published version. Take down policy While the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has been uploaded in error or has been deemed to be commercially or otherwise sensitive.

show abstract

Multi-objective test case prioritization in highly configurable systems: A case study

Cited by 79 publications

References 68 publications

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

Constraint programming heuristics for configuring optimal products in multi product lines

Many-Objective Test Suite Generation for Software Product Lines

Contact Info

Product

Resources

About