Constrained Interaction Testing: A Systematic Literature Study

Ahmed, Bestoun S.; Zamli, Kamal Z.; Afzal, Wasif; Bureš, Miroslav

doi:10.1109/access.2017.2771562

Cited by 57 publications

(45 citation statements)

References 108 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…One can find two recent trends in research for combinatorial interaction testing: handling of constraints [28] and the application of meta-heuristic algorithms. Many current studies focus on the use of meta-heuristic algorithms as part of the greedy computational approach for CA construction [29,30,31].…”

Section: Combinatorial T − Wise Test Suite Generatorsmentioning

confidence: 99%

“…The Q-table will be updated accordingly based on the quality/performance of the current state-action pairs (lines 18-24). Like EMCQ, the Monte Carlo Metropolis probability controls the selection of search operators when the quality of the solution improves (lines [25][26][27][28][29][30]. This probability decreases with iteration (T ).…”

Section: Q-learning Monte Carlo Hyper-heuristic Strategymentioning

confidence: 99%

See 1 more Smart Citation

An evaluation of Monte Carlo-based hyper-heuristic for interaction testing of industrial embedded software applications

et al. 2020

Self Cite

View full text Add to dashboard Cite

Hyper-heuristic is a new methodology for the adaptive hybridization of meta-heuristic algorithms to derive a general algorithm for solving optimization problems. This work focuses on the selection type of hyper-heuristic, called the Exponential Monte Carlo with Counter (EMCQ). Current implementations rely on the memory-less selection that can be counterproductive as the selected search operator may not (historically) be the best performing operator for the current search instance. Addressing this issue, we propose to integrate the memory into EMCQ for combinatorial t − wise test suite generation using reinforcement learning based on the Q-learning mechanism, called Q-EMCQ. The limited application of combinatorial test generation on industrial programs can impact the use of such techniques as Q-EMCQ. Thus, there is a need to evaluate this kind of approach against relevant industrial software, with a purpose to show the degree of interaction required to cover the code as well as finding faults. We applied Q-EMCQ on 37 real-world industrial programs written in Function Block Diagram (FBD) language, which is used for developing a train control management system at Bombardier Transportation Sweden AB. The results of this study show that Q-EMCQ is an efficient technique for test case generation. Additionally, unlike the t-wise test suite generation, which deals with the minimization problem, we have also subjected Q-EMCQ to a maximization problem involving the general module clustering to demonstrate the effectiveness of our approach. The results show the Q-EMCQ is also capable of outperforming the original EMCQ as well as several recent meta/hyper-heuristic including Modified Choice Function (MCF), Tabu High Level Hyper-Heuristic(HHH), Teaching Learning based Optimization (TLBO), Sine Cosine Algorithm (SCA) and Symbiotic Optimization Search (SOS) in clustering quality

show abstract

Section: Combinatorial T − Wise Test Suite Generatorsmentioning

confidence: 99%

Section: Q-learning Monte Carlo Hyper-heuristic Strategymentioning

confidence: 99%

An evaluation of Monte Carlo-based hyper-heuristic for interaction testing of industrial embedded software applications

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Kuhn et al [5] broadly covered many practical aspects of unconstrained CIT. Ahmed et al [6] presented an intensive and systematic literature survey of constrained CIT, addressing the current generation strategies, tools, and applications. From the mentioned studies, it can be observed that there are three categories of generation approaches that can be classified based on generation techniques: greedy algorithms, metaheuristic search algorithms, and mathematical or algebraic methods.…”

Section: Related Workmentioning

confidence: 99%

“…strained CIT (CCIT) can be seen as a current trend and future direction of CIT [10,6,3]. CCIT is considered a part of CIT that is a nondeterministic polynomial time (NP-hard) problem [11].…”

mentioning

confidence: 99%

Generation and Application of Constrained Interaction Test Suites Using Base Forbidden Tuples with a Mixed Neighborhood Tabu Search

Hasan

Ahmed

Potrus

et al. 2020

Int. J. Soft. Eng. Knowl. Eng.

Self Cite

View full text Add to dashboard Cite

To ensure the quality of current highly configurable software systems, intensive testing is needed to test all the configuration combinations and detect all the possible faults. This task becomes more challenging for most modern software systems when constraints are given for the configurations. Here, intensive testing is almost impossible, especially considering the additional computation required to resolve the constraints during the test generation process. In addition, this testing process is exhaustive and time-consuming. Combinatorial interaction strategies can systematically reduce the number of test cases to construct a minimal test suite without affecting the effectiveness of the tests. This paper presents a new efficient searchbased strategy to generate constrained interaction test suites to cover all possible combinations. The paper also shows a new application of constrained interaction testing in software fault searches. The proposed strategy initially generates the set of all possible t − tuple combinations; then, it filters out the set by removing the forbidden t − tuples using the base forbidden tuple (BFT) approach. The strategy also utilizes a mixed neighborhood tabu search (TS) to construct optimal or near-optimal constrained test suites. The efficiency of the proposed method is evaluated through a comparison against two well-known state-of-the-art tools. The evaluation consists of three sets of experiments for 35 standard benchmarks. Additionally, the effectiveness and quality of the results are assessed using a real-world case study. Experimental results show that the proposed strategy outperforms one of the competitive strategies, ACTS, for approximately 83% of the benchmarks and achieves similar results to CASA for 65% of the benchmarks when the interaction strength is 2. For an interaction strength of 3, the proposed method outperforms other competitive strategies for approximately 60% and 42% of the benchmarks. The proposed strategy can also generate constrained interaction test suites for an interaction strength of 4, which is not possible for many strategies. The real-world case study shows that the generated test suites can effectively detect injected faults using mutation testing.Software testing plays a significant role in specifying the quality of developed software. High-quality software can be achieved by employing an intensive testing process to detect most of the bugs. Ideally, identifying all possible bugs is achievable by applying exhaustive testing with a large test suite. However, exhaustive testing leads to more time spent in software testing, which increases the development cost. Software that is released without being tested well (due to a release deadline constraint or any other reasons) loses trust in the market. As estimated in [1], software faults can generate a drop in the product stock price of 4 − 6% on average (for companies encountering multiple software failures) and further generate almost 3 billion dollars of market losses. Hence, software testers should gener...

show abstract

“…. v k )) or MCA(N ; t, k, v k ) may be adopted when the number of component values varies, while the Constrained Covering Array (CCA) may be adopted when there are constraints among the values of input parameters [3].…”

Section: Background and Literaturementioning

confidence: 99%

Towards an Automated Unified Framework to Run Applications for Combinatorial Interaction Testing

Ahmed

Pahim²,

Rosa

et al. 2019

Proceedings of the Evaluation and Assessment on Software Engineering

Self Cite

View full text Add to dashboard Cite

Combinatorial interaction testing (CIT) is a well-known technique, but the industrial experience is needed to determine its effectiveness in different application domains. We present a case study introducing a unified framework for generating, executing and verifying CIT test suites, based on the open-source Avocado test framework. In addition, we present a new industrial case study to demonstrate the effectiveness of the framework. This evaluation showed that the new framework can generate, execute, and verify effective combinatorial interaction test suites for detecting configuration failures (invalid configurations) in a virtualization system.

show abstract

Constrained Interaction Testing: A Systematic Literature Study

Cited by 57 publications

References 108 publications

An evaluation of Monte Carlo-based hyper-heuristic for interaction testing of industrial embedded software applications

An evaluation of Monte Carlo-based hyper-heuristic for interaction testing of industrial embedded software applications

Generation and Application of Constrained Interaction Test Suites Using Base Forbidden Tuples with a Mixed Neighborhood Tabu Search

Towards an Automated Unified Framework to Run Applications for Combinatorial Interaction Testing

Contact Info

Product

Resources

About