An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Abstract

Background: Combinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in detecting faults due to the interaction of several input parameters. Recent empirical studies show that greedy algorithms are still competitive for CIT. It is thus interesting to investigate new approaches to address CIT test case generation via greedy solutions and to perform rigorous evaluations within the greedy context. Methods: We present a new greedy algorithm for unconstrained CIT, T-Tuple Reallocation (TTR), to generate CIT test suites specifically via the Mixed-value Covering Array (MCA) technique. The main reasoning behind TTR is to generate an MCA M by creating and reallocating t-tuples into this matrix M, considering a variable called goal (ζ ). We performed two controlled experiments addressing cost-efficiency and only cost. Considering both experiments, we did 3200 executions related to 8 solutions. In the first controlled experiment, we compared versions 1.1 and 1.2 of TTR in order to check whether there is significant difference between both versions of our algorithm. In such experiment, we jointly considered cost (size of test suites) and efficiency (time to generate the test suites) in a multi-objective perspective. In the second controlled experiment we confronted TTR 1.2 with five other greedy algorithms/tools for unconstrained CIT: IPOG-F, jenny, IPO-TConfig, PICT, and ACTS. We performed two different evaluations within this second experiment where in the first one we addressed cost-efficiency (multi-objective) and in the second only cost (single objective). Results: Results of the first controlled experiment indicate that TTR 1.2 is more adequate than TTR 1.1 especially for higher strengths (5, 6). In the second controlled experiment, TTR 1.2 also presents better performance for higher strengths (5, 6) where only in one case it is not superior (in the comparison with IPOG-F). We can explain this better performance of TTR 1.2 due to the fact that it no longer generates, at the beginning, the matrix of t-tuples but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M. Conclusion: Considering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). For lower strengths, other solutions, like IPOG-F, may be better alternatives.

Level of combinatorial thinking in solving mathematical problems

“…TTR is a software program for computational problems and mathematical cases. The results indicate that there are better results in solving combinatorial cases, which means the use of TTR software has a better impact (Balera & Júnior, 2017).…”

“…Furthermore, combinatorial thinking is important because it is a basic thinking ability that must be continuously developed towards critical thinking abilities and skills. Still, many students encounter difficulty in combinatorial thinking (Batanero et al 1997;English, 2005;Lockwood, 2012;Eizenberg & Zaslavsky, 2004) Strategies to overcome combinatorial thinking difficulties have been applied by several researchers (Balera & Júnior, 2017;Hayashi & Ohsawa, 2013;Melusova & Vidermanova, 2015;Pizlo & Li, 2005). Hayashi & Ohsawa (2013) unravels that the game market innovator facilitates students to think combinatorially, which is to foster student creativity and innovation.…”

Actual approaches in solving complicated problems and realization of new scientific projects

“…Другие условия, такие как максимальное количество объектов в каждой позиции или возможность наличия пустых позиций, также могут быть важными для поиска решения проблемы. Для каждого упомянутого типа соединений не существует отдельной комбинаторной формулы или правила, и, более того, одна и та же комбинаторная операция может быть связана с двумя разными задачами распределения элементов в соединении [6][7]. Распределение n объектов по m позициям, с математической точки зрения, эквивалентно установлению отображения из множества n объектов в множество m позиций.…”

Hyper-Heuristics based on Reinforcement Learning, Balanced Heuristic Selection and Group Decision Acceptance

“…In other words, more problems are required to conclude about generalisation. This is known as a threat to external validity related to the population in the context of controlled experiments [65,66].…”