Applicability of non-specification-based approaches to logic testing for software

Abstract

Testing is a crucial part of the developnient of highly dependable systems. In this paper we consider testing of an implementation that is intended to satisfy a boolean forniula. I n the literature, specification-based testing has been suggested f o r this purpose. Typically, such methods first hypothesize a fault class and then generate tests. However there is alniost no research that justifies fault classes proposed previously. Moreover specifications amenable to autoiiiutic test generation are not always available to testers in practice. Based 011 these observations, we examine the qq,dicubi/io, of nori-specI~cntiori-based approaches, which need no spec$cation in the form of a boolean forniula to create tests. We conipare a specification-based approach to two non-specificatiori-based approaches, naniely, random testing and conibinatorial testing, which is an emerging technique based on conibinatorial designs. The results of an experiment show that conibinatorial testing is often coniparative to specijcation-based testing and is always niuch superior to random testing.

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

“…Type of Experiment Constraints Included? CT / RT ART / RT CT / ART [8], [9], [10], [11], [12], [13], [14], [15], [16] Case Analysis No √ [17], [18], [19] Simulation No √ [20], [21] Empirical Study Yes √ [5] Formal Analysis No √ [22] Simulation No √ [23], [24], [25], [26] Empirical Study No √ [27] Simulation Yes √ [28] Simulation No √ √ √ both theoretically and empirically [22], [23], [24], [25], [26]. However, these studies assumed no constraints between parameters, which may be problematic as real-world applications often have constrained domains [29], [30].…”

“…Bell and Vouk [10] compared 2-way and random testing for detecting security faults on two network related products, and Pretschner et al [11] compared them for testing access control policies on four cases. Both studies showed the superiority of 2-way CT. Additionally, Kobayashi et al [9] applied τ -way and random testing to examine logical expressions from the specification of TCAS II. Their work was further expanded by Ballance et al [17] and Vilkomir et al [18], where automatically generated expressions were used for simulation.…”

“…Table 1 summarises previous studies of CT, RT and ART. When comparing CT against RT, previous studies often lead to controversial results: some suggested that CT is more effective than random test suite of similar or larger size [8], [9], [10], [11], [12], [17], [18], [21], while others found that the difference between CT and RT is not as large as people expect [13], [14], [15], [16], [19], [20]. Most of these previous studies are derived from a few example cases; only two ( [20] and [21]) reported empirical results beyond case analysis.…”