Priority Integration for Weighted Combinatorial Testing

Abstract

Priorities (weights) for parameter values can improve the effectiveness of combinatorial testing. Previous approaches have employed weights to derive high-priority test cases either earlier or more frequently. Our approach integrates these order-focused and frequency-focused prioritizations. We show that our priority integration realizes a small test suite providing high-priority test cases early and frequently in a good balance. We also propose two algorithms that apply our priority integration to existing combinatorial test generation algorithms. Experimental results using numerous test models show that our approach improves the existing approaches w.r.t. Order-focused and frequency-focused metrics, while overheads in the size and generation time of test suites are small.<p>QC 20170109</p

Distance-Integrated Combinatorial Testing

“…We previously proposed a t-way test generation [12] to construct test suites where higher priority test cases and parameter-values appear early and frequently for SUT models whose parameter-values are prioritized. The previous method considers increasing both the coverage called weight coverage and the metric called KL divergence.…”

Test Effectiveness Evaluation of Prioritized Combinatorial Testing: A Case Study

“…To evaluate test suites, pricot uses both weight coverage and KL divergence [2]. Table II shows a pairwise test suite that is generated by pricot with co.cf, together with cumulative weight coverage and KL divergence of its test cases.…”

“…Existing prioritized combinatorial test generation algorithms [1], [2], [6], [10], [16] have evaluated their test suites with weight coverage and KL divergence but not fault detection effectiveness as described in Section I.…”

“…The algorithm in [2], which we call pricot, integrates the order-focused prioritization (shortly co) and the frequencyfocused prioritization (shortly cf) with a size-focused prioritization (shortly cs) which considers that the size of a test suite should be small. To realize a small test suite where high-priority test cases appear early and frequently in a good balance, pricot takes a prioritization order of cs, co, and cf (e. g., cs > co > cf, denoted by cs.co.cf) as an input and generates a pairwise test suite that considers the weights in the given order.…”

“…We use prioritized pairwise test suites generated by pricot [2] for the constructed SUT models with constraints and weights. For each model, we use nine variants of test suites generated with the following prioritization orders: 1) cs, 2) co, 3) cf, 4) cs.co, 5) co.cs, 6) cs.cf, 7) co.cf, 8) cs.co.cf, and 9) co.cs.cf.…”