Is XML-Based Test Case Prioritization for Validating WS-BPEL Evolution Effective in Both Average and Adverse Scenarios?

Abstract

AbstractIn real life, a tester can only afford to apply one test case prioritization technique to one test suite against a service-oriented workflow application once in the regression testing of the application, even if it results in an adverse scenario such that the actual performance in the test session is far below the average. It is unclear whether the factors of test case prioritization techniques known to be significant in terms of average performance can be extrapolated to adverse scenarios. In this paper, we examine whether such a factor or technique may consistently affect the rate of fault detection in both the average and adverse scenarios. The factors studied include prioritization strategy, artifacts to provide coverage data, ordering direction of a strategy, and the use of executable and non-executable artifacts. The results show that only a minor portion of the 10 studied techniques, most of which are based on the iterative strategy, are consistently effective in both average and adverse scenarios. To the best of our knowledge, this paper presents the first piece of empirical evidence regarding the consistency in the effectiveness of test case prioritization techniques and factors of service-oriented workflow applications between average and adverse scenarios.

Connecting the Average and the Non-Average

“…The comparison result can be one of the three possible cases: Mi is significantly more effective than C1, Mi has no significant difference from C1, and Mi is significantly less effective than C1. We followed (Jia et al 2014) to use ">", "=", and "<" to represent the above three cases, respectively. We also want to study whether Mi in the adverse scenarios is statistically more effective than the mean effectiveness of C1.…”

“…M3 and M4: Total XPath selection coverage prioritization (Total-XPath-Selection) and Additional XPath selection coverage prioritization (Addtl-XPath-Selection) (Jia et al 2014). They are the same as M1 and M2, respectively, except that M3 and M4 measure test coverage in terms of XPath selections rather than BPEL activities.…”

“…Descending XRG node coverage prioritization (Descending-XRG-Node) (Jia et al 2014). Each technique first partitions test cases into groups such that all the test cases with the same number of XRG nodes are placed in the same group.…”

“…The results showed that less than half of all the techniques and factors exhibited such consistency. This paper significantly extends the preliminary version (Jia et al 2014). We investigate two completely new research questions (denoted by RQ1 and RQ2), which are refined from the research questions presented in the preliminary version.…”

“…In the preliminary version of this paper (Jia et al 2014), we reported the first controlled experiment investigating whether the effectiveness results (the rate of fault detection measured by APFD (Elbaum et al 2002)) of both TCP techniques and their design factors can be extrapolated from the average scenarios to the adverse scenarios. The controlled experiment included 10 TCP techniques plus two control techniques, eight benchmarks, and 100 test suites per benchmark with 100 repeated applications of every TCP technique on every such test suite.…”