Integrating automated test generation into the WYSIWYT spreadsheet testing methodology

Abstract

Spreadsheet languages, which include commercial spreadsheets and various research systems, have had a substantial impact on end-user computing. Research shows, however, that spreadsheets often contain faults. Thus, in previous work, we presented a methodology that helps spreadsheet users test their spreadsheet formulas. Our empirical studies have shown that end users can use this methodology to test spreadsheets more adequately and efficiently; however, the process of generating test cases can still represent a significant impediment. To address this problem, we have been investigating how to incorporate automated test case generation into our testing methodology in ways that support incremental testing and provide immediate visual feedback. We have utilized two techniques for generating test cases, one involving random selection and one involving a goal-oriented approach. We describe these techniques and their integration into our testing environment, and report results of an experiment examining their effectiveness and efficiency.

AutoTest: A Tool for Automatic Test Case Generation in Spreadsheets

“…Since we are comparing AutoTest against HMT, we follow the evaluation of HMT described in [13] and take into consideration two dependent variables.…”

“…Our system, on the other hand, always produces the same output given the same starting spreadsheet configuration. We reproduce the results from [13] in Tables 1 and 2 and compare with the numbers obtained by running AutoTest on the same spreadsheets. 7 AutoTest is also efficient in the sense that it only generates one test case per feasible DU pair for the formula that is being tested.…”

“…WYSIWYT has been developed for the Forms/3 spreadsheet language [7] and gives the user feedback of the overall level of testedness of the spreadsheet by means of a progress bar. "Help Me Test" (HMT) is a component of the Forms/3 engine that does automatic test case generation to help the users minimize the cost of testing their spreadsheets [13]. Automatic test case generation has also been studied for generalpurpose programming languages [9,15].…”

“…Therefore the ultimate effectiveness score for HMT is averaged over 35 runs and the median response time over 35 runs has been presented in [13]. Our system, on the other hand, always produces the same output given the same starting spreadsheet configuration.…”

“…The notion of DU coverage as a test adequacy criteria is presented in Section 5, and in Section 6 we describe how AutoTest generates DU adequate test cases. A comparative evaluation of AutoTest against the "Help Me Test" (HMT) [13] test case generation system of Forms/3 [7] is described in Section 7. We present conclusions and plans for future work in Section 8.…”