Empirically Evaluating the Quality of Automatically Generated and Manually Written Test Suites

Kracht, Jeshua S.; Petrovic, Jacob Z.; Walcott-Justice, Kristen

doi:10.1109/qsic.2014.33

Cited by 15 publications

(6 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Development environments also can make a difference. Kracht et al [25] andAlkaoud and Walcott [6] discuss the differences observed in test suites in these scenarios.…”

Section: Test Case Writing Attributesmentioning

confidence: 99%

Characterization of Open-Source Applications and Test Suites

Balasubramanian¹,

Walcott²

2018

IJSEA

View full text Add to dashboard Cite

show abstract

“…Development environments also can make a difference. Kracht et al [25] andAlkaoud and Walcott [6] discuss the differences observed in test suites in these scenarios.…”

Section: Test Case Writing Attributesmentioning

confidence: 99%

Characterization of Open-Source Applications and Test Suites

Balasubramanian¹,

Walcott²

2018

IJSEA

View full text Add to dashboard Cite

show abstract

“…There are studies investigating the use of both manual testing and automated coverage-based test generation of real-world programs. Several researchers have performed case studies [21], [13], [17] and focused on already created manual test suites while others performed controlled experiments [8] with human participants manually creating and automatically generating test suites. These results kindled our interest in studying how manual testing compares to automated coverage-based test generation in an industrial safety-critical control software domain.…”

Section: Related Workmentioning

confidence: 99%

A Comparative Study of Manual and Automated Testing for Industrial Control Software

Enoiu

Sundmark

Čaušević

et al. 2017

2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)

View full text Add to dashboard Cite

Automated test generation has been suggested as a way of creating tests at a lower cost. Nonetheless, it is not very well studied how such tests compare to manually written ones in terms of cost and effectiveness. This is particularly true for industrial control software, where strict requirements on both specification-based testing and code coverage typically are met with rigorous manual testing. To address this issue, we conducted a case study in which we compared manually and automatically created tests. We used recently developed real-world industrial programs written in the IEC 61131-3, a popular programming language for developing industrial control systems using programmable logic controllers. The results show that automatically generated tests achieve similar code coverage as manually created tests, but in a fraction of the time (an average improvement of roughly 90%). We also found that the use of an automated test generation tool does not result in better fault detection in terms of mutation score compared to manual testing. Specifically, manual tests more effectively detect logical, timer and negation type of faults, compared to automatically generated tests. The results underscore the need to further study how manual testing is performed in industrial practice and the extent to which automated test generation can be used in the development of reliable systems.

show abstract

“…Specifically, Kracht et al [25] applied EvoSuite [14] on a number of software projects with manual test cases, and reported that the two types of test cases are similarly effective. Compared to that work, our study considers a different test-case-generation technique (i.e., DSE), and conducted more detailed investigation on different code depths, test-suite complementarity, hard-to-kill mutants, as well as the qualitative features of the test suites.…”

Section: Related Workmentioning

confidence: 99%

Experience report: how is dynamic symbolic execution different from manual testing? a study on KLEE

Wang

Zhang²,

Tanofsky

2015

Proceedings of the 2015 International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Software testing has been the major approach to software quality assurance for decades, but it typically involves intensive manual efforts. To reduce manual efforts, researchers have proposed numerous approaches to automate test-case generation, which is one of the most time-consuming tasks in software testing. One most recent achievement in the area is Dynamic Symbolic Execution (DSE), and tools based on DSE, such as KLEE, have been reported to generate test suites achieving higher code coverage than manually developed test suites. However, besides the competitive code coverage, there have been few studies to compare DSE-based test suites with manually developed test suites more thoroughly on various metrics to understand the detailed differences between the two testing methodologies. In this paper, we revisit the experimental study on the KLEE tool and GNU CoreUtils programs, and compare KLEE-based test suites with manually developed test suites on various aspects. We further carried out a qualitative study to investigates the reasons behind the differences in statistical results. The results of our studies show that while KLEE-based test suites are able to generate test cases with higher code coverage, they are relatively less effective on covering hard-to-cover code and killing mutants. Furthermore, our qualitative study reveals that KLEEbased test suites have advantages in exploring error-handling code and exhausting options, but are less effective on generating valid string inputs and exploring meaningful program behaviors.

show abstract

Empirically Evaluating the Quality of Automatically Generated and Manually Written Test Suites

Cited by 15 publications

References 34 publications

Characterization of Open-Source Applications and Test Suites

Characterization of Open-Source Applications and Test Suites

A Comparative Study of Manual and Automated Testing for Industrial Control Software

Experience report: how is dynamic symbolic execution different from manual testing? a study on KLEE

Contact Info

Product

Resources

About