Benefits and limitations of automated software testing: Systematic literature review and practitioner survey

Rafi, Dudekula Mohammad; Moses, Katam Reddy Kiran; Petersen, Kai; Mäntylä, Martti

doi:10.1109/iwast.2012.6228988

Cited by 67 publications

(32 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Version control system Automated test suite. The benefits of automated software testing are widely accepted both in the academic sphere and between practitioners [100]. The two main approaches to automated testing are regression and unit testing, which are not mutually exclusive and the majority of projects employ both methods in their automated test suites.…”

Section: Static Analysis Namementioning

confidence: 99%

Open source software in quantum computing

2018

View full text Add to dashboard Cite

Open source software is becoming crucial in the design and testing of quantum algorithms. Many of the tools are backed by major commercial vendors with the goal to make it easier to develop quantum software: this mirrors how well-funded open machine learning frameworks enabled the development of complex models and their execution on equally complex hardware. We review a wide range of open source software for quantum computing, covering all stages of the quantum toolchain from quantum hardware interfaces through quantum compilers to implementations of quantum algorithms, as well as all quantum computing paradigms, including quantum annealing, and discrete and continuous-variable gate-model quantum computing. The evaluation of each project covers characteristics such as documentation, licence, the choice of programming language, compliance with norms of software engineering, and the culture of the project. We find that while the diversity of projects is mesmerizing, only a few attract external developers and even many commercially backed frameworks have shortcomings in software engineering. Based on these observations, we highlight the best practices that could foster a more active community around quantum computing software that welcomes newcomers to the field, but also ensures high-quality, well-documented code.

show abstract

Section: Static Analysis Namementioning

confidence: 99%

Open source software in quantum computing

2018

View full text Add to dashboard Cite

show abstract

“…A more recent systematic literature review by Rafi et al also identified that there is a lack of research that focuses on the challenges and solutions of automated software testing (Rafi et al 2012). In the review, 24,706 papers were surveyed but only 25 reports were found with empirical evidence about the benefits and drawbacks of automated testing.…”

Section: Related Workmentioning

confidence: 99%

On the long-term use of visual gui testing in industrial practice: a case study

Alégroth

Feldt

2017

Empir Software Eng

View full text Add to dashboard Cite

Visual GUI Testing (VGT) is a tool-driven technique for automated GUI-based testing that uses image recognition to interact with and assert the correctness of the behavior of a system through its GUI as it is shown to the user. The technique's applicability, e.g. defect-finding ability, and feasibility, e.g. time to positive return on investment, have been shown through empirical studies in industrial practice. However, there is a lack of studies that evaluate the usefulness and challenges associated with VGT when used long-term (years) in industrial practice. This paper evaluates how VGT was adopted, applied and why it was abandoned at the music streaming application development company, Spotify, after several years of use. A qualitative study with two workshops and five well chosen employees is performed at the company, supported by a survey, which is analyzed with a grounded theory approach to answer the study's three research questions. The interviews provide insights into the challenges, problems and limitations, but also benefits, that Spotify experienced during the adoption and use of VGT. However, due to the technique's drawbacks, VGT has been abandoned for a new technique/framework, simply called the Test interface. The Test interface is considered more robust and flexible for Spotify's needs but has several drawbacks, including that it does not test the actual GUI as shown to the user like VGT does. From the study's results it is concluded that VGT can be used long-term in industrial practice but it requires organizational change as well as engineering best practices to be beneficial. Through synthesis of the study's results, and results from previous work, a set of guidelines are presented that aim to aid practitioners to adopt and use VGT in industrial practice. However, due to the abandonment of the technique, future research is required to analyze in what types of projects the technique is, and is not, long-term viable. To this end, we also present Spotify's Test interface solution for automated GUI-based testing and conclude that it has its own benefits and drawbacks.

show abstract

“…This has the advantage that the primary studies can also be examined by other reviewers, and this will support the reliability and reproducibility of our SMS. Rafi et al (2012) provide a systematic literature review and survey on benefits and limitations of automated software testing. They consider the 25 studies and conduct data extraction for both benefits and limitations of automated software testing.…”

Section: Threats To Validitymentioning

confidence: 99%

“…Using explicit models helps to structure the process of deriving tests and support the reuse, reproduction, and documentation of test cases. In addition, MBT enables the automated production and execution of test cases, which on its turn reduces the cost and time of testing and increases the quality of test cases (Rafi et al 2012). MBT has been applied for testing both functional and nonfunctional properties.…”

mentioning

confidence: 99%

Model-based testing for software safety: a systematic mapping study

Gurbuz

Tekinerdoğan

2017

Software Qual J

View full text Add to dashboard Cite

Testing safety-critical systems is crucial since a failure or malfunction may result in death or serious injuries to people, equipment, or environment. An important challenge in testing is the derivation of test cases that can identify the potential faults. Model-based testing adopts models of a system under test and/or its environment to derive test artifacts. This paper aims to provide a systematic mapping study to identify, analyze, and describe the state-of-the-art advances in model-based testing for software safety. The systematic mapping study is conducted as a multi-phase study selection process using the published literature in major software engineering journals and conference proceedings. We reviewed 751 papers and 36 of them have been selected as primary studies to answer our research questions. Based on the analysis of the data extraction process, we discuss the primary trends and approaches and present the identified obstacles. This study shows that model-based testing can provide important benefits for software safety testing. Several solution directions have been identified, but further research is critical for reliable model-based testing approach for safety.

show abstract

Benefits and limitations of automated software testing: Systematic literature review and practitioner survey

Cited by 67 publications

References 22 publications

Open source software in quantum computing

Open source software in quantum computing

On the long-term use of visual gui testing in industrial practice: a case study

Model-based testing for software safety: a systematic mapping study

Contact Info

Product

Resources

About