Exploring the industry's challenges in software testing: An empirical study

Abstract

ContextSoftware testing is an important and costly software engineering activity in the industry. Despite the efforts of the software testing research community in the last several decades, various studies show that still many practitioners in the industry report challenges in their software testing tasks.ObjectiveTo shed light on industry's challenges in software testing, we characterize and synthesize the challenges reported by practitioners. Such concrete challenges can then be used for a variety of purposes, eg, research collaborations between industry and academia.MethodOur empirical research method is opinion survey. By designing an online survey, we solicited practitioners' opinions about their challenges in different testing activities. Our dataset includes data from 72 practitioners from eight different countries.ResultsOur results show that test management and test automation are considered the most challenging among all testing activities by practitioners. Our results also include a set of 104 concrete challenges in software testing that may need further investigations by the research community.ConclusionWe conclude that the focal points of industrial work and academic research in software testing differ. Furthermore, the paper at hand provides valuable insights concerning practitioners' “pain” points and, thus, provides researchers with a source of important research topics of high practical relevance.

Design, Monitoring, and Testing of Microservices Systems: The Practitioners' Perspective

“…About testing tools, JUnit, JMeter, and Selenium are commonly used by practitioners of both MSAbased and non-MSA-based systems (e.g., [125,126,127,128]), for example, Selenium is used as a tool to perform testing on MSA-based systems (see Figure 19), and is also employed for testing non-MSA-based systems due to its test automation capabilities. Regarding testing challenges, the results of our survey and other surveys (e.g., [127,131,132]) show that two testing challenges (i.e., integration testing, creating and implementing automated tests) are commonly faced by practitioners of both MSA-based and non-MSA-based systems.…”

“…We explored several survey studies on testing conducted in the context of non-MSA-based systems (e.g., [123,124,125,126,127,128,129,130,131,132]), and we compared our key findings on testing strategies, tools, and challenges with these surveys. Regarding testing strategies, the results of our survey and the surveys related to non-MSA-based systems (e.g., [123,124,125,126,127]) show that unit testing is commonly used as a de facto strategy for testing all types of systems.…”

Software test results exploration and visualization with continuous integration and nightly testing

“…Garousi and Elberzhager [17] argue that not only test execution ought to be automated, other activities including test results reporting are well served by automation. Similarly, Garousi et al [18] explored how industry practitioners perceived challenges in software testing activities. The challenge of test results reporting was perceived as the least severe among nine categories (test automation and test management were the most challenging ones).…”

Improving test automation maturity: A multivocal literature review

“…The increased level of sophistication of test automation technology allows the increased test scale, test coverage, test efficiency, and test effectiveness [23]. • People capability: the workforce is able to leverage the competencies in test automation [24,4,25]. A variety of expertise is required for people to be able to work on test automation [4,25].…”