How is exploratory testing used? A state-of-the-practice survey

Abstract

Context: Exploratory Testing has experienced a rise in popularity in the industry with the emergence of agile development practices, yet it remains unclear, in which domains and how it is used in practice. Goal: To study how software engineers understand and apply the principles of exploratory testing, as well as the specific advantages and difficulties they experience. Method: We conducted an online survey in the period June to August 2013 among Estonian and Finnish software developers and testers. Results: Our main findings are that the majority of testers, developers, and test managers using ET, (1) apply ET to usabilitycritical, performance-critical, security-critical and safety-critical software to a high degree; (2) use ET very flexibly in all types of test levels, activities, and phases; (3) perceive ET as an approach that supports creativity during testing and that is effective and efficient; and (4) find that ET is not easy to use and has little tool support. Conclusions: The high degree of application of ET in critical domains is particularly interesting and indicates a need for future research to obtain a better understanding of the effects of ET in these domains. In addition, our findings suggest that more support to ET users should be given (guidance and tools).

A Decision Support Method for Recommending Degrees of Exploration in Exploratory Testing

“…Exploratory testing is defined as simultaneous learning, test design and test execution [9]. ET is perceived to be flexible and applicable to different types of activities, test levels and phases [4]. Existing literature showcases a good amount of evidence regarding the merits of ET, such as its defect detection effectiveness, cost effectiveness and high performance for detecting critical defects [4], [10], [11] and [12].…”

“…ET is perceived to be flexible and applicable to different types of activities, test levels and phases [4]. Existing literature showcases a good amount of evidence regarding the merits of ET, such as its defect detection effectiveness, cost effectiveness and high performance for detecting critical defects [4], [10], [11] and [12]. During the exploratory testing process, the testers may interact with the application and take the information it provides to react, change course or explore the application's functionality without any constraint [13].…”

“…Furthermore, SBTM reflects the concept of time boxing the test sessions to track testing where a test session is a basic testing work unit and an uninterrupted block of reviewable and chartered test effort [3]. Most practitioners claim that the test charter is an important element of SBTM [2], [3] and [4].…”

Fostering the diversity of exploratory testing in web applications

“…ET is frequently organized in sessions with a limited timeframe [3,5], whose duration can vary from few minutes to several hours. An ET session can involve a single tester or a group of testers collaborating.…”

Checklists to Support Test Charter Design in Exploratory Testing

“…James Bach defines exploratory testing as simultaneous learning, test design and test execution [3]. Existing literature reflects that ET is widely used for testing complex systems as well and is perceived to be flexible in all types of test levels, activities and phases [7] [13]. In the context of quality, ET has amassed a good amount of evidence on overall defect detection effectiveness, cost effectiveness and high performance for detecting critical defects [1] [9] [10] [11] [13].…”