Towards a hybrid testing process unifying exploratory testing and scripted testing

Abstract

SUMMARY Given the current state of the art in research, practitioners are faced with the challenge of choosing scripted testing (ST) or exploratory testing (ET). This study aims at systematically incorporating strengths of ET and ST in a hybrid testing process to overcome the weaknesses of each. We utilized systematic review and practitioner interviews to identify strengths and weaknesses of ET and ST. Strengths of ET were mapped to weaknesses of ST and vice versa. Noblit and Hare's lines‐of‐argument method was used for data analysis. The results of the mapping were used as input to codesign a hybrid process with experienced practitioners. We found a clear need to create a hybrid process as follows: (i) both ST and ET provide strengths and weaknesses, and these depend on some particular conditions, which prevents preference of one approach to another; and (ii) the mapping showed that it is possible to address the weaknesses in one process by the strengths of the other in a hybrid form. With the input from literature and industry experts, a flexible and iterative hybrid process was designed. Practitioners can clearly benefit from using a hybrid process given the mapping of advantages and disadvantages. Copyright © 2013 John Wiley & Sons, Ltd.

How is exploratory testing used? A state-of-the-practice survey

“…The three top-ranked advantages of ET mentioned by respondents are "Supports creativity", "Efficient", and "Effective". This largely corresponds to what other researchers have found, e.g., Shah et al [18] and Itkonen et al [7][9]. The three top-ranked disadvantages of ET mentioned by respondents are "high requirement for tester", "inflexible", "hard to record", "not good for complicated project".…”

“…In a recent article, Shah et al conducted a comprehensive literature survey and interviews to investigate into strengths and weaknesses of ET and scripted testing [18]. Regarding strengths, the authors found ET to be cost-effective due to little and focused documentation, good resource utilization, rapid feedback, and quick familiarization with the product.…”

Comparing a Hybrid Testing Process with Scripted and Exploratory Testing: An Experimental Study with Practitioners

“…ST is a plan-driven testing process where test cases are designed prior to test execution in order to structure and to guide the testing tasks. This approach provides several benefits such as test awareness, test coverage, repeatability, and tracking [1], [17].…”

“…A few studies such as [9], [16], [17] mentioned that ET makes better use of testers creativity and skills to discover the bugs that ST may not uncover because of its mechanical nature. On the other hand, some shortcomings of the ET approach have been also reported such as; the difficulty in assessing whether all features are tested, unavailability of oracles, and invisibility of the quality of testing [16], [17], [18].…”

“…On the other hand, some shortcomings of the ET approach have been also reported such as; the difficulty in assessing whether all features are tested, unavailability of oracles, and invisibility of the quality of testing [16], [17], [18]. Furthermore, Agruss et al [1] stated that ET is not suitable while performing acceptance testing.…”