Impact of Education and Experience Level on the Effectiveness of Exploratory Testing: An Industrial Case Study

Gebizli, Ceren Sahin; Sözer, Hasan

doi:10.1109/icstw.2017.8

Cited by 28 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since ET is used as a manual testing method, research on it has mostly focused on the human factors that alter e ectiveness and e ciency. An industrial case study by Gebizli and Sozer (2017) with 19 practitioners of di erent educational backgrounds and experience levels show that both factors a ects e ciency but only experience a ects the number of critical failures detected. Micallef et al (2016) found that trained testers employed di erent types of exploratory strategies than untrained testers.…”

Section: Exploratory Testingmentioning

confidence: 99%

Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing

Zhu¹,

Bayley²

2021

Preprint

View full text Add to dashboard Cite

Testing has been widely recognised as di cult for AI applications. is paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. In these strategies, testing aims at exploring the data space of a classi cation or clustering application to discover the boundaries between classes that the machine learning application de nes.is enables the tester to understand precisely the behaviour and function of the so ware under test. In the paper, three variants of exploratory strategies are presented with the algorithms implemented in the automated datamorphic testing tool Morphy. e correctness of these algorithms are formally proved. eir capability and cost of discovering borders between classes are evaluated via a set of controlled experiments with manually designed subjects and a set of case studies with real machine learning models.

show abstract

Section: Exploratory Testingmentioning

confidence: 99%

Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing

Zhu¹,

Bayley²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Practitioners have reported SBT from as varied domains as web applications and MedTech [17,14,18]. While ET is widely used by practitioners, the associated techniques 75 can be considered understudied: there is only a small set of associated studies [2,4], and a lack of comparative [9] and secondary studies [19].…”

Section: Related Workmentioning

confidence: 99%

“…Itkonen et al studied the application of ET, identifying how focused domain and system knowledge is used as a test oracle, while holistic domain and system knowledge is used for on-the-fly test design [20]. Results by Gebizli and Szer suggest that the level of domain and system experience influences the efficiency and ability to identify critical software faults [4]. Shah et al argues that it is not the level but the higher reliance on knowledge that is decisive, since the requirements associated with certain types of faults will more frequently be implicit [21].…”

Section: Related Workmentioning

confidence: 99%

Exploratory testing: Do contextual factors influence software fault identification?

Asplund

2019

Information and Software Technology

View full text Add to dashboard Cite

Context: Exploratory Testing (ET) is a manual approach to software testing in which learning, test design and test execution occurs simultaneously. Still a developing topic of interest to academia, although as yet insufficiently investigated, most studies focus on the skills and experience of the individual tester.However, contextual factors such as project processes, test scope and organisational boundaries are also likely to affect the approach.Objective: This study explores contextual differences between teams of testers at a MedTec firm developing safety-critical products to ascertain whether contextual factors can influence the outcomes of ET, and what associated implications can be drawn for test management.Method: A development project was studied in two iterations, each consisting of a quantitative phase testing hypotheses concerning when ET would identify faults in comparison to other testing approaches and a qualitative phase involving interviews.Results: Influence on ET is traced to how the scope of tests focus learning on different types of knowledge and imply an asymmetry in the strength and number of information flows to test teams. Conclusions:While test specialisation can be attractive to software development organisations, results suggest changes to processes and organisational * Corresponding author structures might be required to maintain test efficiency throughout projects: the responsibility for test cases might need to be rotated late in projects, and asymmetries in information flows might require management to actively strengthen the presence and connections of test teams throughout the firm. However, further research is needed to investigate whether these results also hold for non safety-critical faults.

show abstract

“…The manual version of the exploratory testing technique has been investigated by several studies, e.g., [3], [4], [7]. The influence of a tester's experience on the efficiency of the testing process has been examined [7].…”

Section: Related Workmentioning

confidence: 99%

“…The exploratory testing approach is a logical choice for testing systems for which a suitable test basis is not available. Even when the test basis is available, and the test cases are created, they can be either obsolete or inconsistent and structured at an excessively high level [4]. Thus, testers employ the exploratory testing technique as a solution for overcoming these obstacles.…”

Section: Introductionmentioning

confidence: 99%

Tapir: Automation Support of Exploratory Testing Using Model Reconstruction of the System Under Test

2018

View full text Add to dashboard Cite

For a considerable number of software projects, the creation of effective test cases is hindered by design documentation that is either lacking, incomplete or obsolete. The exploratory testing approach can serve as a sound method in such situations. However, the efficiency of this testing approach strongly depends on the method, the documentation of explored parts of a system, the organization and distribution of work among individual testers on a team, and the minimization of potential (very probable) duplicities in performed tests. In this paper, we present a framework for replacing and automating a portion of these tasks. A screen-flow-based model of the tested system is incrementally reconstructed during the exploratory testing process by tracking testers' activities. With additional metadata, the model serves for an automated navigation process for a tester. Compared with the exploratory testing approach, which is manually performed in two case studies, the proposed framework allows the testers to explore a greater extent of the tested system and enables greater detection of the defects present in the system. The results show that the time efficiency of the testing process improved with framework support. This efficiency can be increased by team-based navigational strategies that are implemented within the proposed framework, which is documented by another case study presented in this paper.

show abstract

Impact of Education and Experience Level on the Effectiveness of Exploratory Testing: An Industrial Case Study

Cited by 28 publications

References 14 publications

Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing

Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing

Exploratory testing: Do contextual factors influence software fault identification?

Tapir: Automation Support of Exploratory Testing Using Model Reconstruction of the System Under Test

Contact Info

Product

Resources

About