Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing

Poon, Pak-Lok; Tse, T. H.; Tang, Sau-Fun; Kuo, Fei-Ching

doi:10.1007/s11219-010-9109-4

Cited by 21 publications

(12 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Beer and Ramler (Beer and Ramler, 2008) studied the role of experience in testing by industrial case studies. In addition, Kettunen et al (Kettunen et al, 2010) recognized the importance of testers' experience, Poon et al (Poon et al, 2011) studied the effect of experience on test case design and Galletta et al (Galletta et al, 1993) report that expertise increases error finding performance.…”

Section: Related Workmentioning

confidence: 99%

An experiment on the effectiveness and efficiency of exploratory testing

et al. 2014

View full text Add to dashboard Cite

The exploratory testing (ET) approach is commonly applied in industry, but lacks scientific research. The scientific community needs quantitative results on the performance of ET taken from realistic experimental settings. The objective of this paper is to quantify the effectiveness and efficiency of ET vs. testing with documented test cases (test case based testing, TCT). We performed four controlled experiments where a total of 24 practitioners and 46 students performed manual functional testing using ET and TCT. We measured the number of identified defects in the 90-minute testing sessions of testing, the detection difficulty, severity and types of the detected defects, and the number of false defect reports. The results show that ET found a significantly greater number of defects. ET also found significantly more defects of varying levels of difficulty, types and severity levels. The two testing approaches, however, did not differ significantly in terms of the number of false defect reports submitted. We conclude that ET was more efficient than TCT in our experiment. ET was also more effective than TCT when detection difficulty, type of defects and severity levels are considered. However, the two approaches are comparable when it comes to the number of false defect reports submitted.

show abstract

Section: Related Workmentioning

confidence: 99%

An experiment on the effectiveness and efficiency of exploratory testing

et al. 2014

View full text Add to dashboard Cite

show abstract

“…All held bachelor degrees in computer science and had a professional experience (in software development and testing) above 10 years. Following the experience threshold used by other works , the participants were considered experienced testers. Furthermore, they had previous hands‐on experience using the involved diagnosis tools (hence, there was no learning curve that could have impacted the results).…”

Section: Assessment Of Trade‐offsmentioning

confidence: 99%

PHOEBE: an automation framework for the effective usage of diagnosis tools in the performance testing of clustered systems

et al. 2017

View full text Add to dashboard Cite

The identification of performance issues and the diagnosis of their root causes are time-consuming and complex tasks, especially in clustered environments. To simplify these tasks, researchers have been developing tools with built-in expertise for practitioners. However, various limitations exist in these tools that prevent their efficient usage in the performance testing of clusters (e.g. the need of manually analysing huge volumes of distributed results). In a previous work, we introduced a policy-based adaptive framework (PHOEBE) that automates the usage of diagnosis tools in the performance testing of clustered systems, in order to improve a tester's productivity, by decreasing the effort and expertise needed to effectively use such tools. This paper extends that work by broadening the set of policies available in PHOEBE, as well as by performing a comprehensive assessment of PHOEBE in terms of its benefits, costs and generality (with respect to the used diagnosis tool). The performed evaluation involved a set of experiments in assessing the different trade-offs commonly experienced by a tester when using a performance diagnosis tool, as well as the time savings that PHOEBE can bring to the performance testing and analysis processes. Our results have shown that PHOEBE can drastically reduce the effort required by a tester to do performance testing and analysis in a cluster. PHOEBE also exhibited consistent behaviour (i.e. similar time-savings and resource utilisations), when applied to a set of commonly used diagnosis tools, demonstrating its generality. Finally, PHOEBE proved to be capable of simplifying the configuration of a diagnosis tool. This was achieved by addressing the identified trade-offs without the need for manual intervention from the tester. PHOEBE is implemented with the multi-agent architecture depicted in Figure 1. There, it can be seen how PHOEBE is composed of three types of agents: the control agent is responsible for interacting with the load testing tool to know when the test starts and ends. It is also responsible for evaluating the policies and propagating the decisions to the other nodes. Meanwhile, the application node agent is responsible for performing the required tasks in each application node (e.g. sampling collection or sending the collected samples to the diagnosis tool). Finally, the diagnosis tool Here, the objective was to evaluate the potential trade-off between the number of samples concurrently processed by a diagnosis tool and the number of resources it requires to process the samples. The following sections describe this experiment and its results. PHOEBE 1863 7.1. Experiment #4: proposed policies assessmentThe objective of this experiment was to evaluate the behaviour of PHOEBE, as well as the set of proposed policies, in order to assess how well they have fulfilled their purpose of addressing the identified trade-offs without the need for manual intervention from the tester. The following sections describe this experiment and its results.7.1.1. Experimental set-up....

show abstract

“…In view of this problem, we conducted empirical studies [5], [21] using informal commercial specifications. The objective of these studies was to investigate the common mistakes made by software testers involving informal specifications under an ad hoc identification approach.…”

Section: Introductionmentioning

confidence: 99%

“…Following on our previous studies [5], [6], [21], we have conducted a further comparative study with 16 experienced software practitioners. The objectives of the study are: (a) To determine the popularity of an incremental identification approach involving informal specification components.…”

Section: Introductionmentioning

confidence: 99%