Assuring fault classification agreement - an empirical evaluation

Henningsson, Kennet; Wohlin, Claes

doi:10.1109/isese.2004.1334897

Cited by 25 publications

(28 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 1 reports the number of feasible V&V techniques and failure types for each of the five phases. [9] show the absence of correlation among failure types, the failure misclassification was randomly chosen from among the other feasible failure types within that phase. Because specific misclassifications could threaten results validity, we repeated the error injection algorithm 10,000 times, for each phase and for each error rate.…”

Section: Discussionmentioning

confidence: 99%

“…Vice versa, the Loss for failure type J is 66% when the error rate is only 5%. We note that according to past studies [6] [7][8] [9], the failure classification error is likely more than 5%. Thus, in a realistic context, bugs of certain failure types are likely very hard to detect, even when choosing the V&V technique that historically caught most of them.…”

Section: Computation Of Lossmentioning

confidence: 99%

“…The present paper sits between two research directions: defect classification and evaluating bug-fix datasets for bias. Defect classification has been investigated in past studies [6] [7][8] [9], where the bug classification error rate of specific subjects under specific circumstances was reported and enhanced. However, so far it is unclear the extent to which a specific error rate (e.g., 10%) decreases the value of the bug repository and for which uses.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

On failure classification: the impact of "getting it wrong"

Falessi

Kidwell

Hayes

et al. 2014

Companion Proceedings of the 36th International Conference on Software Engineering

View full text Add to dashboard Cite

Bug classification is a well-established practice which supports important activities such as enhancing verification and validation (V&V) efficiency and effectiveness. The state of the practice is manual and hence classification errors occur. This paper investigates the sensitivity of the value of bug classification (specifically, failure type classification) to its error rate; i.e., the degree to which misclassified historic bugs decrease the V&V effectiveness (i.e., the ability to find bugs of a failure type of interest). Results from the analysis of an industrial database of more than 3,000 bugs show that the impact of classification error rate on V&V effectiveness significantly varies with failure type. Specifically, there are failure types for which a 5% classification error can decrease the ability to find them by 66%. Conversely, there are failure types for which the V&V effectiveness is robust to very high error rates. These results show the utility of future research aimed at: 1) providing better tool support for decreasing human errors in classifying the failure type of bugs, 2) providing more robust approaches for the selection of V&V techniques, and 3) including robustness as an important criterion when evaluating technologies.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Computation Of Lossmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

On failure classification: the impact of "getting it wrong"

Falessi

Kidwell

Hayes

et al. 2014

Companion Proceedings of the 36th International Conference on Software Engineering

View full text Add to dashboard Cite

show abstract

“…More generally, it occurs when two or more threads attempts to access shared resources held by other threads, and none of the threads can give them up [18,21]. During deadlock, all involved threads are in a waiting state.…”

Section: Concurrency Bug Classificationmentioning

confidence: 99%

“…It occurs when concurrent threads perform conflicting accesses by trying to update the same memory location or shared variable [18,19].…”

Section: Concurrency Bug Classificationmentioning

confidence: 99%

Concurrency bugs in open source software: a case study

Asadollah

Sundmark

Eldh

et al. 2017

J Internet Serv Appl

View full text Add to dashboard Cite

Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixing-times would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software, understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful. This paper presents an empirical study focusing on understanding the differences and similarities between concurrency bugs and other bugs, as well as the differences among various concurrency bug types in terms of their severity and their fixing time, and reproducibility. Our basis is a comprehensive analysis of bug reports covering several generations of five open source software projects. The analysis involves a total of 11860 bug reports from the last decade, including 351 reports related to concurrency bugs. We found that concurrency bugs are different from other bugs in terms of their fixing time and severity while they are similar in terms of reproducibility. Our findings shed light on concurrency bugs and could thereby influence future design and development of concurrent software, their debugging and testing, as well as related tools.

show abstract

On the number and nature of faults found by random testing

Ciupa

Pretschner

Manuel

et al. 2011

Software Testing Verif & Rel

View full text Add to dashboard Cite

Intuition suggests that random testing should exhibit a considerable difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. This article first evaluates the variance over time of the number of faults detected by randomly testing object-oriented software that is equipped with contracts. It presents the results of an empirical study based on 1215 h of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. The analysis of over 6 million failures triggered during the experiments shows that the relative number of faults detected by random testing over time is predictable, but that different runs of the random test case generator detect different faults. The experiment also suggests that the random testing quickly finds faults: the first failure is likely to be triggered within 30 s. The second part of this article evaluates the nature of the faults found by random testing. To this end, it first explains a fault classification scheme, which is also used to compare the faults found through random testing with those found through manual testing and with those found in field use of the software and recorded in user incident reports. The results of the comparisons show that each technique is good at uncovering different kinds of faults. None of the techniques subsumes any of the others; each brings distinct contributions. This supports a more general conclusion on comparisons between testing strategies: the number of detected faults is too coarse a criterion for such comparisons - the nature of faults must also be considered

show abstract

Assuring fault classification agreement - an empirical evaluation

Cited by 25 publications

References 10 publications

On failure classification: the impact of "getting it wrong"

On failure classification: the impact of "getting it wrong"

Concurrency bugs in open source software: a case study

On the number and nature of faults found by random testing

Contact Info

Product

Resources

About