MUSEUM: Debugging real-world multilingual programs using mutation analysis

Hong, Shin; Kwak, Taehoon; Lee, Byeongcheol; Jeon, Yiru; Ko, Bongseok; Kim, Yun‐Ho; Kim, Moonzoo

doi:10.1016/j.infsof.2016.10.002

Cited by 29 publications

(12 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For each faulty program, we use EvoSuite version 1.0.7 6 to generate the additional regression tests. To run the most recent version of EvoSuite, we use DEFECTS4J version 2.0.0, unlike the metric comparison study in Section III-B that used 1.5.0.…”

Section: Discussionmentioning

confidence: 99%

FDG: A Precise Measurement of Fault Diagnosability Gain of Test Cases

An,

Yoo

2021

Preprint

View full text Add to dashboard Cite

Many existing fault localisation techniques become less effective or even inapplicable when not adequately supported by a rich test suite. To overcome this challenge, we present a human-in-the-loop fault localisation technique, QFID, that works with only a small number of initial failing test cases. We augment the failing test cases with automatically generated test data and elicit oracles from a human developer to label the test cases. A new result-aware test prioritisation metric allows us to significantly reduce the labelling effort by prioritising the test cases to achieve maximum localisation accuracy. An evaluation with EvoSuite and our test prioritisation metric shows that QFID can significantly increase the localisation accuracy. After only ten human labellings, QFID can localise 27% and 66% of realworld faults in DEFECTS4J at the top and within the top ten, respectively. This is a 13 and 2 times higher performance than when using the initial test cases. QFID is also resilient to human errors, retaining 80% of its acc@1 performance on average when we introduce a 30% error rate to the simulated human oracle.

show abstract

Section: Discussionmentioning

confidence: 99%

FDG: A Precise Measurement of Fault Diagnosability Gain of Test Cases

An,

Yoo

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Le et al merged information retrieval (IR)based FL and SBFL for a more effective FL. Their proposed solution analyzes two types of debugging hints: (1) test cases and their outcomes, along with (2) textual contents in associated bug reports. Yet another direction is mutation‐based FL . Program mutants are generated and used to detect faulty program elements.…”

Section: Conclusion and Lessons Learnedmentioning

confidence: 99%

An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems

et al. 2019

View full text Add to dashboard Cite

Pure spectrum-based fault localization (SBFL) is a well-studied statistical debugging technique that only takes a set of test cases (some failing and some passing) and their code coverage as input and produces a ranked list of suspicious program elements to help the developer identify the location of a bug that causes a failed test case. Studies show that pure SBFL techniques produce good ranked lists for small programs. However, our previous study based on the iBugs benchmark that uses the AspectJ repository shows that, for realistic programs, the accuracy of the ranked list is not suitable for human developers. In this paper, we confirm this based on a combined empirical evaluation with the iBugs and the Defects4J benchmark. Our experiments show that, on average, at most ∼40%, ∼80%, and ∼90% of the bugs can be localized reliably within the first 10, 100, and 1000 ranked lines, respectively, in the Defects4J benchmark.To reliably localize 90% of the bugs with the best performing SBFL metric D * , ∼450 lines have to be inspected by the developer. For human developers, this remains unsuitable, although the results improve compared with the results for the AspectJ benchmark. Based on this study, we can clearly see the need to go beyond pure SBFL and take other information, such as information from the bug report or from version history of the code lines, into consideration.

show abstract

“…As mentioned by Papadakis et al . [1], recently, a few researchers mutate program code or runtime states to explore various mutant behaviour for fault localization [33–35], for automated program repairs [36] and for automated program improvements [37, 38]. There exist few test generation techniques [39], [40] that utilize runtime values and execution paths observed from mutated program executions.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, mutation analysis has been used to precisely localize a fault in a program. Mutation‐based fault localization [33–35, 42, 43] locates a fault in the target program code by observing how the behaviours of the faulty program change according to the program code mutation.…”

Section: Related Workmentioning

confidence: 99%

DEMINER: test generation for high test coverage through mutant exploration

Kim

Hong

2019

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

Summary Most software testing techniques test a target program as it is and fail to utilize valuable information of diverse test executions on many variants/mutants of the original program in test generation. This paper proposes a new test generation technique DEMINER, which utilizes mutant executions to guide test generation on the original program for high test coverage. DEMINER first generates various mutants of an original target program and then extracts runtime information of mutant executions, which covered unreached branches by the mutation effects. Using the obtained runtime information, DEMINER inserts guideposts, artificial branches to replay the observed mutation effects, to the original target programs. Finally, DEMINER runs automated test generation on the original program with guideposts and achieves higher test coverage. We implemented DEMINER for C programs through software mutation and guided test generation such as concolic testing and fuzzing. We have shown the effectiveness of DEMINER on six real‐world target programs: Busybox‐ls, Busybox‐printf, Coreutils‐sort, GNU‐find, GNU‐grep and GNU‐sed. The experiment results show that DEMINER improved branch coverage by 63.4% and 19.6% compared with those of the conventional concolic testing techniques and the conventional fuzzing techniques on average, respectively.

show abstract

MUSEUM: Debugging real-world multilingual programs using mutation analysis

Cited by 29 publications

References 5 publications

FDG: A Precise Measurement of Fault Diagnosability Gain of Test Cases

FDG: A Precise Measurement of Fault Diagnosability Gain of Test Cases

An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems

DEMINER: test generation for high test coverage through mutant exploration

Contact Info

Product

Resources

About