Assisting Bug Report Assignment Using Automated Fault Localisation: An Industrial Case Study

Sohn, Jeongju; An, Gabin; Hong, Jingun; Hwang, Dongwon; Yoo, Shin

doi:10.1109/icst49551.2021.00041

Cited by 7 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on these results, we argue that while our outcome may not be as good as those reported by prior fault localisation studies [39], [40], that is mainly due to the inherently low diagnosability of a test suite (e.g., covering too many classes in the same fashion). This test-suite adequacy issue commonly exists in the fault localisation field [51] and is not limited to flaky class identification. Hence, we posit that the performance of our approach can improve along with the advances in fault localisation techniques.…”

Section: Discussionmentioning

confidence: 99%

“…For this aim, we use voting as our ensemble learning method. We opted for voting since it does not require an additional cost for model generation and its effectiveness has already been demonstrated by previous fault localisation studies [51], [52] 2) Approach: Voting between models is performed in two phases: candidate selection and voting. During the candidate selection phase, all the participating models compute their own suspciousness scores for the candidates.…”

Section: B Rq2 -Code and Change Metrics 1) Motivationmentioning

confidence: 99%

“…Previous studies on voting-based FL showed that varying the number of votes that each candidate receives based on its actual rank in individual models can improve the localisation performance even further [51], [52]. Hence, rather than assigning the same number of votes to each candidate, we allow individual models to cast a different number of votes for each candidate based on its location in the ranking.…”

Section: B Rq2 -Code and Change Metrics 1) Motivationmentioning

confidence: 99%

See 2 more Smart Citations

What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness

Habchi¹,

Haben²,

Sohn³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Flaky tests are defined as tests that manifest nondeterministic behaviour by passing and failing intermittently for the same version of the code. These tests cripple continuous integration with false alerts that waste developers' time and break their trust in regression testing. To mitigate the effects of flakiness, both researchers and industrial experts proposed strategies and tools to detect and isolate flaky tests. However, flaky tests are rarely fixed as developers struggle to localise and understand their causes. Additionally, developers working with large codebases often need to know the sources of nondeterminism to preserve code quality, i.e., avoid introducing technical debt linked with non-deterministic behaviour, and to avoid introducing new flaky tests. To aid with these tasks, we propose re-targeting Fault Localisation techniques to the flaky component localisation problem, i.e., pinpointing program classes that cause the non-deterministic behaviour of flaky tests. In particular, we employ Spectrum-Based Fault Localisation (SBFL), a coverage-based fault localisation technique commonly adopted for its simplicity and effectiveness. We also utilise other data sources, such as change history and static code metrics, to further improve the localisation. Our results show that augmenting SBFL with change and code metrics ranks flaky classes in the top-1 and top-5 suggestions, in 26% and 47% of the cases. Overall, we successfully reduced the average number of classes inspected to locate the first flaky class to 19% of the total number of classes covered by flaky tests. Our results also show that localisation methods are effective in major flakiness categories, such as concurrency and asynchronous waits, indicating their general ability to identify flaky components.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: B Rq2 -Code and Change Metrics 1) Motivationmentioning

confidence: 99%

Section: B Rq2 -Code and Change Metrics 1) Motivationmentioning

confidence: 99%

See 1 more Smart Citation

What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness

Habchi¹,

Haben²,

Sohn³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…While precise, dynamic test execution information such test coverage is not always available, mainly due to the difficulty and the cost of data collection [1,17,18,26,37]. For instance, test coverage, one of the most frequently used dynamic code analysis information, requires instrumentation, which may or may not be possible to perform in given environments.…”

Section: Introductionmentioning

confidence: 99%

“…For instance, test coverage, one of the most frequently used dynamic code analysis information, requires instrumentation, which may or may not be possible to perform in given environments. Even when the employed environment supports test coverage collection, this functionality is often turned-off, as it introduces significant overheads in order to log the related information [1,18,26]. Furthermore, when developers are under a fast-release cycle, they are unlikely to have enough time for data collection at the first place [5,18,37].…”

Section: Introductionmentioning

confidence: 99%

Using Evolutionary Coupling to Establish Relevance Links Between Tests and Code Units. A case study on fault localization

Sohn¹,

Papadakis²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Many software engineering techniques, such as fault localization, operate based on relevance relationships between tests and code. These relationships are often inferred through the use of dynamic test execution information (test execution traces) that approximate the link between relevant code units and asserted, by the tests, program behaviour. Unfortunately, in practice dynamic information is not always available due to the overheads introduced by the instrumentation or the nature of the production environments. To deal with this issue, we propose CEMENT, a static technique that automatically infers such test and code relationships given the projects' evolution. The key idea is that developers make relevant changes on test and code units at the same period of time, i.e., co-evolution of tests and code units reflects a probable link between them. We evaluate CEMENT on 15 open source projects and show that it indeed captures relevant links. Additionally, we perform a fault localization case study where we compare CEMENT with an existing Information Retrieval-based Fault Localization (IRFL) technique and show that it achieves comparable performance. A further analysis of our results reveals a small overlap between the faults successfully localized by the two approaches suggesting complementarity. In particular, out of the 39 successfully localized faults, two are common while CEMENT and IRFL localize 16 and 21. These results demonstrate that test and code evolutionary coupling can effectively support test and debugging activities.

show abstract

A Systematic Exploration of Mutation‐Based Fault Localization Formulae

Wang,

Wei,

Chen

et al. 2024

Software Testing Verif & Rel

View full text Add to dashboard Cite

Fault localization (FL) aims to automatically find the location of bugs in a software program. In the family of FL approaches, spectrum‐based fault localization (SBFL) is the most widely used and has been extensively studied, which computes the suspicious scores of a code element to be buggy via test coverage information. Mutation‐based fault localization (MBFL) further analyses the relationship between mutants and test results. Although MBFL involves more information, it also faces the following limitations. (1) To precisely evaluate suspicious sources, SBFL approaches proposed dozens of formulae based on coverage, while only a few of them have been adopted by MBFL. (2) The current MBFL approaches are based on the assumption that the capability in localizing bugs of each mutant is the same, so they assign the same weight to the mutants that change the test outputs and consider the mutant from the same code element equivalent. In this paper, we intend to enrich the MBFL family by the following two approaches. First, we collect 25 typical SBFL formulae and transform them into MBFL versions. Second, we propose new assumptions that the mutants should have different weights in computing suspicious sources and propose BLMu, a novel MBFL approach that treats mutants differently. We also propose novel metrics for MBFL by considering the high cost of mutation analysis. We perform large‐scale experiments by evaluating all the MBFL approaches against 395 real‐world Java bugs of the Defects4J benchmark. Our evaluation results reveal that BLMu demonstrates a substantial improvement over both MUSE and Metallaxis, the most popular MBFL approaches, at both the statement level and the method level. Specifically, in terms of Top‐1, BLMu improves by 106% at the statement level and 69% at the method level. When compared with other types of FL approaches, MBFL outperforms typical SBFL approaches while still far behind the state‐of‐the‐art learning‐based FL approaches.

show abstract

Assisting Bug Report Assignment Using Automated Fault Localisation: An Industrial Case Study

Cited by 7 publications

References 28 publications

What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness

What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness

Using Evolutionary Coupling to Establish Relevance Links Between Tests and Code Units. A case study on fault localization

A Systematic Exploration of Mutation‐Based Fault Localization Formulae

Contact Info

Product

Resources

About