Are datasets for information retrieval-based bug localization techniques trustworthy?

Kim, Misoo; Lee, Eunseok

doi:10.1007/s10664-021-09946-8

Cited by 14 publications

(9 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kochhar et al [30] investigate potential biases in datasets used for IRFL experiments, showing that the inclusion of bug reports where the reporter already localizes the fault causes a bias and distorts retrieval performance results. Kim and Lee [23] point out issues regarding representability and generalizability of benchmarks used in IRFL. Hirsch and Hofer [27] survey public bug localization benchmarks and identify issues in data availability, data quality, and reproducibility.…”

Section: Related Workmentioning

confidence: 99%

“…Along with MRR and TopN, MAP is one of the most popular performance metrics used in IRFL research [23]. The Retrieved documents in the context of IRFL experiments is the set of bug fix locations in the codebase, most often presented at file level, and commonly referred to as ground truth in IRFL.…”

Section: Definitionsmentioning

confidence: 99%

“…To make the numerous approaches comparable, well established retrieval performance metrics from the wider field of IR are employed. The most popular metrics in IRFL are Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), and TopN [23]. Researchers in the IR community have already discussed the characteristics of these retrieval performance metrics [24], [25] in depth, including discussions and evaluations on statistics around these metrics [26].…”

Section: Introductionmentioning

confidence: 99%

“…Researchers in the IR community have already discussed the characteristics of these retrieval performance metrics [24], [25] in depth, including discussions and evaluations on statistics around these metrics [26]. While researchers in the narrower field of IRFL have already investigated issues related to datasets [23], [27], evaluation methodologies [28], [29], and common assumptions in IRFL [30], [31], the limitations of the employed metrics and practical considerations have stayed untouched. To the best of our knowledge, there has been no investigation of the metrics' suitability for the field of IRFL, and no investigation into the practical application of these metrics in the context of IRFL research.…”

Section: Introductionmentioning

confidence: 99%

“…We show that the conditions that allow for such erroneous results due to improper metric implementation are common in Bench4BL. While we perform our demonstration using the Bench4BL dataset, we expect these conditions to be met in the majority of IRFL datasets/benchmarks [23].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

The MAP Metric in Information Retrieval Fault Localization

Hirsch,

Hofer

2023

2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)

View full text Add to dashboard Cite

The MAP (Mean Average Precision) metric is one of the most popular performance metrics in the field of Information Retrieval Fault Localization (IRFL). However, there are problematic implementations of this MAP metric used in IRFL research. These implementations deviate from the text book definitions of MAP, rendering the metric sensitive to the truncation of retrieval results and inaccuracies and impurities of the used datasets. The application of such a deviating metric can lead to performance overestimation. This can pose a problem for comparability, transferability, and validity of IRFL performance results. In this paper, we discuss the definition and mathematical properties of MAP and common deviations and pitfalls in its implementation. We investigate and discuss the conditions enabling such overestimation: the truncation of retrieval results in combination with ground truths spanning multiple files and improper handling of undefined AP results. We demonstrate the overestimation effects using the Bench4BL benchmark and five well known IRFL techniques. Our results indicate that a flawed implementation of the MAP metric can lead to an overestimation of the IRFL performance, in extreme cases by up to 70 %. We argue for a strict adherence to the text book version of MAP with the extension of undefined AP values to be set to 0 for all IRFL experiments. We hope that this work will help to improve comparability and transferability in IRFL research.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Definitionsmentioning

confidence: 99%