Feature location benchmark for extractive software product line adoption research using realistic and synthetic Eclipse variants

Abstract

Context: It is common belief that high impact research in software reuse requires assessment in non-trivial, comparable, and reproducible settings. However, software artefacts and common representations are usually unavailable. Also, establishing a representative ground truth is a challenging and debatable subject. Feature location in the context of software families, which is key for software product line adoption, is a research field that is becoming more mature with a high proliferation of techniques. Objective: We present EFLBench, a benchmark and a framework to provide a common ground for the evaluation of feature location techniques in families of systems. Method: EFLBench leverages the efforts made by the Eclipse Community which provides feature-based family artefacts and their plugin-based implementations. Eclipse is an active and non-trivial project and thus, it establishes an unbiased ground truth which is realistic and challenging. Results: EFLBench is publicly available and supports all tasks for feature location techniques integration, benchmark construction and benchmark usage. We demonstrate its usage, simplicity and reproducibility by comparing four techniques in Eclipse releases. As an extension of our previously published work, we consider a decade of Eclipse releases and we also contribute an approach to automatically generate synthetic Eclipse variants to benchmark feature location techniques in tailored settings. We present and discuss three strategies for this automatic generation and we present the results using different settings. Conclusion: EFLBench is a contribution to foster the research in feature location in families of systems providing a common framework and a set of baseline techniques and results.

Analyzing the Impact of Refactoring Variants on Feature Location

“…As shown by Martinez et al [16], feature location in the extractive approach of SPLE can be illustrated by the Fig. 1.…”

“…None of these feature location techniques consider the changes of refactorings. There are a number of benchmarks [16,15,6] that have been used to evaluate FL techniques performance [5,24,18], while we used one of them to estimate the performance of FL techniques with refactored variants. We have previously presented the ArgoUML SPL benchmark, which have been used to evaluate many FL techniques [5,18,19].…”

“…One of the main steps in the extractive approach is what is referred to as Feature Location (FL). FL is a traceability recovery task for identifying the implementation elements associated to each feature among the family variants that are created using clone-and-own [16].…”

Evolving System Families in Space and Time

“…In addition, our empirical analysis showed that features have been changed over time by adding or removing an entire file and/or fewer lines. The information retrieved from the mapping between artifacts and features in relation to the relevant information was compared by measuring precision, recall, and F1 score, which are metrics commonly used to evaluate feature location techniques [5,21,23]. In summary, we achieved higher precision and recall for information retrieved from the systems' variants, ranging from 99%-100% and 93%-99%, respectively, at filelevel and line-level granularity.…”

Maintenance Web Based Applications Using Feature Location

“…Another FL research used crowd-based screen cast to help programmers find a sample of code from video tutorial [2]. Many others were used information retrieval (IR) to discover FL within a source code [3]; [4][5][6][7]. Most of its used Java-based code as data sets of their experiment.…”