Multi-Objective Software Effort Estimation: A Replication Study

Abstract

Replication studies increase our confidence in previous results when the findings are similar each time, and help mature our knowledge by addressing both internal and external validity aspects. However, these studies are still rare in certain software engineering fields. In this paper, we replicate and extend a previous study, which denotes the current state-of-the-art for multi-objective software effort estimation, namely CoGEE. We investigate the original research questions with an independent implementation and the inclusion of a more robust baseline (LP4EE), carried out by the first author, who was not involved in the original study. Through this replication, we strengthen both the internal and external validity of the original study. We also answer two new research questions investigating the effectiveness of CoGEE by using four additional evolutionary algorithms (i.e., IBEA, MOCell, NSGA-III, SPEA2) and a well-known Java framework for evolutionary computation, namely JMetal (rather than the previously used R software), which allows us to strengthen the external validity of the original study. The results of our replication confirm that: (1) CoGEE outperforms both baseline and state-of-the-art benchmarks statistically significantly (p < 0.001); (2) CoGEE's multi-objective nature makes it able to reach such a good performance; (3) CoGEE's estimation errors lie within claimed industrial human-expert-based thresholds. Moreover, our new results show that the effectiveness of CoGEE is generally not limited to nor dependent on the choice of the multi-objective algorithm. Using CoGEE with either NSGA-II, NSGA-III, or MOCell produces human competitive results in less than a minute. The Java version of CoGEE has decreased the running time by over 99.8% with respect to its R counterpart. We have made publicly available the Java code of CoGEE to ease its adoption, as well as, the data used in this study in order to allow for future replication and extension of our work.

MEG: Multi-objective Ensemble Generation for Software Defect Prediction

“…These include the use of other diversity measures, and performance measures such as the classification stability [67] and the effort required for source code inspection [33,44]. • Investigating MOEAs other than NSGA-II (e.g., MOEA/D, IBEA, MOCell, SPEA2) in order to assess to what extent the effectiveness of MEG varies depending on the underlying multi-objective algorithm [69]. • Investigating Deep-Learning (DL) in combination with MEG, to assess if this would further increase the ensemble prediction performance.…”

DeLag: Using Multi-Objective Optimization to Enhance the Detection of Latency Degradation Patterns in Service-Based Systems

“…In particular, DeLag uses NSGA-II [30] to build (successively-improved) Pareto-optimal solutions, while seeking new non-dominating pattern sets. We rely on NSGA-II because it has been shown to be effective on a wide variety of multi-objective search problems both within and outside the software engineering domain [26], [31], [32], [33], [34], [35], [36]. Using different search algorithms might potentially improve DeLag effectiveness, however, given the extensiveness of our experimental evaluation, we deem this investigation out of the scope of this work, and we leave it for future studies.…”

Investigating the Effectiveness of Clustering for Story Point Estimation

“…Similar to previous studies on software effort estimation, we use measurements that are built upon the error (or absolute error) between the predicted value and the actual value. These measures (defined in Equations 1, 2 and 3) have been found in previous work to be unbiased towards under-or overestimations [6], [14], [30]- [32]. These measure are the Mean Absolute Error (MAE), the Median Absolute Error (MdAE), and the Standard Accuracy (SA).…”

“…Respectively, Â12 higher than 0.5 means that the first algorithm is more likely to produce better predictions. The effect size is considered small for 0.6 ≤ Â12 < 0.7, medium for 0.7 ≤ Â12 ≤ 0.8, and large for Â12 ≥ 0.8, although these thresholds are not definitive [6].…”

“…To this end, teams mainly rely on expert estimation methods like Planning Poker and Delphi [3]. However, expert judgment has been shown to be prone to bias due to its reliance on subjective assessment [4]- [6]. This motivated several research endeavours to find automated ways to predict story points of a task given its features with the aim of avoiding inaccurate estimations by human judgement, in addition to, and more importantly to an agile team, producing consistent estimations throughout the project's lifecycle.…”