Investigating the Significance of the Bellwether Effect to Improve Software Effort Prediction: Further Empirical Study

Abstract

Context: In addressing how best to estimate how much effort is required to develop software, a recent study found that using exemplary and recently completed projects [forming Bellwether moving windows (BMW)] in software effort prediction (SEP) models leads to relatively improved accuracy. More studies need to be conducted to determine whether the BMW yields improved accuracy in general, since different sizing and aging parameters of the BMW are known to affect accuracy. Objective: To investigate the existence of exemplary projects (Bellwethers) with defined window size and age parameters, and whether their use in SEP improves prediction accuracy. Method: We empirically investigate the moving window assumption based on the theory that the prediction outcome of a future event depends on the outcomes of prior events. Sampling of Bellwethers was undertaken using three introduced Bellwether methods (SSPM, SysSam, and RandSam). The ergodic Markov chain was used to determine the stationarity of the Bell-wethers. Results: Empirical results show that 1) Bellwethers exist in SEP and 2) the BMW has an approximate size of 50 to 80 exemplary projects that should not be more than 2 years old relative to the new projects to be estimated. Conclusion: The study's results add further weight to the recommended use of Bellwethers for improved prediction accuracy in SEP.

Adaptive Discretization Using Golden Section to Aid Outlier Detection for Software Development Effort Estimation

“…The performance comparison with previous works is a difficult task for researchers in the field of SDEE due to lack of complete implementation details, discrepancies in usage of validation schemes and performance measures. However, we have compared the performance of our proposed models with ML models obtained as per previous study [80] and IQR [1] and Cooks distance [1], [46], [47] outlier identification and removal techniques based models. In addition to this, we have compared the results with well-known state-of-art baseline SDEE method Automatically Transformed Linear Model (ATLM) [81].…”

“…Cooks' distance measures the change in residual values of all data points from full regression model to refitted regression model after omitting i th data point [39]. To identify outliers using Cooks' distance, we have followed the same approach based on previous studies [46], [1], [47].…”

“…1) with all data points, i.e. without using any outlier identification and removal method 2) with only non-outlier data points identified after using proposed method GSAD 3) with only non-outlier data points identified after using existing outlier identification method IQR [1] 4) with only non-outlier data points identified after using existing outlier identification method Cooks based on studies [1], [46], [47] Table 5 lists all the parameter values that have been investigated for each SDEE method in this study. The best hyperparameters for each learning algorithm have been selected using grid-search technique with 10x3 repeated cross-fold validation.…”

“…A variety of performance measures have been used by researchers to showcase the performances of SDEE models. The number of outliers identified using GSAD, IQR [1] and Cooks [1], [46], [47] These performance measures do not measure and/ or represent the same facet of performance. The reliability in measurement of performance largely depends on performance evaluation measure [72], [20].…”

Learning Actionable Analytics from Multiple Software Projects

“…-Quality = defects reported by the development team; -Improvement = lowered likelihood of future defects. This paper advocates the use of the bellwether effect (Krishna et al, 2016(Krishna et al, , 2017Mensah et al, 2018) to generate plans. This effect states that: " .…”