A replication study on the effects of weighted moving windows for software effort estimation

Abstract

Context: Recent studies have shown that estimation accuracy can be affected by only using a window of recent projects as training data for building an effort estimation model. The idea has been extended for regression-based estimation by weighting projects differently according to their order within the window. This significantly improved the accuracy of estimation in a single-company dataset from the ISBSG repository. Objective: To investigate the effects on estimation accuracy of using weighted moving windows with a new dataset, and compare results across datasets. Method: Using a dataset drawn from the Finnish dataset (studied previously with regard to windows but not with weighting), and using a fixed-size window policy, we examine the effect on estimation accuracy of using weighted moving windows. Results: The use of weighting functions could improve the estimation accuracy significantly, compared to using unweighted windows, with larger window sizes. The steepness of the weighting functions affects their effectiveness. However, in this dataset it is better to use a growing portfolio (retaining all past projects as training data) than to use windows. Conclusions:The results reinforce previous studies: the use of weighting functions can significantly improve the accuracy of regression-based estimation, compared to not using weighting, but in this dataset the use of moving windows reduces estimation accuracy.

A Stratification and Sampling Model for Bellwether Moving Window

“…The moving window approach has recently been supplemented with weighting functions which further improve the estimation accuracy of effort estimation models. This theory was postulated by Amasaki and Lokan [10][2] [18] [17] and was empirically confirmed to improve the estimation accuracy when using large-sized moving windows. These previous studies found software effort estimation (SEE) models built using recently completed projects (weighted/unweighted moving window) superior to models that use all available historical projects (referred to as the growing portfolio).…”

“…The moving window theory is based on the assumption that recent projects are likely to share similar characteristics with new projects. Recent studies [10][2] [7] [17] support the theory by Lokan and Mendes that, the use of moving window as the training set improves the estimation accuracy of effort estimation models. The moving window approach has recently been supplemented with weighting functions which further improve the estimation accuracy of effort estimation models.…”

“…The window age refers to the elapsed time of projects (within the moving window) that has existed for not more than t calendar years or calendar months [7]. Previous studies [2][5] [10] [18] randomly used different moving window sizes such as 20, 30, 40…120 projects. Similarly, different moving window ages (such as 1 year, 1.5 years, 2 years…) were used by previous studies [7][12] [17].…”

“…Similarly, different moving window ages (such as 1 year, 1.5 years, 2 years…) were used by previous studies [7][12] [17]. These arbitrary window sizing and window aging parameters significantly affect the estimation accuracy irrespective of the weighting functions used [2] [10]. This is mainly due to the variations in the sample sizes drawn from the chronological datasets.…”

“…This is mainly due to the variations in the sample sizes drawn from the chronological datasets. Based on previous moving window studies [10][5][2] [12], this study aims at addressing the unstable window sizing and window aging constraint to further improve the estimation accuracy of predictive models. The existence of exemplary projects referred to as Bellwether has successfully been used to improve the relative estimation accuracy in software defect prediction by Krishna et al [1].…”