The impact of parameter tuning on software effort estimation using learning machines

Song, Liying; Minku, Leandro L.; Yao, Xin

doi:10.1145/2499393.2499394

Cited by 70 publications

(52 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, other clustering methods, base learners, project input attributes, project features for clustering, parameter values [41] and (automated) tuning procedures [1,9] could be investigated, besides the proposal of a more streamlined approach to update CC models.…”

Section: Discussionmentioning

confidence: 99%

“…Therefore, they have similar threats to validity as follows. When using machine learning approaches, it is important that the approaches being compared use fair parameter choices in comparison to each other in order to address internal validity [29,41]. In this paper, both the RTs used as WC learners and within Dycom and Clustering Dycom used the same parameters, which were the ones more likely to obtain good results in the literature [34].…”

Section: Threats To Validitymentioning

confidence: 99%

See 1 more Smart Citation

Clustering Dycom

Minku

Hou

2017

Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering

Self Cite

View full text Add to dashboard Cite

Background: Software Effort Estimation (SEE) can be formulated as an online learning problem, where new projects are completed over time and may become available for training. In this scenario, a Cross-Company (CC) SEE approach called Dycom can drastically reduce the number of Within-Company (WC) projects needed for training, saving the high cost of collecting such training projects. However, Dycom relies on splitting CC projects into different subsets in order to create its CC models. Such splitting can have a significant impact on Dycom's predictive performance. Aims: This paper investigates whether clustering methods can be used to help finding good CC splits for Dycom. Method: Dycom is extended to use clustering methods for creating the CC subsets. Three different clustering methods are investigated, namely Hierarchical Clustering, K-Means, and ExpectationMaximisation. Clustering Dycom is compared against the original Dycom with CC subsets of different sizes, based on four SEE databases. A baseline WC model is also included in the analysis. Results: Clustering Dycom with K-Means can potentially help to split the CC projects, managing to achieve similar or better predictive performance than Dycom. However, K-Means still requires the number of CC subsets to be pre-defined, and a poor choice can negatively affect predictive performance. EM enables Dycom to automatically set the number of CC subsets while still maintaining or improving predictive performance with respect to the baseline WC model. Clustering Dycom with Hierarchical Clustering did not offer significant advantage in terms of predictive performance. Conclusion: Clustering methods can be an effective way to automatically generate Dycom's CC subsets.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Threats To Validitymentioning

confidence: 99%

Clustering Dycom

Minku

Hou

2017

Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…Finally, they concluded that learning learners is an active research area and much further work is required before we can understand the costs and benefits of this approach. In (Song et al 2013) the authors proposed a framework to investigate to what extent parameter settings affect the performance of learning machines in software effort estimation, and what learning machines are more sensitive to their parameters. They concluded that different learning machines have different sensitivity to their parameter settings.…”

Section: Framework For Benchmarking Prediction Modelsmentioning

confidence: 99%

A genetic algorithm based framework for software effort prediction

Murillo-Morera

Quesada-López

Castro-Herrera

et al. 2017

J Softw Eng Res Dev

View full text Add to dashboard Cite

Background: Several prediction models have been proposed in the literature using different techniques obtaining different results in different contexts. The need for accurate effort predictions for projects is one of the most critical and complex issues in the software industry. The automated selection and the combination of techniques in alternative ways could improve the overall accuracy of the prediction models. Objectives: In this study, we validate an automated genetic framework, and then conduct a sensitivity analysis across different genetic configurations. Following is the comparison of the framework with a baseline random guessing and an exhaustive framework. Lastly, we investigate the performance results of the best learning schemes. Methods: In total, six hundred learning schemes that include the combination of eight data preprocessors, five attribute selectors and fifteen modeling techniques represent our search space. The genetic framework, through the elitism technique, selects the best learning schemes automatically. The best learning scheme in this context means the combination of data preprocessing + attribute selection + learning algorithm with the highest coefficient correlation possible. The selected learning schemes are applied to eight datasets extracted from the ISBSG R12 Dataset. Results: The genetic framework performs as good as an exhaustive framework. The analysis of the standardized accuracy (SA) measure revealed that all best learning schemes selected by the genetic framework outperforms the baseline random guessing by 45-80%. The sensitivity analysis confirms the stability between different genetic configurations. Conclusions: The genetic framework is stable, performs better than a random guessing approach, and is as good as an exhaustive framework. Our results confirm previous ones in the field, simple regression techniques with transformations could perform as well as nonlinear techniques, and ensembles of learning machines techniques such as SMO, M5P or M5R could optimize effort predictions.

show abstract

“…1 At time t, we may wish to estimate projects p t+1 and p t+2 based on a model trained with all projects completed up to time t. Then, once we reach time t + 1, we may wish to provide an updated prediction for project p t+2 based on a model trained with all projects completed up to time t + 1. This approach has been used, for example, in a study of the impact of parameter tuning on SEE (Song et al 2013). This study was based on three WC datasets and five ML approaches (MLP, Bag + MLP, RT, Bag + RT and k-NN).…”

Section: Chronological Splitting Approachesmentioning

confidence: 99%

“…This section analyses DCL's sensitivity to parameters, revealing whether DCL's predictive performance can be influenced by parameter tuning (Song et al 2013) and which parameters influence its predictive performance the most.…”

Section: The Impact Of Dcl's Parametersmentioning

confidence: 99%

Which models of the past are relevant to the present? A software effort estimation approach to exploiting useful past models

Minku

Yao

2016

Autom Softw Eng

Self Cite

View full text Add to dashboard Cite

Software Effort Estimation (SEE) models can be used for decision-support by software managers to determine the effort required to develop a software project. They are created based on data describing projects completed in the past. Such data could include past projects from within the company that we are interested in (WC projects) and/or from other companies (cross-company, i.e., CC projects). In particular, the use of CC data has been investigated in an attempt to overcome limitations caused by the typically small size of WC datasets. However, software companies operate in non-stationary environments, where changes may affect the typical effort required to develop software projects. Our previous work showed that both WC and CC models of the past can become more or less useful over time, i.e., they can sometimes be helpful and sometimes misleading. So, how can we know if and when a model created based on past data represents well the current projects being estimated? We propose an approach called Dynamic Cross-company Learning (DCL) to dynamically identify which WC or CC past models are most useful for making predictions to a given company at the present. DCL automatically emphasizes the predictions given by these models in order to improve predictive performance. Our experiments comparing DCL against existing WC and CC approaches show that DCL is successful in improving SEE by A preliminary version of this work appeared as Minku and Yao (2012a

show abstract

The impact of parameter tuning on software effort estimation using learning machines

Cited by 70 publications

References 33 publications

Clustering Dycom

Clustering Dycom

A genetic algorithm based framework for software effort prediction

Which models of the past are relevant to the present? A software effort estimation approach to exploiting useful past models

Contact Info

Product

Resources

About