A systematic review of effect size in software engineering experiments

Kampenes, Vigdis By; Dybå, Tore; Hannay, Jo Erskine; Sjøberg, Dag I. K.

doi:10.1016/j.infsof.2007.02.015

Cited by 301 publications

(236 citation statements)

References 36 publications

Supporting

Mentioning

224

Contrasting

Unclassified

Order By: Relevance

“…The effect sizes calculated for the aforementioned studies [15,18] are r = −.17 and r = −.04, respectively. Both effect sizes are considered small, according to guidelines for empirical studies in software engineering given by Kempenes et al [89]. The effect size estimate, reported in this paper in Section 5.3, is small as well (only 0.3 percent of the variance in BC can be explained by DevM eth).…”

Section: Discussion Conclusion and Future Workmentioning

confidence: 94%

The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment

Madeyski

2010

Information and Software Technology

View full text Add to dashboard Cite

Background : Test-First programming is regarded as one of the software development practices that can make unit tests to be more rigorous, thorough and effective in fault detection. Code coverage measures can be useful as indicators of the thoroughness of unit test suites, while mutation testing turned out to be effective at finding faults. Objective: This paper presents an experiment in which Test-First vs TestLast programming practices are examined with regard to branch coverage and mutation score indicator of unit tests. Method : Student subjects were randomly assigned to Test-First and TestLast groups. In order to further reduce pre-existing differences among subjects, and to get a more sensitive measure of our experimental effect, multivariate analysis of covariance was performed. Results: Multivariate tests results indicate that there is no statistically significant difference between Test-First and Test-Last practices on the combined dependent variables, i.e. branch coverage and mutation score indicator, (F (2, 9) = .52, p > .05), even if we control for the pre-test results, the subjects' experience, and when the subjects who showed deviations from the assigned programming technique are excluded from the analysis. Conclusion: According to the preliminary results presented in this paper, the benefits of the Test-First practice in this specific context can be considered minor. Limitation: It is probably the first-ever experimental evaluation of the impact of Test-First programming on mutation score indicator of unit tests and further experimentation is needed to establish evidence.

show abstract

Section: Discussion Conclusion and Future Workmentioning

confidence: 94%

The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment

Madeyski

2010

Information and Software Technology

View full text Add to dashboard Cite

show abstract

“…To allow an interpretation of effect sizes in a software engineering context, Kampenes et al (2007) therefore proposed magnitude labels based on a systematic review of effect size in 92 software engineering controlled experiments. The sample size is limited but gives a rough estimation of what constitutes small, medium and large effect sizes in the software engineering domain.…”

Section: Guidelines For Interpreting Effect Size Magnitudementioning

confidence: 99%

“…The study described here was designed to investigate the use of multi-site studies in order to address the problems of small sample sizes in Software Engineering experiments, see Dybå et al (2006) and Kampenes et al (2007). The topic for the multi-site experiment concerned the extent to which structured abstracts were clearer and more complete than conventional abstracts.…”

Section: Background To the Multi-site Experimentsmentioning

confidence: 99%

Robust Statistical Methods for Empirical Software Engineering

et al. 2016

View full text Add to dashboard Cite

There have been many changes in statistical theory in the past 30 years, including increased evidence that non-robust methods may fail to detect important results. The statistical advice available to software engineering researchers needs to be updated to address these issues. This paper aims both to explain the new results in the area of robust analysis methods and to provide a large-scale worked example of the new methods. We summarise the results of analyses of the Type 1 error efficiency and power of standard parametric and non-parametric statistical tests when applied to non-normal data sets. We identify parametric and non-parametric methods that are robust to non-normality. We present an analysis of a large-scale software engineering experiment to illustrate their use. We illustrate the use of kernel density plots, and parametric and non-parametric methods using four different software engineering data sets. We explain why the methods are necessary and the rationale for selecting a specific analysis. We suggest using kernel density plots rather than box plots to visualise data distributions. For parametric analysis, we recommend trimmed means, which can support reliable tests of the differences between the central location of two or

show abstract

“…This metric assigns more weight to the large N studies and is calculated using formula (5) [46]. For studies in the Software Engineering field, the effect size calculated using point-biserial r correlation is rated as follows: small (0-0.193), medium (0.194-0.456), or large (above 0.456) [49]. The Hunter-Schmidt Method also allows a chi-square significance test of homogeneity across the studies to be performed.…”

Section: Meta-analysismentioning

confidence: 99%

“…For each experiment, this table reports the population effect size through the unweighted mean r after applying Mullen and Rosenthal's cluster analysis method [58]. This effect has also been classified as Small (S), Medium (M) and Large (L) following the same classification [49] for effect sizes as those applied to the weighted effect size in Table 6.…”

Section: Meta-analysismentioning

confidence: 99%

Validating a model-driven software architecture evaluation and improvement method: A family of experiments

González-Huerta

Insfrán

Abrahão

et al. 2015

Information and Software Technology

View full text Add to dashboard Cite

ElsevierGonzález Huerta, J.; Insfrán Pelozo, CE.; Abrahao Gonzales, SM.; Scanniello, G. (2015 Context: Software architectures should be evaluated during the early stages of software development in order to verify whether the Non-Functional Requirements (NFRs) of the product can be fulfilled. This activity is even more crucial in Software Product Line (SPL) development, since it is also necessary to identify whether the NFRs of a particular product can be achieved by exercising the variation mechanisms provided by the product line architecture or whether additional transformations are required. These issues have motivated us to propose QuaDAI, a method for the derivation, evaluation and improvement of software architectures in model-driven SPL development.Objective: We present in this paper the results of a family of four experiments carried out to empirically validate the evaluation and improvement strategy of QuaDAI. Method:The family of experiments was carried out by 92 participants: Computer Science Master's and undergraduate students from Spain and Italy. The goal was to compare the effectiveness, efficiency, perceived ease of use, perceived usefulness and intention to use with regard to participants using the evaluation and improvement strategy of QuaDAI as opposed to the Architecture Tradeoff Analysis Method (ATAM). Results:The main result was that the participants produced their best results when applying QuaDAI, signifying that the participants obtained architectures with better values for the NFRs faster, and that they found the method easier to use, more useful and more likely to be used. The results of the meta-analysis carried out to aggregate the results obtained in the individual experiments also confirmed these results. Conclusions:The results support the hypothesis that QuaDAI would achieve better results than ATAM in the experiments and that QuaDAI can be considered as a promising approach with which to perform architectural evaluations that occur after the product architecture derivation in model-driven SPL development processes when carried out by novice software evaluators.

show abstract

A systematic review of effect size in software engineering experiments

Cited by 301 publications

References 36 publications

The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment

The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment

Robust Statistical Methods for Empirical Software Engineering

Validating a model-driven software architecture evaluation and improvement method: A family of experiments

Contact Info

Product

Resources

About