Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment

Kitchenham, Barbara; Madeyski, Lech; Brereton, Pearl

doi:10.1007/s10664-019-09747-0

Cited by 33 publications

(17 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The advantage to the qualitative approach is that it can use more nuance in labeling design discussions at more specific level; the tradeoff of course is such labeling is labourintensive. Summary A true meta-analysis [24], [30] of the related work is not feasible in the area of design mining. Conventional metaanalysis is applied on primary studies that conduct experiments in order to support inference to a population, which is not the study logic of the studies considered here.…”

Section: B Mining Design Discussionmentioning

confidence: 99%

Cross-Dataset Design Discussion Mining

Mahadi

Tongay

Ernst

2020

2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)

View full text Add to dashboard Cite

Being able to identify software discussions that are primarily about design-which we call design mining-can improve documentation and maintenance of software systems. Existing design mining approaches have good classification performance using natural language processing (NLP) techniques, but the conclusion stability of these approaches is generally poor. A classifier trained on a given dataset of software projects has so far not worked well on different artifacts or different datasets. In this study, we replicate and synthesize these earlier results in a meta-analysis. We then apply recent work in transfer learning for NLP to the problem of design mining. However, for our datasets, these deep transfer learning classifiers perform no better than less complex classifiers. We conclude by discussing some reasons behind the transfer learning approach to design mining.

show abstract

Section: B Mining Design Discussionmentioning

confidence: 99%

Cross-Dataset Design Discussion Mining

Mahadi

Tongay

Ernst

2020

2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)

View full text Add to dashboard Cite

show abstract

“…A true meta-analysis [61,40] of the related work is not feasible in the area of design mining. Conventional meta-analysis is applied on primary studies that conduct experiments in order to support inference to a population, which is not the study logic of the studies considered here.…”

Section: Discussionmentioning

confidence: 99%

Conclusion stability for natural language based mining of design discussions

2021

View full text Add to dashboard Cite

Developer discussions range from in-person hallway chats to comment chains on bug reports. Being able to identify discussions that touch on software design would be helpful in documentation and refactoring software. Design mining is the application of machine learning techniques to correctly label a given discussion artifact, such as a pull request, as pertaining (or not) to design. In this work we demonstrate a simple example of how design mining works. We first replicate an existing state-of-the-art design mining study to show how conclusion stability is poor on different artifact types and different projects. Then we introduce two techniques-augmentation and context specificity-that greatly improve the conclusion stability and cross-project relevance of design mining. Our new approach achieves AUC-ROC of 0.88 on within dataset classification and 0.84 on the cross-dataset classification task. iv Contents Supervisory Committee

show abstract

“…The advantage to the qualitative approach is that it can use more nuance in labeling design discussions at a more specific level; the tradeoff of course is such labeling is labour-intensive. Summary A true meta-analysis [28,31] of the related work is not feasible in the area of design mining. Conventional meta-analysis is applied on primary studies that conduct experiments in order to support inference to a population, which is not the study logic of the studies considered here.…”

Section: Mining Design Discussionmentioning

confidence: 99%

Conclusion Stability for Natural Language Based Mining of Design Discussions

Mahadi¹,

Ernst²,

Tongay³

2021

Preprint

View full text Add to dashboard Cite

Developer discussions range from in-person hallway chats to comment chains on bug reports. Being able to identify discussions that touch on software design would be helpful in documentation and refactoring software. Design mining is the application of machine learning techniques to correctly label a given discussion artifact, such as a pull request, as pertaining (or not) to design. In this paper we demonstrate a simple example of how design mining works. We then show how conclusion stability is poor on different artifact types and different projects. We show two techniques-augmentation and context specificity-that greatly improve the conclusion stability and cross-project relevance of design mining. Our new approach achieves AUC of 0.88 on within dataset classification and 0.80 on the cross-dataset classification task.

show abstract

Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment

Cited by 33 publications

References 37 publications

Cross-Dataset Design Discussion Mining

Cross-Dataset Design Discussion Mining

Conclusion stability for natural language based mining of design discussions

Conclusion Stability for Natural Language Based Mining of Design Discussions

Contact Info

Product

Resources

About