Reproducibility in the technical debt domain

Abstract

Context It is crucial to understand how reproducible the measurement results in the scientific publications are, as reproducibility is one of the cornerstones of engineering. Objective The goal of this study is to investigate the scientific publications presented at the premier technical debt conferences by understanding how reproducible the reported findings are. Method We conducted a systematic literature review of 135 unique papers published at the “International Workshop on Managing Technical Debt” and the “International Conference on Managing Technical Debt”, the premier scientific conference series on technical debt. Results Only 44 of the investigated 135 papers presented numerical evidence and only 5 papers listed the tools, the availability of the tools, and the version of the tools used. For the rest of the papers additional information would have been needed for the potential reproducibility. One of the published papers even referred to a pornographic site as a source of a toolset for empirical research.