Towards Applicable Just-In-Time Defect Prediction: A Practical Perspective on Effort Awareness

Abstract

Reviewing software changes is crucial, as it mitigates the introduction of defects and thereby saves time and reduces costs. Just-in-time (JIT) defect prediction has emerged as an approach to support the review process by predicting the likelihood of defects in new commits. Effort-aware evaluations were proposed to better manage developers' limited time and to analyze the applicability of JIT defect prediction approaches. However, current effort-aware approaches neglect the time-dependent nature of software engineering when evaluating the performance and rank commits to find most defective commits with limited effort. This reveals a gap between the evaluation of models in research and their application in practice, where a timely decision for every single commit is needed. Thus, evaluating models based on effort-aware evaluations does not reflect the actual performance in real-world scenarios and may misrepresent their applicability. In order to evaluate if state-of-the-art JIT defect prediction approaches are applicable, we address these limitations and suggest a more practical evaluation strategy that shifts the focus to conserving effort while ensuring that defective commits are still examined. We further elaborate on the possibility to tune the model prediction and allow for a more reliable application of JIT defect prediction models. Conducting a case study of six open-source projects, we find that the additional performance metrics allow new insights into the potential of JIT defect prediction and state-of-the-art defect prediction can be applicable under the right assumptions. Taking this new perspective, while still reviewing 95% of defective commits, on average, 46% of the non-defective commits are correctly identified by JIT defect prediction models and can be skipped. Thus, while maintaining high recall levels, 20% of the total avoidable effort can be saved employing JIT defect prediction models.