Escaping the Time Pit: Pitfalls and Guidelines for Using Time-Based Git Data

Flint, Samuel W.; Chauhan, Jigyasa; Dyer, Robert

doi:10.1109/msr52588.2021.00022

Cited by 8 publications

(4 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that this study is an extension of our prior paper (Flint et al, 2021a). Compared to the previous paper, this work adds the following additional contributions:…”

Section: Introductionmentioning

confidence: 91%

Pitfalls and guidelines for using time-based Git data

Flint

Chauhan²,

Dyer³

2022

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Many software engineering research papers rely on time-based data (e.g., commit timestamps, issue report creation/update/close dates, release dates). Like most real-world data however, time-based data is often dirty. To date, there are no studies that quantify how frequently such data is used by the software engineering research community, or investigate sources of and quantify how often such data is dirty. Depending on the research task and method used, including such dirty data could affect the research results. This paper presents an extended survey of papers that utilize time-based data, published in the Mining Software Repositories (MSR) conference series. Out of the 754 technical track and data papers published in MSR 2004-2021, we saw at least 290 (38%) papers utilized time-based data. We also observed that most time-based data used in research papers comes in the form of Git commits, often from GitHub. Based on those results, we then used the Boa and Software Heritage infrastructures to help identify and quantify several sources of dirty Git timestamp data. Finally we provide guidelines/best practices for researchers utilizing time-based data from Git repositories. Keywords literature review • time data • mining software repositoriesThis paper is a revised and extended version of Flint et al. (2021a).

show abstract

“…Note that this study is an extension of our prior paper (Flint et al, 2021a). Compared to the previous paper, this work adds the following additional contributions:…”

Section: Introductionmentioning

confidence: 91%

Pitfalls and guidelines for using time-based Git data

Flint

Chauhan²,

Dyer³

2022

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

show abstract

“…There is some chance that the author timestamps used to chronologically order the data contain errors resulting from, e.g., misconfigured clocks or problems when converting old repositories into GitHub. However, such issues tend to affect a very small proportion of commits [34], being unlikely to negatively affect our analyses. An additional threat is the use of Commit Guru to collect labelled software changes.…”

Section: Threats To Validitymentioning

confidence: 98%

“…All software changes were chronologically ordered based on the author timestamp, which is provided by Git as the timestamp when the commit was first created by the author. Author timestamp is recommended over commiter timestamp for studies involving time-based Git data [34]. This timestamp is used in our paper as the moment in time when a JIT-SDP model is required to provide a prediction for this software change.…”

Section: Datasetsmentioning

confidence: 99%

Cross-Project Online Just-In-Time Software Defect Prediction

Tabassum

Minku

Feng³

2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Cross-Project (CP) Just-In-Time Software Defect Prediction (JIT-SDP) makes use of CP data to overcome the lack of data necessary to train well performing JIT-SDP classifiers at the beginning of software projects. However, such approaches have never been investigated in realistic online learning scenarios, where Within-Project (WP) software changes naturally arrive over time and can be used to automatically update the classifiers. We provide the first investigation of when and to what extent CP data are useful for JIT-SDP in such realistic scenarios. For that, we propose three different online CP JIT-SDP approaches that can be updated with incoming CP and WP training examples over time. We also collect data on 9 proprietary software projects and use 10 open source software projects to analyse these approaches. We find that training classifiers with incoming CP+WP data can lead to absolute improvements in G-mean of up to 53.89% and up to 35.02% at the initial stage of the projects compared to classifiers using WP-only and CP-only data, respectively. Using CP+WP data was also shown to be beneficial after a large number of WP data were received. Using CP data to supplement WP data helped the classifiers to reduce or prevent large drops in predictive performance that may occur over time, leading to absolute G-Mean improvements of up to 37.35% and 48.16% compared to WP-only and CP-only data during such periods, respectively. During periods of stable predictive performance, absolute improvements were of up to 29.03% and up to 41.25% compared to WP-only and CP-only classifiers, respectively. Our results highlight the importance of using both CP and WP data together in realistic online JIT-SDP scenarios.

show abstract

“…Falessi et al (2020) reports on the importance of preserving the order of data between the training and testing set. Afterward, the same issue was deeply discussed in Flint et al (2021) Thus, results are unrealistic if the underlying evaluation does not preserve the order of data. Falessi et al (2022) show that dormant defects impact classifiers' accuracy and hence its evaluation.…”

Section: Evaluationsmentioning

confidence: 99%

On effort-aware metrics for defect prediction

2022

View full text Add to dashboard Cite

Context Advances in defect prediction models, aka classifiers, have been validated via accuracy metrics. Effort-aware metrics (EAMs) relate to benefits provided by a classifier in accurately ranking defective entities such as classes or methods. PofB is an EAM that relates to a user that follows a ranking of the probability that an entity is defective, provided by the classifier. Despite the importance of EAMs, there is no study investigating EAMs trends and validity. Aim The aim of this paper is twofold: 1) we reveal issues in EAMs usage, and 2) we propose and evaluate a normalization of PofBs (aka NPofBs), which is based on ranking defective entities by predicted defect density. Method We perform a systematic mapping study featuring 152 primary studies in major journals and an empirical study featuring 10 EAMs, 10 classifiers, two industrial, and 12 open-source projects. Results Our systematic mapping study reveals that most studies using EAMs use only a single EAM (e.g., PofB20) and that some studies mismatched EAMs names. The main result of our empirical study is that NPofBs are statistically and by orders of magnitude higher than PofBs. Conclusions In conclusion, the proposed normalization of PofBs: (i) increases the realism of results as it relates to a better use of classifiers, and (ii) promotes the practical adoption of prediction models in industry as it shows higher benefits. Finally, we provide a tool to compute EAMs to support researchers in avoiding past issues in using EAMs.

show abstract

Escaping the Time Pit: Pitfalls and Guidelines for Using Time-Based Git Data

Cited by 8 publications

References 33 publications

Pitfalls and guidelines for using time-based Git data

Pitfalls and guidelines for using time-based Git data

Cross-Project Online Just-In-Time Software Defect Prediction

On effort-aware metrics for defect prediction

Contact Info

Product

Resources

About