The Delta Maintainability Model: Measuring Maintainability of Fine-Grained Code Changes

Abstract

Existing maintainability models are used to identify technical debt of software systems. Targeting entire codebases, such models lack the ability to determine shortcomings of smaller, fine-grained changes. This paper proposes a new maintainability model-the Delta Maintainability Model (DMM)-to measure fine-grained code changes, such as commits, by adapting and extending the SIG Maintainability Model. DMM categorizes changed lines of code into low and high risk, and then uses the proportion of low risk change to calculate a delta score. The goal of the DMM is twofold: first, producing meaningful and actionable scores; second, compare and rank the maintainability of fine-grained modifications. We report on an initial study of the model, with the goal of understanding if the adapted measurements from the SIG Maintainability Model suit the fine-grained scope of the DMM. In a manual inspection process for 100 commits, 67 cases matched the expert judgment. Furthermore, we report an exploratory empirical study on a data set of DMM scores on 3,017 issue-fixing commits of four open source and four closed source systems. Results show that the scores of DMM can be used to compare and rank commits, providing developers with a means to do root cause analysis on activities that impacted maintainability and, thus, address technical debt at a finer granularity.

CVEfixes: automated collection of vulnerabilities and their fixes from open-source software

“…We extract this information at the commit-, file-and method level, where each entry in the commits table is associated to one or more cves via the fixes table. In addition, the commits table stores meta-information about the commit, such as the author, time and date, commit message, if it is a merge commit (often to include pull requests in projects using GitHub Flow), the number of lines added or deleted, and Delta Maintainability Model metrics related to the change [36]. Extracting the Modified Files: Every entry in the commits table is linked via the commit hash to one or more file_changes that were included to fix the vulnerability (or vulnerabilities) associated with the commit.…”

Fixing vulnerabilities potentially hinders maintainability

“…BCH checks GitHub codebases against 10 maintainability guidelines (Visser 2016) that were empirically validated in previous work (Bijlsma et al 2012;Malavolta et al 2018;Cruz et al 2019;di Biase et al 2019). SIG has devised these guidelines after many years of experience: analyzing more than 15 million lines of code every week, SIG maintains the industry's largest benchmark, containing more than 10 billion lines of code across 200+ technologies; SIG is the only lab in the world certified by T ÜViT to issue ISO 25010 certificates 10 .…”

“…In 2019, Cruz et al (Cruz et al 2019) proposed a formula to calculate maintainability based on the BCH's guidelines and measured the impact of energy-oriented fixes on software maintainability. Recent work proposed a new maintainability model to measure fine-grained code changes by adapting/extending the BCH model (di Biase et al 2019). Our work uses the same base model (SIG-MM) but considers a broader set of guidelines.…”

“…There are other well-known standards and models that have been proposed to increase software security: Common Criteria (2009) which received negative criticism regarding the costs associated and poor technical evaluation; the OWASP Application Security Verification Standard (ASVS) (The OWASP Foundation 2009) which is focused only on web applications, and a model proposed by Xu et al (2013) for rating software security (arguably, it was one of the first steps taken by SIG to introduce security on their maintainability model). Nevertheless, our study uses BCH to provide an assessment of maintainability in software for the following reasons: BCH integrates a total of 10 different code metrics; and, code metrics were empirically validated in previous work (Bijlsma et al 2012;Malavolta et al 2018;Cruz et al 2019;di Biase et al 2019).…”

Reproducibility in the technical debt domain

“…"The Delta Maintainability Model: Measuring Maintainability of Fine-Grained Code Changes" ( [25]) proposes and uses a new maintainability model "Delta Maintainability Model" (DMM) for fine-grained measure-ments of code changes, without making the tool implemented and used for measurements available (or at least it is not referenced in the article). "Detecting Bad Smells with Machine Learning Algorithms: An Empirical Study" ( [19]) uses the tools "JDeodorant", "JSpirit", "PMD", "DECOR", "Organic".…”