An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems

Abstract

Machine Learning (ML), including Deep Learning (DL), systems, i.e., those with ML capabilities, are pervasive in today's data-driven society. Such systems are complex; they are comprised of ML models and many subsystems that support learning processes. As with other complex systems, ML systems are prone to classic technical debt issues, especially when such systems are long-lived, but they also exhibit debt specific to these systems. Unfortunately, there is a gap of knowledge in how ML systems actually evolve and are maintained. In this paper, we fill this gap by studying refactorings, i.e., source-to-source semanticspreserving program transformations, performed in real-world, open-source software, and the technical debt issues they alleviate. We analyzed 26 projects, consisting of 4.2 MLOC, along with 327 manually examined code patches. The results indicate that developers refactor these systems for a variety of reasons, both specific and tangential to ML, some refactorings correspond to established technical debt categories, while others do not, and code duplication is a major crosscutting theme that particularly involved ML configuration and model code, which was also the most refactored. We also introduce 14 and 7 new ML-specific refactorings and technical debt categories, respectively, and put forth several recommendations, best practices, and anti-patterns. The results can potentially assist practitioners, tool developers, and educators in facilitating long-term ML system usefulness.Index Terms-empirical studies, refactoring, machine learning systems, technical debt, software repository mining I. In t r o d u c t io nIn the big data era, Machine Learning (ML), including Deep Learning (DL), systems are pervasive in modern society. Central to these systems are dynamic ML models, whose behavior is ultimately defined by their input data. However, such systems do not only consist of ML models; instead, ML systems typically encompass complex subsystems that support ML processes [1]. ML systems-like other long-lived, complex systems-are prone to classic technical debt [2 ] issues; yet, they also exhibit debt specific to such systems [3]. While work exist on applying software engineering (SE) rigor to ML systems [4]- [12], there is generally a gap of knowledge in how ML systems actually evolve and are maintained. As ML systems become more difficult and expensive to maintain [1 ], understanding the kinds of modifications developers are required to make to such systems-our overarching research question-is of the utmost importance.To fill this gap, we performed an empirical study on common refactorings, i.e., source-to-source semantics preserving program transformations-a widely accepted mechanism for effectively reducing technical debt [13]-[16]-in real-world, open-source ML systems. We set out to discover (i) the kinds of refactorings-both specific and tangential to ML-performed, (ii) whether particular refactorings occurred more often in model code vs. other supporting subsystems, (iii) the types of technical debt ...

PYEVOLVE: Automating Frequent Code Changes in Python ML Systems

“…Despite Python's and ML's meteoric rise [58]- [61], support for automated code evolution is still in its infancy. To advance the science and tooling for automating code changes in Python, we built PYEVOLVE, which infers transformation rules and then applies them.…”

Demystifying Dependency Bugs in Deep Learning Stack

“…Many studies have empirically investigated different aspects in developing, deploying and maintaining DL systems, e.g., software engineering for DL systems [2,14,35], challenges in developing DL systems [1,62] and deploying DL systems [8], pain-points in using cloud services of computer vision [10], accuracy variance in training DL systems [43], performance variance in deploying DL models to different mobile devices and web browsers [15,33], discussion topics about DL frameworks [17], API evolution in DL frameworks [66], technical debt in DL frameworks [31,32] and DL systems [38,45,50], and dependency supply chain of DL libraries [16,49].…”

Machine Learning Systems are Bloated and Vulnerable

“…While technical debt can have solid technical reasons, this debt-like fiscal debt-needs to be serviced by, e.g., refactoring code, and removing unnecessary code (dead code) and unnecessary dependencies. The technical debt of machine learning (ML) systems has recently come under scrutiny [42,55,61]. Sculley et al [55] identify eight main sources of technical debt in ML systems including common code issues such as dead code that is never used and duplicate code.…”