MIDST: an enhanced development environment that improves the maintainability of a data science analysis

Abstract

With the increasing ability to generate actionable insight from data, the field of data science has seen significant growth. As more teams develop data science solutions, the analytical code they develop will need to be enhanced in the future, by an existing or a new team member. Thus, the importance of being able to easily maintain and enhance the code required for an analysis will increase. However, to date, there has been minimal research on the maintainability of an analysis done by a data science team. To help address this gap, data science maintainability was explored by (1) creating a data science maintainability model, (2) creating a new tool, called MIDST (Modular Interactive Data Science Tool), that aims to improve data science maintainability, and then (3) conducting a mixed method experiment to evaluate MIDST. The new tool aims to improve the ability of a team member to update and rerun an existing data science analysis by providing a visual data flow view of the analysis within an integrated code and computational environment. Via an analysis of the quantitative and qualitative survey results, the experiment found that MIDST does help improve the maintainability of an analysis. Thus, this research demonstrates the importance of enhanced tools tohelp improve the maintainability of data science projects.

Current approaches for executing big data science projects—a systematic literature review

“…Five papers explored how different tools, both custom and commercial, could be used to support various aspects of the execution of the data science projects. The tools explored focused on communication and collaboration ( Marin, 2019 ; Wang et al, 2019 ), Continuous Integration/Continuous Development ( Chen et al, 2020 ), the maintainability of a data science project ( Saltz et al, 2020 ) and a tool to improve the coordination of the data science team ( Crowston et al, 2021 ).…”

Tools for monitoring software quality in information systems development and maintenance: five key challenges and a design proposal

“…However, continuous software quality assessment based on such standards can be a cumbersome and costly task for organizations with large software portfolios. While project documentation can be helpful, it alone is rarely sufficient for developer teams to maintain a software system [27]. As software systems grow in number and complexity, manual software quality assessment becomes progressively more demanding.…”