The evolution of CRISP-DM for Data Science: Methods, Processes and Frameworks

Abstract

The expansion of Data Science projects in organizations has been led by three factors: the growth in the amount of data generated, the evolution in storage capacity, and the increase in computational capabilities. However, most of these projects fail to deliver the expected value: 82% of the teams do not use any process model. Despite the popularity of Agile Methods, their adoption in Data Science projects is still scarce. Most of the existing research focuses on algorithms. There is a lack of studies on agility in Data Science. This Systematic Literature Review (SLR) was performed to identify and evaluate 16 studies that can answer how to adapt and apply CRISP-DM using different approaches — methods, frameworks, or process models. In addition, it shows how CRISP-DM has evolved over the last few decades, with derivations emerging from rigid processes to agile methods. This research then analyzes the 16 tailored models and examines the similarities and differences between CRISP-DM derivatives. As a result, it summarizes the CRISP-DM adaptation patters identified, such as phase addition, phase modification, features and tools addition, and integration with other approaches. Consequently, this SLR showcases how CRISP-DM is a robust, flexible, and highly adaptable model that can be extended to different business domains. Finally, it proposes a theoretical guide to modify and customize CRISP-DM for Data Science projects.