Achieving Lean Data Science Agility Via Data Driven Scrum

Abstract

This paper first reviews the concept of a lean data science project and defines four principles that a team should follow to achieve lean data science. It then describes a new team process framework called Data Driven Scrum (DDS) which enables lean data science project agility and addresses the key challenges that have been identified when using Scrum in a data science context. As compared to Scrum, DDS increases the focus in observing and analyzing the output of each iteration (i.e., each experiment). DDS also defines capability-based iterations (as compared to Scrum's time-based sprints). The paper then reports on how to integrate DDS with data science workflow frameworks such as CRISP-DM as well as on a pilot study of an organization that adopted DDS.

Introduction to the Minitrack on Metaverse

“…However, a key challenge for successful data science project completion is not technical in nature, but rather, driven by the process used to execute data science projects [2]. For example, it has been noted that most data science projects do not leverage welldefined process methodologies [3] [4]. Thus, it is not surprising that poor coordination and collaboration are notable challenges that have led to data science project failure [3] [5].…”