KDD meets Big Data

Processes and practices used in data science projects have been reshaping especially over the last decade. These are different from their software engineering counterparts. However, to a large extent, data science relies on software, and, once taken to use, the results of a data science project are often embedded in software context. Hence, seeking synergy between software engineering and data science might open promising avenues. However, while there are various studies on data science workflows and data science project teams, there have been no attempts to combine these two very interlinked aspects. Furthermore, existing studies usually focus on practices within one company. Our study will fill these gaps with a multi-company case study, concentrating both on the roles found in data science project teams as well as the process. In this paper, we have studied a number of practicing data scientists to understand a typical process flow for a data science project. In addition, we studied the involved roles and the teamwork that would take place in the data context. Our analysis revealed three main elements of data science projects: Experimentation, Development Approach, and Multidisciplinary team(work). These key concepts are further broken down to 13 different sub-themes in total. The found themes pinpoint critical elements and challenges found in data science projects, which are still often done in an ad-hoc fashion. Finally, we compare the results with modern software development to analyse how good a match there is.

show abstract

“…There are also extensions (e.g. [2,7]) on these methodologies that aim at tackling some of the problems the practitioners have identified.…”

Section: Background and Related Workmentioning

confidence: 99%

Demystifying Data Science Projects: A Look on the People and Process of Data Science Today

Aho

Sievi-Korte

Kilamo

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…The Data Science Edge (DSE) methodology is introduced along two articles by Grady et al [64,76]. It is a enhanced process model to accommodate big data technologies and data science activities.…”

Section: Data Science Edgementioning

confidence: 99%

“…Nevertheless, one of the main shortcomings of CRISP-DM is that it does not explain how teams should organize to carry out the defined processes and does not address any of the above mentioned team management issues. In this sense, in words of [64], CRISP-DM needs a better integration with management processes, demands to align with software and agile development methodologies, and instead of simple checklists, it also needs method guidance for individual activities within stages.…”

Section: Crisp-dmmentioning

confidence: 99%

Data Science Methodologies: Current Challenges and Future Approaches

2021

View full text Add to dashboard Cite

Data science has employed great research efforts in developing advanced analytics, improving data models and cultivating new algorithms. However, not many authors have come across the organizational and socio-technical challenges that arise when executing a data science project: lack of vision and clear objectives, a biased emphasis on technical issues, a low level of maturity for ad-hoc projects and the ambiguity of roles in data science are among these challenges. Few methodologies have been proposed on the literature that tackle these type of challenges, some of them date back to the mid-1990, and consequently they are not updated to the current paradigm and the latest developments in big data and machine learning technologies. In addition, fewer methodologies offer a complete guideline across team, project and data & information management. In this article we would like to explore the necessity of developing a more holistic approach for carrying out data science projects. We first review methodologies that have been presented on the literature to work on data science projects and classify them according to the their focus: project, team, data and information management. Finally, we propose a conceptual framework containing general characteristics that a methodology for managing data science projects with a holistic point of view should have. This framework can be used by other researchers as a roadmap for the design of new data science methodologies or the updating of existing ones.

show abstract

“…In particular, there have been various approaches in the fields of Semantic Web and Linked Open Data for DM, although their full potential is still to be unlocked [ 7 ]. Traditional DM processes still face major challenges in terms of massive data [ 40 ]. In addition, the application of data mining still faces serious challenges, one of which is reproducing already known knowledge.…”

Section: Overview Of Semantic Data Mining Approachesmentioning

confidence: 99%

Semantic Data Mining in Ubiquitous Sensing: A Survey

Nalepa

Bobek

Kutt

et al. 2021

Sensors

View full text Add to dashboard Cite

Mining ubiquitous sensing data is important but also challenging, due to many factors, such as heterogeneous large-scale data that is often at various levels of abstraction. This also relates particularly to the important aspects of the explainability and interpretability of the applied models and their results, and thus ultimately to the outcome of the data mining process. With this, in general, the inclusion of domain knowledge leading towards semantic data mining approaches is an emerging and important research direction. This article aims to survey relevant works in these areas, focusing on semantic data mining approaches and methods, but also on selected applications of ubiquitous sensing in some of the most prominent current application areas. Here, we consider in particular: (1) environmental sensing; (2) ubiquitous sensing in industrial applications of artificial intelligence; and (3) social sensing relating to human interactions and the respective individual and collective behaviors. We discuss these in detail and conclude with a summary of this emerging field of research. In addition, we provide an outlook on future directions for semantic data mining in ubiquitous sensing contexts.

show abstract

KDD meets Big Data

Cited by 18 publications

References 8 publications

Demystifying Data Science Projects: A Look on the People and Process of Data Science Today

Demystifying Data Science Projects: A Look on the People and Process of Data Science Today

Data Science Methodologies: Current Challenges and Future Approaches

Semantic Data Mining in Ubiquitous Sensing: A Survey

Contact Info

Product

Resources

About