Automating data science

Abstract

Given the complexity of data science projects and related demand for human expertise, automation has the potential to transform the data science process.

Um Estudo sobre Ensino de Engenharia de Dados nas Universidades Brasileiras: Estado Atual e Perspectivas de Mercado

“…A Engenharia de Dados (ED) está definida nesse artigo como o processo de adquirir, organizar e preparar dados para serem consumidos em análises exploratórias, como entrada de algoritmos ou outros contextos [5]. Em um nível mais abstrato, a ED resolve problemas relacionados à organização e à qualidade dos dados, bem como à extração de características de entidades do mundo real representadas em meio digital [11].…”

“…Em particular, é sabido que 80% do trabalho envolvido nesse tipo de atividade está associado a tarefas como integrar, preparar, transformar e manipular, em grande e pequena escala, os dados a serem utilizados. Neste artigo, empregamos o termo de Engenharia de Dados (ED) para nos referirmos a essas atividades [5]. Assim, nosso ponto de vista nessa pesquisa é que para atender adequadamente às demandas da sociedade e do mercado, é muito importante que os profissionais de computação formados pelas universidades dominem os principais métodos e técnicas de ED.…”

Tracing and Visualizing Human-ML/AI Collaborative Processes through Artifacts of Data Work

“…The skills required to conduct data work remain sufficiently complex, making it inaccessible to many experts with the relevant domain context but needing more technical acumen [44]. Recent innovations have developed techniques that automatically carry out data work, for example, model selection or certain data preparation steps, thereby lowering barriers of use to non-technical experts [18]. Initially, this so-called automated machine learning technology (AutoML) focused primarily on the analysis phase.…”

Reproducibility in 2023 – An End-to-End Template for Analysis and Manuscript Writing

“…The goal of Data Science is to generate new insights from data and encompasses a wide spectrum of data handling procedures, from the initial data collection further on to data processing and preparation, as well as data exploration and statistical modeling, finally to visualization, interpretation and publishing the results [1]. The breadth and complexity of data science scenarios requires comprehensive expertise and experience in a wide range of related topics and domains, e.g.…”

“…mathematics, statistics, computer science, software development, and domain knowledge of the problem to solve, besides others. Simultaneously, there is increasing interest in automating parts of the data science process, if not the entire process [1]. Additionally, researchers from a number of disciplines in the computer sciences call for reproducibility or reproducible research as a minimum achievable standard for assessing the value of scientific claims, especially when full independent replication of a study is not possible, for example, due to lack of resources such as time and money [2][3][4][5].…”