Tasks and Visualizations Used for Data Profiling: A Survey and Interview Study

Abstract

The use of good-quality data to inform decision making is entirely dependent on robust processes to ensure it is fit for purpose. Such processes vary between organisations, and between those tasked with designing and following them. In this paper we report on a survey of 53 data analysts from many industry sectors, 24 of whom also participated in in-depth interviews, about computational and visual methods for characterizing data and investigating data quality. The paper makes contributions in two key areas. The first is to data science fundamentals, because our lists of data profiling tasks and visualization techniques are more comprehensive than those published elsewhere. The second concerns the application question "what does good profiling look like to those who routinely perform it?", which we answer by highlighting the diversity of profiling tasks, unusual practice and exemplars of visualization, and recommendations about formalizing processes and creating rulebooks.

A Data-Centric AI Paradigm for Socio-Industrial and Global Challenges

“…On the other hand, a lot of efforts have been recently devoted to increasing data quality with the help of automated pipelines, data engineering frameworks, and prototypes. The implementation of technical solutions like data lakes [105], low-latency data infrastructure [106], feature stores [107], data warehouses [108,109], data branching [110], AutoML for data management [111], data strew-ships [112], data fusion techniques [113], data taxonomies [114], data-quality enhancement pipelines [115], data mesh and fabric [116], addressing imbalances in data [117], smart bots for data quality enhancement [118], data ontologies [119], data quality evaluation metrics [120], synthetic data generation tools [120], data profiling [121], reference stores for data quality [122], and data validation pipelines [123,124], to name a few, are vastly contributing in the feasibility and affordability of DC-AI-based solutions. In the future, more developments are expected in data quality enhancement, leading to the realization of DC-AI across many enterprises.…”