Incorporating Data Context to Cost-Effectively Automate End-to-End Data Wrangling

Koehler, Martin; Abel, E. W.; Bogatu, Alex; Civili, Cristina; Mazilu, Lacramioara; Κωνσταντίνου, Νικόλαος; Fernandes, Alvaro A. A.; Keane, John; Libkin, Leonid; Paton, Norman W.

doi:10.1109/tbdata.2019.2907588

Cited by 9 publications

(2 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This research area is commonly called AutoML. It turned to be a hot research area in recent years (Bilalli et al, 2019;Giovanelli et al, 2021;Koehler et al, 2021;Quemy, 2019). (Kedziora et al, 2020) provides an excellent state of the art of this research area.…”

Section: Machine Learningmentioning

confidence: 99%

Advances on Data Management and Information Systems

Darmont

Novikov²,

Wrembel

et al. 2022

Inf Syst Front

View full text Add to dashboard Cite

The research and technological area of data management encompasses various concepts, techniques, algorithms and technologies, including data modeling, data integration and ingestion, transactional data management, query languages, query optimization, physical data storage, data structures, analytical techniques (including On-Line Analytical Processing -OLAP), as well as service creation and orchestration (Garcia-Molina et al., 2009). Data management technologies are core components of every information system, either centralized or distributed, deployed in an onpremise hardware architecture or in a cloud ecosystem. Data management technologies have been used in commercial, mature products for decades. They were originally developed for managing structured data (mainly expressed in the relational data model).Yet, the ubiquitous big data (Azzini et al., 2021) require development of new data management techniques, suitable for the variety of data formats (from structured, through semi-structured, to unstructured), overwhelming data volumes and velocity of big data generation. These new techniques draw upon the concepts applied to managing relational data.

show abstract

Section: Machine Learningmentioning

confidence: 99%

Advances on Data Management and Information Systems

Darmont

Novikov²,

Wrembel

et al. 2022

Inf Syst Front

View full text Add to dashboard Cite

show abstract

“…Ajax [53] brings a SQL-like language to data transformations. Early work on automating end-to-end data wrangling seems promising, but there is likely much more to do [98].…”

Section: Current Research Lines and Future Challengesmentioning

confidence: 99%

Towards Data Wrangling Automation through Dynamically-Selected Background Knowledge

Ochando¹

View full text Add to dashboard Cite

Data science is essential for the extraction of value from data. However, the most tedious part of the process, data wrangling, implies a range of mostly manual formatting, identification and cleansing manipulations. Data wrangling still resists automation partly because the problem strongly depends on domain information, which becomes a bottleneck for state-of-the-art systems as the diversity of domains, formats and structures of the data increases. In this thesis we focus on generating algorithms that take advantage of the domain knowledge for the automation of parts of the data wrangling process. We illustrate the way in which general program induction techniques, instead of domain-specific languages, can be applied flexibly to problems where knowledge is important, through the dynamic use of domain-specific knowledge. More generally, we argue that a combination of knowledge-based and dynamic learning approaches leads to successful solutions. We propose several strategies to automatically select or construct the appropriate background knowledge for several data wrangling scenarios. The key idea is based on choosing the best specialised background primitives according to the context of the particular problem to solve. We address two scenarios. In the first one, we handle personal data (names, dates, telephone numbers, etc.) that are presented in very different string formats and have to be transformed into a unified format. The problem is how to build a compositional transformation from a large set of primitives in the domain (e.g., handling months, years, days of the week, etc.). We develop a system (BK-ADAPT) that guides the search through the background knowledge by extracting several meta-features from the examples characterising the column domain. In the second scenario, we face the transformation of data matrices in generic programming languages such as R, using an input matrix and some cells of the output matrix as examples. We also develop a system guided by a treebased search (AUTOMAT[R]IX) that uses several constraints, prior primitive probabilities and textual hints to efficiently learn the transformations. With these systems, we show that the combination of inductive programming with the dynamic selection of the appropriate primitives from the background knowledge is able to improve the results of other state-of-the-art-and more specific-data wrangling approaches.

show abstract