Towards a Cost Model to Optimize User-Defined Functions in an ETL Workflow Based on User-Defined Performance Metrics

Ali, Syed Muhammad; Wrembel, Robert

doi:10.1007/978-3-030-28730-6_27

Cited by 10 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[13] proposes to include formalized annotations when deploying an UDF, so that the system can learn basic behavioral aspects of the UDF from the annotations. Finally, we proposed an architecture and method for executing UDFs in a parallel environment [9]. The optimization of a DPP in the architecture presented in Figure 3 should draw upon these solutions.…”

Section: Challenge 3: Workflow Optimizationmentioning

confidence: 99%

Data Engineering for Data Science: Two Sides of the Same Coin

Romero

Wrembel

2020

Big Data Analytics and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

A de facto technological standard of data science is based on notebooks (e.g., Jupyter), which provide an integrated environment to execute data workflows in different languages. However, from a data engineering point of view, this approach is typically inefficient and unsafe, as most of the data science languages process data locally, i.e., in workstations with limited memory, and store data in files. Thus, this approach neglects the benefits brought by over 40 years of R&D in the area of data engineering, i.e., advanced database technologies and data management techniques. In this paper, we advocate for a standardized data engineering approach for data science and we present a layered architecture for a data processing pipeline (DPP). This architecture provides a comprehensive conceptual view of DPPs, which next enables the semi-automation of the logical and physical designs of such DPPs.

show abstract

Section: Challenge 3: Workflow Optimizationmentioning

confidence: 99%

Data Engineering for Data Science: Two Sides of the Same Coin

Romero

Wrembel

2020

Big Data Analytics and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our Proposed Approach: We present FunMap, an interpreter of RML+FnO, that converts a data integration system defined using RML+FnO into an equivalent one where RML mappings are function-free. FunMap resembles existing mapping translation proposals (e.g., [2,5,17]) and empowers a KG creation process with optimization techniques to reduce execution time. Transformations of data sources include the projection of the attributes used in the RML+FnO mappings.…”

Section: Introductionmentioning

confidence: 99%

FunMap: Efficient Execution of Functional Mappings for Knowledge Graph Creation

Jozashoori,

Chaves-Fraga,

Iglesias

et al. 2020

Preprint

View full text Add to dashboard Cite

Data has exponentially grown in the last years, and knowledge graphs constitute powerful formalisms to integrate a myriad of existing data sources. Transformation functions -specified with functionbased mapping languages like FunUL and RML+FnO -can be applied to overcome interoperability issues across heterogeneous data sources. However, the absence of engines to efficiently execute these mapping languages hinders their global adoption. We propose FunMap, an interpreter of function-based mapping languages; it relies on a set of lossless rewriting rules to push down and materialize the execution of functions in initial steps of knowledge graph creation. Although applicable to any function-based mapping language that supports joins between mapping rules, FunMap feasibility is shown on RML+FnO. FunMap reduces data redundancy, e.g., duplicates and unused attributes, and converts RML+FnO mappings into a set of equivalent rules executable on RMLcompliant engines. We evaluate FunMap performance over real-world testbeds from the biomedical domain. The results indicate that FunMap reduces the execution time of RML-compliant engines by up to a factor of 18, furnishing, thus, a scalable solution for knowledge graph creation.

show abstract

Framework to Optimize Data Processing Pipelines Using Performance Metrics

Ali

Wrembel

2020

Big Data Analytics and Knowledge Discovery

View full text Add to dashboard Cite

Towards a Cost Model to Optimize User-Defined Functions in an ETL Workflow Based on User-Defined Performance Metrics

Cited by 10 publications

References 23 publications

Data Engineering for Data Science: Two Sides of the Same Coin

Data Engineering for Data Science: Two Sides of the Same Coin

FunMap: Efficient Execution of Functional Mappings for Knowledge Graph Creation

Framework to Optimize Data Processing Pipelines Using Performance Metrics

Contact Info

Product

Resources

About