Massively parallel data analysis with PACTs on Nephele

Alexandrov, A.; Heimel, Max; Markl, Volker; Battré, Dominic; Hueske, Fabian; Nijkamp, Erik; Ewen, Stephan; Kao, Odej; Warneke, Daniel

doi:10.14778/1920841.1921056

Cited by 42 publications

(20 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The disruptions discussed in this study present exciting opportunities for the business intelligence community. We started to address hard problems, the solution of which can greatly impact the future course of data platforms and tools such as identifying text mining services for niche data [7], or investigating data processing infrastructures for large scale data such as [1].…”

Section: Resultsmentioning

confidence: 99%

Pricing Approaches for Data Markets

Muschalle

Stahl

Löser

et al. 2013

Lecture Notes in Business Information Processing

View full text Add to dashboard Cite

Abstract. Currently, multiple data vendors utilize the cloud-computing paradigm for trading raw data, associated analytical services, and analytic results as a commodity good. We observe that these vendors often move the functionality of data warehouses to cloud-based platforms. On such platforms, vendors provide services for integrating and analyzing data from public and commercial data sources. We present insights from interviews with seven established vendors about their key challenges with regard to pricing strategies in different market situations and derive associated research problems for the business intelligence community.

show abstract

Section: Resultsmentioning

confidence: 99%

Pricing Approaches for Data Markets

Muschalle

Stahl

Löser

et al. 2013

Lecture Notes in Business Information Processing

View full text Add to dashboard Cite

show abstract

“…The plan with the lowest cost is selected as an optimized query plan. A PACT program is executed on a three-tier architecture [38] composed of: a PACT compiler, engine Nephele [39], and a distributed file system.…”

Section: Parallelism In Traditional Data Flowmentioning

confidence: 99%

From conceptual design to performance optimization of ETL workflows: current state of research and open problems

Ali

Wrembel

2017

The VLDB Journal

View full text Add to dashboard Cite

In this paper, we discuss the state of the art and current trends in designing and optimizing ETL workflows. We explain the existing techniques for: (1) constructing a conceptual and a logical model of an ETL workflow, (2) its corresponding physical implementation, and (3) its optimization, illustrated by examples. The discussed techniques are analyzed w.r.t. their advantages, disadvantages, and challenges in the context of metrics such as autonomous behavior, support for quality metrics, and support for ETL activities as user-defined functions. We draw conclusions on still open research and technological issues in the field of ETL. Finally, we propose a theoretical ETL framework for ETL optimization.

show abstract

“…These limitations led to the development of workflow engines targeted at largescale data analytics [7,4,23]. These engines allow for more flexible compositions of user-defined code than map-reduce while keeping the ability to parallelize tasks.…”

Section: Workflow Enginesmentioning

confidence: 99%

“…In this paper we extend these ideas to support both relational operators and UDFs. A similar approach is PACT [4] where contracts are bound to tasks as pre-and post-conditions. These contracts enable rewrites of the workflow.…”

Section: Workflow Enginesmentioning

confidence: 99%

See 1 more Smart Citation

A study of partitioning and parallel UDF execution with the SAP HANA database

Große

May

Lehner

2014

Proceedings of the 26th International Conference on Scientific and Statistical Database Management

View full text Add to dashboard Cite

Large-scale data analysis relies on custom code both for preparing the data for analysis as well as for the core analysis algorithms. The map-reduce framework offers a simple model to parallelize custom code, but it does not integrate well with relational databases. Likewise, the literature on optimizing queries in relational databases has largely ignored user-defined functions (UDFs). In this paper, we discuss annotations for user-defined functions that facilitate optimizations that both consider relational operators and UDFs. We believe this to be the superior approach compared to just linking map-reduce evaluation to a relational database because it enables a broader range of optimizations. In this paper we focus on optimizations that enable the parallel execution of relational operators and UDFs for a number of typical patterns. A study on real-world data investigates the opportunities for parallelization of complex data flows containing both relational operators and UDFs.

show abstract

Massively parallel data analysis with PACTs on Nephele

Cited by 42 publications

References 5 publications

Pricing Approaches for Data Markets

Pricing Approaches for Data Markets

From conceptual design to performance optimization of ETL workflows: current state of research and open problems

A study of partitioning and parallel UDF execution with the SAP HANA database

Contact Info

Product

Resources

About