RHEEM: enabling cross-platform data processing

Abstract

Solving business problems increasingly requires going beyond the limits of a single data processing platform (platform for short), such as Hadoop or a DBMS. As a result, organizations typically perform tedious and costly tasks to juggle their code and data across different platforms. Addressing this pain and achieving automatic cross-platform data processing is quite challenging: finding the most efficient platform for a given task requires quite good expertise for all the available platforms. We present Rheem, a general-purpose cross-platform data processing system that decouples applications from the underlying platforms. It not only determines the best platform to run an incoming task, but also splits the task into subtasks and assigns each subtask to a specific platform to minimize the overall cost (e.g., runtime or monetary cost). It features (i) a robust interface to easily compose data analytic tasks; (ii) a novel cost-based optimizer able to find the most efficient platform in almost all cases; and (iii) an executor to efficiently orchestrate tasks over different platforms. As a result, it allows users to focus on the business logic of their applications rather than on the mechanics of how to compose and execute them. Using different real-world applications with Rheem, we demonstrate how cross-platform data processing can accelerate performance by more than one order of magnitude compared to single-platform data processing.

RHEEMix in the data jungle: a cost-based optimizer for cross-platform systems

“…Contributions We delve into the cross-platform optimizer of Rheem [3,4,47], our open-source cross-platform system [55]. While we present the system design of Rheem in [4] and briefly discuss the data movement aspect in [43], in this paper, we describe in detail how our cost-based cross-platform optimizer tackles all of the above research challenges. 2 The idea is to split a single task into multiple atomic operators and to find the most suitable platform for each operator (or set of operators) so that its total cost is minimized.…”

“…6). (4) We explain how we exploit our optimization pipeline for performing progressive optimization to deal with poor cardinality estimates (Sect. 7).…”

“…Before delving into the details, let us briefly outline Rheem, our open-source cross-platform system, so as to establish our optimizer's context. Rheem decouples applications from platforms with the goal of enabling cross-platform data processing [4,5]. Although decoupling data processing was the driving motive when designing Rheem, we also adopted a three-layer optimization approach envisioned in [5].…”

“…In turn, the executor orchestrates and monitors the execution of the generated execution plan on the selected platforms. For more details about Rheem's data model and architecture, we would like to refer the interested reader to [4,55]. In this paper, we focus on the cross-platform optimizer.…”

“…Cross-platform data processing As a result, today's data analytics often need to perform cross-platform data processing, i.e., running their tasks on more than one platform. Research and industry communities have identified this need [5,62] and have proposed systems to support different aspects of cross-platform data processing [4,7,13,25,27,30]. We have identified four situations in which an application requires support for cross-platform data processing [4,40]:…”