From a stream of relational queries to distributed stream processing

Zou, Qiong; Wang, Huayong; Soulé, Robert; Hirzel, Martin; Andrade, Henrique; Gedik, Buğra; Wu, Kun-Lung

doi:10.14778/1920841.1921012

Cited by 20 publications

(13 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In traditional decision Support System (DSS) workloads characterized by TPC-H and TPC-DS, many queries are performed on multiple occurrences of the same table [12]. It is also very common to find such queries in spatial database systems [13] and other applications [14]. For Web data analysis, a query (e.g.…”

Section: The Performance Gapmentioning

confidence: 99%

YSmart: Yet Another SQL-to-MapReduce Translator

Lee

Luo

Huai

et al. 2011

2011 31st International Conference on Distributed Computing Systems

127

View full text Add to dashboard Cite

MapReduce has become an effective approach to big data analytics in large cluster systems, where SQL-like queries play important roles to interface between users and systems. However, based on our Facebook daily operation results, certain types of queries are executed at an unacceptable low speed by Hive (a production SQL-to-MapReduce translator). In this paper, we demonstrate that existing SQL-to-MapReduce translators that operate in a one-operation-to-one-job mode and do not consider query correlations cannot generate high-performance MapReduce programs for certain queries, due to the mismatch between complex SQL structures and simple MapReduce framework. We propose and develop a system called YSmart, a correlation aware SQL-to-MapReduce translator. YSmart applies a set of rules to use the minimal number of MapReduce jobs to execute multiple correlated operations in a complex query. YSmart can significantly reduce redundant computations, I/O operations and network transfers compared to existing translators. We have implemented YSmart with intensive evaluation for complex queries on two Amazon EC2 clusters and one Facebook production cluster. The results show that YSmart can outperform Hive and Pig, two widely used SQL-to-MapReduce translators, by more than four times for query execution.

show abstract

Section: The Performance Gapmentioning

confidence: 99%

YSmart: Yet Another SQL-to-MapReduce Translator

Lee

Luo

Huai

et al. 2011

2011 31st International Conference on Distributed Computing Systems

127

View full text Add to dashboard Cite

show abstract

“…The systems community has developed parallel stream systems like System S [23] and S4 [14]. These systems adopt a workflow-based programming model and leave many systems issues such as memory management and I/O operations to user code.…”

Section: Parallel Stream Processingmentioning

confidence: 99%

A platform for scalable one-pass analytics using MapReduce

Mazur

Diao

et al. 2011

Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data

130

View full text Add to dashboard Cite

Today's one-pass analytics applications tend to be data-intensive in nature and require the ability to process high volumes of data efficiently. MapReduce is a popular programming model for processing large datasets using a cluster of machines. However, the traditional MapReduce model is not well-suited for one-pass analytics, since it is geared towards batch processing and requires the data set to be fully loaded into the cluster before running analytical queries. This paper examines, from a systems standpoint, what architectural design changes are necessary to bring the benefits of the MapReduce model to incremental one-pass analytics. Our empirical and theoretical analyses of Hadoop-based MapReduce systems show that the widely-used sort-merge implementation for partitioning and parallel processing poses a fundamental barrier to incremental one-pass analytics, despite various optimizations. To address these limitations, we propose a new data analysis platform that employs hash techniques to enable fast in-memory processing, and a new frequent key based technique to extend such processing to workloads that require a large key-state space. Evaluation of our Hadoop-based prototype using real-world workloads shows that our new platform significantly improves the progress of map tasks, allows the reduce progress to keep up with the map progress, with up to 3 orders of magnitude reduction of internal data spills, and enables results to be returned continuously during the job.

show abstract

“…As such, an intrinsic requirement it has concerns the ability to provide the building blocks for developing application from multiple domains [12]. For example, finance engineering applications [11,17,19,24] have the need for specific types of operators whereas intelligent traffic management [25] might have the need for a different set of operators altogether. Although many imperative programming languages provide object-oriented and modularization constructs, the SPL language relies on primitive and composite operators as its fundamental constructs [18].…”

Section: Rationale For Employing a Model-based Design In Streamsmentioning

confidence: 99%

A model‐based framework for building extensible, high performance stream processing middleware and programming language for IBM InfoSphere Streams

Gedik

Andrade

2011

Softw Pract Exp

Self Cite

View full text Add to dashboard Cite

This work presents an extensive case study on the model-based design of a commercial-grade stream processing middleware (IBM's InfoSphere Streams) its runtime and language (SPL) compiler. The model-based underpinnings are pervasive throughout the whole environment, from describing inter-process communication interfaces and objects to the design of the extensibility mechanism in the runtime and language. In addition to many software engineering advantages such as consistent, uniform, and self-documented integration among the different parts of the system, we show intrinsic performance benefits to the platform derived from this design approach. First, we demonstrate how an incremental compilation strategy employed by the SPL compiler and rooted on the model description of the application, extracted by the compiler as part of the application building process, leads to better compile-time performance. Second, we discuss how the model-based code generation strategy employed by the SPL compiler also leads to increased runtime performance, by specializing the generated code to particular characteristics of the runtime environment. Finally, we show how the extensibility strategy used in the SPL language leads to automatic syntactic and semantic checks at compile time, while enabling behavioral reasoning and specific optimizations at runtime. 1364 B. GEDIK AND H. ANDRADE providing additional capabilities to address new requirements imposed by customers and evolving business demands.The model-based framework forms the basis of the extensibility provided by our language and runtime system. Their research prototype roots and evolution into a commercial-grade software platform are briefly discussed in Section 2. In Section 3, we look in detail at how we used these ideas throughout the entire system and the specific benefits we derived from relying on this approach. For example, a model description can be used to communicate information between the different parts of a system that must interact with one another. Furthermore, we use the notion of a model to capture and expose as much information as deemed necessary to allow detailed reasoning about a program to be carried out at compile time as is described in Section 4. As will be shown, this approach minimizes runtime overheads that are typical of stream computing platforms that rely on interpreting queries, as is the case for many SQL query engines. This leads us to the runtime environment and its model-driven backbone, which we discuss in Section 5.Specifically, this paper presents a model-driven framework we used to develop a new stream processing language (SPL) as well as the distributed runtime (InfoSphere Streams) supporting this language, starting with their research counterparts, respectively, the SPADE language and the System S middleware.Approaching a design task as a model-driven process, whereby a common integration framework is made explicit, is a methodology that has been used in disparate areas such as manufacturing, embedded systems, communication systems, as ...

show abstract

From a stream of relational queries to distributed stream processing

Cited by 20 publications

References 30 publications

YSmart: Yet Another SQL-to-MapReduce Translator

YSmart: Yet Another SQL-to-MapReduce Translator

A platform for scalable one-pass analytics using MapReduce

A model‐based framework for building extensible, high performance stream processing middleware and programming language for IBM InfoSphere Streams

Contact Info

Product

Resources

About