An architecture for compiling UDF-centric workflows

Crotty, Andrew; Galakatos, Alex; Dursun, Kayhan; Kraska, Tim; Binnig, Carsten; Çetintemel, Uǧur; Zdonik, Stanley B.

doi:10.14778/2824032.2824045

Cited by 74 publications

(57 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1536). This is in line with reports in [8], [9], [10] that the majority of clusters are rather small and have fewer than 50 nodes. Also, we experimented with raw datasets at the orders of hundreds gigabytes (the corresponding RDDs are 1-5 times larger), since, this is the typical dataset processed even in companies that are notorious for their big data application demands [11].…”

Section: Our Setting and The Benchmarking Applicationssupporting

confidence: 93%

Dynamic Configuration of Partitioning in Spark Applications

Gounaris

Kougka

Tous

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-Spark has become one of the main options for large-scale analytics running on top of shared-nothing clusters. This work aims to make a deep dive into the parallelism configuration and shed light on the behavior of parallel spark jobs. It is motivated by the fact that running a Spark application on all the available processors does not necessarily imply lower running time, while may entail waste of resources. We first propose analytical models for expressing the running time as a function of the number of machines employed. We then take another step, namely to present novel algorithms for configuring dynamic partitioning with a view to minimizing resource consumption without sacrificing running time beyond a user-defined limit. The problem we target is NP-hard. To tackle it, we propose a greedy approach after introducing the notions of dependency graphs and of the benefit from modifying the degree of partitioning at a stage; complementarily, we investigate a randomized approach. Our polynomial solutions are capable of judiciously use the resources that are potentially at user's disposal and strike interesting trade-offs between running time and resource consumption. Their efficiency is thoroughly investigated through experiments based on real execution data.

show abstract

Section: Our Setting and The Benchmarking Applicationssupporting

confidence: 93%

Dynamic Configuration of Partitioning in Spark Applications

Gounaris

Kougka

Tous

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…Unless otherwise stated, we report median values in seconds. In most cases, we do not exploit the full computational power of MareNostrum but use a more limited amount of cores, motivated by the fact that most Hadoop clusters to date are relatively small [17].…”

Section: Resultsmentioning

confidence: 99%

Spark deployment and performance evaluation on the MareNostrum supercomputer

Tous

Gounaris

Tripiana

et al. 2015

2015 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-In this paper we present a framework to enable data-intensive Spark workloads on MareNostrum, a petascale supercomputer designed mainly for compute-intensive applications. As far as we know, this is the first attempt to investigate optimized deployment configurations of Spark on a petascale HPC setup. We detail the design of the framework and present some benchmark data to provide insights into the scalability of the system. We examine the impact of different configurations including parallelism, storage and networking alternatives, and we discuss several aspects in executing Big Data workloads on a computing system that is based on the compute-centric paradigm. Further, we derive conclusions aiming to pave the way towards systematic and optimized methodologies for finetuning data-intensive application on large clusters emphasizing on parallelism configurations.

show abstract

“…In-Database advanced analytics. Recent work at the intersection of databases and machine learning are extensively trying to facilitate efficient in-database analytics and have built frameworks and systems to realize such an integration [7,14,15,17,[55][56][57][58][59][60][61] (see [62] for a survey of various methods and systems). DAnA takes a step forward and exposes FPGA acceleration for in-Database analytics by providing a specialized component, Strider, that directly interfaces with the database to alleviate some of the shortcomings of the traditional Von-Neumann architecture in general purpose compute systems.…”

Section: Related Workmentioning

confidence: 99%

In-RDBMS hardware acceleration of advanced analytics

et al. 2018

View full text Add to dashboard Cite

The data revolution is fueled by advances in machine learning, databases, and hardware design. Programmable accelerators are making their way into each of these areas independently. As such, there is a void of solutions that enables hardware acceleration at the intersection of these disjoint fields. This paper sets out to be the initial step towards a unifying solution for in-Database Acceleration of Advanced Analytics (DAnA). Deploying specialized hardware, such as FPGAs, for in-database analytics currently requires hand-designing the hardware and manually routing the data. Instead, DAnA automatically maps a high-level specification of advanced analytics queries to an FPGA accelerator. The accelerator implementation is generated for a User Defined Function (UDF), expressed as a part of an SQL query using a Pythonembedded Domain-Specific Language (DSL). To realize an efficient in-database integration, DAnA accelerators contain a novel hardware structure, Striders, that directly interface with the buffer pool of the database. Striders extract, cleanse, and process the training data tuples that are consumed by a multi-threaded FPGA engine that executes the analytics algorithm. We integrate DAnA with PostgreSQL to generate hardware accelerators for a range of real-world and synthetic datasets running diverse ML algorithms. Results show that DAnA-enhanced PostgreSQL provides, on average, 8.3× end-to-end speedup for real datasets, with a maximum of 28.2×. Moreover, DAnA-enhanced PostgreSQL is, on average, 4.0× faster than the multi-threaded Apache MADLib running on Greenplum. DAnA provides these benefits while hiding the complexity of hardware design from data scientists and allowing them to express the algorithm in ≈30-60 lines of Python.

show abstract

An architecture for compiling UDF-centric workflows

Cited by 74 publications

References 23 publications

Dynamic Configuration of Partitioning in Spark Applications

Dynamic Configuration of Partitioning in Spark Applications

Spark deployment and performance evaluation on the MareNostrum supercomputer

In-RDBMS hardware acceleration of advanced analytics

Contact Info

Product

Resources

About