Asynchronous and fault-tolerant recursive datalog evaluation in shared-nothing engines

Wang, Jingjing; Balazinska, Magdalena; Halperin, Daniel

doi:10.14778/2824032.2824052

Cited by 49 publications

(42 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5 of [39] and the program in Fig. 2 of [52]. Although the iterated fixpoint computation produces the correct results for these programs, this is not true for other programs.…”

Section: Examplementioning

confidence: 99%

See 1 more Smart Citation

Scaling up the performance of more powerful Datalog systems on multicore machines

2016

View full text Add to dashboard Cite

Extending RDBMS technology to achieve performance and scalability for queries that are much more powerful than those of SQL-2 has been the goal of deductive database research for more than thirty years. The DeALS system has made major progress toward this goal, by (1) Datalog extensions that support the more powerful recursive queries needed in advanced applications, and (2) superior performance for both traditional recursive queries and those made possible by the new extensions, while (3) delivering competitive performance with commercial RDBMSs on non-recursive queries. In this paper, we focus on the techniques used to support the in-memory evaluation of Datalog programs on multicore machines. In DeALS, a Datalog program is represented as an AND/OR tree, and multiple copies of the same AND/OR tree are used to access the tables in the database concurrently during the parallel evaluation. We describe compilation techniques that (1) recognize when the given program is lock-free, (2) transform a locking program into a lock-free program, and (3) find an efficient parallel plan that correctly evaluates the program while minimizing the use of locks and other overhead required for parallel evaluation. Extensive experiments demonstrate the effectiveness of the proposed techniques.

show abstract

“…5 of [39] and the program in Fig. 2 of [52]. Although the iterated fixpoint computation produces the correct results for these programs, this is not true for other programs.…”

Section: Examplementioning

confidence: 99%

“…However, it seems that both [39] and [52] ignore this problem, and allow unrestricted use of count or sum in recursion, e.g., the program in Fig. 5 of [39] and the program in Fig.…”

Section: Examplementioning

confidence: 99%

Scaling up the performance of more powerful Datalog systems on multicore machines

2016

View full text Add to dashboard Cite

show abstract

“…This has the effect of giving smaller labels a "head start" and reduces the amount of churn among nodes with large identifiers. It accomplishes the same goal as asynchronous systems, like Myria [40] and SociaLite [38], that propagate more successful labels.…”

Section: Explanations For Connected Componentsmentioning

confidence: 99%

“…Figure 14 also depicts the performance of the state-of-the-art systems for graph analytics, Myria [40] and SociaLite [38], on the label propagation algorithm. Note that we were unable to get SociaLite to report more than 1.7M labels for the Twitter graph, thus, we do not report its elapsed time for this dataset.…”

Section: Explanations For Connected Componentsmentioning

confidence: 99%

Explaining outputs in modern data analytics

et al. 2016

View full text Add to dashboard Cite

We report on the design and implementation of a general framework for interactively explaining the outputs of modern data-parallel computations, including iterative data analytics. To produce explanations, existing works adopt a naive backward tracing approach which runs into known issues; naive backward tracing may identify: (i) too much information that is difficult to process, and (ii) not enough information to reproduce the output, which hinders the logical debugging of the program. The contribution of this work is twofold. First, we provide methods to effectively reduce the size of explanations based on the first occurrence of a record in an iterative computation. Second, we provide a general method for identifying explanations that are sufficient to reproduce the target output in arbitrary computations -- a problem for which no viable solution existed until now. We implement our approach on differential dataflow , a modern high-throughput, low-latency dataflow platform. We add a small (but extensible) set of rules to explain each of its data-parallel operators, and we implement these rules as differential dataflow operators themselves. This choice allows our implementation to inherit the performance characteristics of differential dataflow, and results in a system that efficiently computes and updates explanatory inputs even as the inputs of the reference computation change. We evaluate our system with various analytic tasks on real datasets, and we show that it produces concise explanations in tens of milliseconds, while remaining faster -- up to two orders of magnitude -- than even the best implementations that do not support explanations.

show abstract

“…A growing body of research on scalable data analytics has brought a renaissance of interest in Datalog because of its ability to specify, declaratively, advanced data-intensive applications that execute efficiently over different systems and architectures, including massively parallel ones (Seo et al 2013;Shkapsky et al 2013;Yang and Zaniolo 2014;Aref et al 2015;Wang et al 2015;Yang et al 2015;Shkapsky et al 2016;Yang et al 2017). A common thread in this new generation of Datalog systems is the use of aggregates in recursion, since aggregates enable the concise expression and efficient support of much more powerful algorithms than those expressible by programs that are stratified w.r.t.…”

Section: Introductionmentioning

confidence: 99%

Fixpoint semantics and optimization of recursive Datalog programs with aggregates

Zaniolo¹,

Yang²,

Das³

et al. 2017

Theory and Practice of Logic Programming

View full text Add to dashboard Cite

A very desirable Datalog extension investigated by many researchers in the last thirty years consists in allowing the use of the basic SQL aggregates min, max, count and sum in recursive rules. In this paper, we propose a simple comprehensive solution that extends the declarative least-fixpoint semantics of Horn Clauses, along with the optimization techniques used in the bottom-up implementation approach adopted by many Datalog systems. We start by identifying a large class of programs of great practical interest in which the use of min or max in recursive rules does not compromise the declarative fixpoint semantics of the programs using those rules. Then, we revisit the monotonic versions of count and sum aggregates proposed in (Mazuran et al. 2013b) and named, respectively, mcount and msum. Since mcount, and also msum on positive numbers, are monotonic in the lattice of set-containment, they preserve the fixpoint semantics of Horn Clauses. However, in many applications of practical interest, their use can lead to inefficiencies, that can be eliminated by combining them with max, whereby mcount and msum become the standard count and sum. Therefore, the semantics and optimization techniques of Datalog are extended to recursive programs with min, max, count and sum, making possible the advanced applications of superior performance and scalability demonstrated by BigDatalog (Shkapsky et al. 2016) and Datalog-MC (Yang et al. 2017). This paper is under consideration for acceptance in TPLP.

show abstract

Asynchronous and fault-tolerant recursive datalog evaluation in shared-nothing engines

Cited by 49 publications

References 15 publications

Scaling up the performance of more powerful Datalog systems on multicore machines

Scaling up the performance of more powerful Datalog systems on multicore machines

Explaining outputs in modern data analytics

Fixpoint semantics and optimization of recursive Datalog programs with aggregates

Contact Info

Product

Resources

About