Extracting and Analyzing Hidden Graphs from Relational Databases

Abstract

Analyzing interconnection structures among underlying entities or objects in a dataset through the use of graph analytics can provide tremendous value in many application domains. However, graphs are not the primary representation choice for storing most data today, and in order to have access to these analyses, users are forced to manually extract data from their data stores, construct the requisite graphs, and then load them into some graph engine in order to execute their graph analysis task. Moreover, in many cases (especially when the graphs are dense), these graphs can be significantly larger than the initial input stored in the database, making it infeasible to construct or analyze such graphs in memory. In this paper we address both of these challenges by building a system that enables users to declaratively specify graph extraction tasks over a relational database schema and then execute graph algorithms on the extracted graphs. We propose a declarative domain specific language for this purpose, and pair it up with a novel condensed, inmemory representation that significantly reduces the memory footprint of these graphs, permitting analysis of larger-than-memory graphs. We present a general algorithm for creating such a condensed representation for a large class of graph extraction queries against arbitrary schemas. We observe that the condensed representation suffers from a duplication issue, that results in inaccuracies for most graph algorithms. We then present a suite of in-memory representations that handle this duplication in different ways and allow trading off the memory required and the computational cost for executing different graph algorithms. We also introduce several novel deduplication algorithms for removing this duplication in the graph, which are of independent interest for graph compression, and provide a comprehensive experimental evaluation over several real-world and synthetic datasets illustrating these trade-offs.

GraphGen

“…Creating the edges here requires doing a self-join on the AuthorPublication table followed by a DISTINCT. A key challenge here is that the size of the self-join is often much larger than the size of the original AuthorPublication table [16]. Although this approach (Approach 3) is likely to be the most attractive in practice, there hasn't been much systematic work on understanding it; one of our goals here is to systematically explore it and discuss the key challenges that come up.…”

“…Ringo [9] also provides operators for converting from an in-memory relational table representation to a graph representation, but is not intended as a layer on top of an RDBMS. Finally, in our prior work on GraphGen, we focused on the problem of exploring hidden graphs within relational databases [17] and dealing with large intermediate results that get generated in the process [16].…”

“…By "large-output", we refer to joins where the cardinality of the join attribute is low compared to the input table sizes , resulting in a large number of tuples in the join. As we showed in our prior work [16], operating on a condensed representation of the graph in-memory may be ideal in such cases. This representation is built to e ciently load the de ned graph into memory without explicitly generating all of the edges.…”

“…GraphGen also provides a vertex-centric programming model on top of this API to simplify writing analysis tasks. We refer the reader to [16] for further details. Currently, GraphGen primarily supports the second option, and we are working on adding support for GraphGenQL through developing translation programs to convert those programs into SQL.…”

Guiding Search in Relational Pathfinding-based Concept Discovery via Bivariate Statistical Methods

“…In case of very large graphs, this basic setting becomes inefficient and mechanisms to speedup or guide search become essential. In literature various techniques including parallelization [13], declaratively specifying graph extraction tasks over database schemas [14], and introduction of mode declarations [15] have been proposed to speed-up graph based concept discovery systems or reduce the search space. In this study, we empirically evaluate performance of three bivariate statistical methods, namely frequency ratio (FR), hazard index (HI), and weight of evidence (WoE), as heuristics to guide search in relational pathfinding-based concept discovery systems.…”