Indexing correlated probabilistic databases

Abstract

With large amounts of correlated probabilistic data being generated in a wide range of application domains including sensor networks, information extraction, event detection etc., effectively managing and querying them has become an important research direction. While there is an exhaustive body of literature on querying independent probabilistic data, supporting efficient queries over large-scale, correlated databases remains a challenge. In this paper, we develop efficient data structures and indexes for supporting inference and decision support queries over such databases. Our proposed hierarchical data structure is suitable both for inmemory and disk-resident databases. We represent the correlations in the probabilistic database using a junction tree over the tuple-existence or attribute-value random variables, and use tree partitioning techniques to build an index structure over it. We show how to efficiently answer inference and aggregation queries using such an index, resulting in orders of magnitude performance benefits in most cases. In addition, we develop novel algorithms for efficiently keeping the index structure up-to-date as changes (inserts, updates) are made to the probabilistic database. We present a comprehensive experimental study illustrating the benefits of our approach to query processing in probabilistic databases.

AFFINITY: Efficiently querying statistical measures on time-series data

“…Similarly, GAMPS [23] uses ratio signals for compressing time-series data and proposes approaches for query processing over such compressed data. More recently, there has been research conducted on indexing and querying correlated uncertain information using probabilistic databases [24,25]. Lastly, Ke et al [26] propose approaches for searching graphs correlated to a given query graph.…”

A generic framework for handling uncertain data with local correlations

“…Note that, our model is generic enough to cover the previously proposed model that assumes either object independence [6,25] or global correlations [28,29,31,16]. Specifically, they, respectively, correspond to two special cases: 1) the database has multiple LCPs, each containing only one object, and 2) the database has only one LCP, which contains all objects correlated with each other.…”

“…Wang et al [31] based on a class of FirstOrder Bayesian Network to model correlated uncertain data, and performed relational operators (e.g., select, project and join). Kanagal and Deshpande [16] indexed correlated probabilistic databases to answer inference/aggregation queries. A large junction tree is recursively divided into partitions via classical tree partition algorithms which form a hierarchical tree structure.…”

Authentication of uncertain data based on k-means clustering

“…The most common examples are sensor networks, moving objects' tracking, data cleaning, and marketing applications [1][2][3]. In order to manage uncertain data, several methods have been proposed for query processing and efficient indexing [4][5][6][7].…”