Hierarchical Bitmap Index: An Efficient and Scalable Indexing Technique for Set-Valued Attributes

Morzy, Mikołaj; Morzy, Tadeusz; Νανόπουλος, Αλέξανδρος; Manolopoulos, Yannis

doi:10.1007/978-3-540-39403-7_19

Cited by 24 publications

(12 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We suggest that a new storage structure will be appropriate because we can use the existing data and gathered statistics during previous operations on existing storage structures to make better decisions for the next appropriate storage structure selection. Previously published results from Bender et al [6], Chen et al [10], and Morzy et al [21] also motivate our decision for the use of hierarchically-organized storage structures.…”

Section: Problem Statement and Motivationmentioning

confidence: 97%

See 1 more Smart Citation

ECOS: Evolutionary Column-Oriented Storage

Rahman

Schallehn

Saake

2011

Lecture Notes in Computer Science

View full text Add to dashboard Cite

As DBMS has grown more powerful over the last decades, they have also become more complex to manage. To achieve efficiency by database tuning is nowadays a hard task carried out by experts. This development inspired the ongoing research on self-tuning to make database systems more easily manageable. In this report, we present a customizable self-tuning storage manager, we termed as Evolutionary Column-Oriented Storage (ECOS). The capability of self-tuning data management with minimal human intervention, which is the main design goal for ECOS, is achieved by dynamically adjusting the storage structures of a column-oriented DBMS according to data size and access characteristics. It is based on the Decomposed Storage Model (DSM) with support for customization at the table-level using five different variations of DSM. Furthermore, it also proposes fine-grained customization of storage structures at the column-level. It uses hierarchicallyorganized storage structures for each column to enable autonomic selection of the suitable storage structure along the hierarchy (as hierarchy-level increases) using an evolution mechanism. Moreover, for ECOS we proposed the concept of an evolution path that provides a reduction of human intervention for database maintenance. We evaluated ECOS empirically using a custom micro benchmark showing performance improvement.

show abstract

Section: Problem Statement and Motivationmentioning

confidence: 97%

“…Morzy et al in [21] proposed a hierarchical bitmap index for indexing set-valued attributes. Later, Chmiel et al in [11] extended that concept to present hierarchically-organized bitmap indexes for indexing dimensional data.…”

Section: Related Workmentioning

confidence: 99%

ECOS: Evolutionary Column-Oriented Storage

Rahman

Schallehn

Saake

2011

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In this context, one commonly used approach has been to employ a bitmap called a 'signature' that is similar to a Bloom filter but typically does not explore the use of multiple hash functions [9,26,25]. Also, without access to a TCAM, one has to resort to linear search of all the bitmaps, which is clearly prohibitively expensive.…”

Section: Set Queries In Databasesmentioning

confidence: 99%

“…For example, database researchers have studied set query problems in order to facilitate efficient querying on set-valued attributes and in the context of data mining tasks [25,24,26,9]. They also occur in many information retrieval problems -in one, we are required to find a set of documents containing the set of query words, and in another, we are required to find if a document is a subset of the set of documents in the corpus.…”

Section: Introductionmentioning

confidence: 99%

Small subset queries and bloom filters using ternary associative memories, with applications

GoelAshish

GuptaPankaj

2010

SIGMETRICS Perform. Eval. Rev.

View full text Add to dashboard Cite

Associative memories offer high levels of parallelism in matching a query against stored entries. We design and analyze an architecture which uses a single lookup into a Ternary Content Addressable Memory (TCAM) to solve the subset query problem for small sets, i.e., to check whether a given set (the query) contains (or alternately, is contained in) any one of a large collection of sets in a database. We use each TCAM entry as a small Ternary Bloom Filter (each 'bit' of which is one of {0,1," * " }) to store one of the sets in the collection. Like Bloom filters, our architecture is susceptible to false positives. Since each TCAM entry is quite small, asymptotic analyses of Bloom filters do not directly apply. Surprisingly, we are able to show that the asymptotic false positive probability formula can be safely used if we penalize the small Bloom filter by taking away just one bit of storage and adding just half an extra set element before applying the formula. We believe that this analysis is independently interesting.The subset query problem has applications in databases, network intrusion detection, packet classification in Internet routers, and Information Retrieval. We demonstrate our architecture on one illustrative streaming application -intrusion detection in network traffic. By shingling the strings in the database, we can perform a single subset query, and hence a single TCAM search, to skip many bytes in the stream. We evaluate our scheme on the open source CLAM anti-virus database, for worst-case as well as random streams. Our architecture appears to be at least one order of magnitude faster than previous approaches. Since the individual Bloom filters must fit in a single TCAM entry (currently 72 to 576 bits), our solution applies only when each set is of a small cardinality. However, this is sufficient for many typical applications. Also, recent algorithms for the subset-query problem use a small-set version as a subroutine.

show abstract

“…This problem has been studied by the database community in various guises, e.g., similarity search [11] and set joins on similarity predicates [15]. Specific solutions advocate the use of signature trees [7], hierarchical bitmap indices [10], and locality sensitive hashing [4], especially the technique of minwise independent permutations [2] that is particularly geared toward the Jaccard's coefficient. In this paper, we combine an AD-tree data structure [9] with the signature tables [1] approach for efficient similarity search in categorical data.…”

Section: Data Structuresmentioning

confidence: 99%

Algorithms for storytelling

Kumar

Ramakrishnan

Helm

et al. 2006

Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

We formulate a new data mining problem called storytelling as a generalization of redescription mining. In traditional redescription mining, we are given a set of objects and a collection of subsets defined over these objects. The goal is to view the set system as a vocabulary and identify two expressions in this vocabulary that induce the same set of objects. Storytelling, on the other hand, aims to explicitly relate object sets that are disjoint (and hence, maximally dissimilar) by finding a chain of (approximate) redescriptions between the sets. This problem finds applications in bioinformatics, for instance, where the biologist is trying to relate a set of genes expressed in one experiment to another set, implicated in a different pathway. We outline an efficient storytelling implementation that embeds the CARTwheels redescription mining algorithm in an A* search procedure, using the former to supply next move operators on search branches to the latter. This approach is practical and effective for mining large datasets and, at the same time, exploits the structure of partitions imposed by the given vocabulary. Three application case studies are presented: a study of word overlaps in large English dictionaries, exploring connections between genesets in a bioinformatics dataset, and relating publications in the PubMed index of abstracts.

show abstract

Hierarchical Bitmap Index: An Efficient and Scalable Indexing Technique for Set-Valued Attributes

Cited by 24 publications

References 15 publications

ECOS: Evolutionary Column-Oriented Storage

ECOS: Evolutionary Column-Oriented Storage

Small subset queries and bloom filters using ternary associative memories, with applications

Algorithms for storytelling

Contact Info

Product

Resources

About