A Parallel Framework for In-Memory Construction of Term-Partitioned Inverted Indexes

Küçükyılmaz, Tayfun; Türk, Ata; Aykanat, Cevdet

doi:10.1093/comjnl/bxr133

Cited by 4 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Assuming the documents are already uniformly distributed on the nodes, constructing a document-based-partitioned index is a relatively trivial task since each node can simply build a local index on its collection. Constructing a term-based-partitioned index via message passing, on the other hand, requires coordination and communication among the nodes Kucukyilmaz et al 2012]. -Load imbalance.…”

Section: Term-based Versus Document-based Inverted Index Partitioningmentioning

confidence: 99%

A term-based inverted index partitioning model for efficient distributed query processing

et al. 2013

View full text Add to dashboard Cite

In a shared-nothing, distributed text retrieval system, queries are processed over an inverted index that is partitioned among a number of index servers. In practice, the index is either document-based or term-based partitioned. This choice is made depending on the properties of the underlying hardware infrastructure, query traffic distribution, and some performance and availability constraints. In query processing on retrieval systems that adopt a term-based index partitioning strategy, the high communication overhead due to the transfer of large amounts of data from the index servers forms a major performance bottleneck, deteriorating the scalability of the entire distributed retrieval system. In this work, to alleviate this problem, we propose a novel inverted index partitioning model that relies on hypergraph partitioning. In the proposed model, concurrently accessed index entries are assigned to the same index servers, based on the inverted index access patterns extracted from the past query logs. The model aims tominimize the communication overhead that will be incurred by future queries while maintaining the computational load balance among the index servers. We evaluate the performance of the proposed model through extensive experiments using a real-life text collection and a search query sample. Our results show that considerable performance gains can be achieved relative to the term-based index partitioning strategies previously proposed in literature. In most cases, however, the performance remains inferior to that attained by document-based partitioning. © 2013 ACM

show abstract

Section: Term-based Versus Document-based Inverted Index Partitioningmentioning

confidence: 99%

A term-based inverted index partitioning model for efficient distributed query processing

et al. 2013

View full text Add to dashboard Cite

show abstract

“…Therefore, advances in high-performance computing provide the infrastructure to handle large workloads [Kucukyilmaz et al 2012]. For this reason, in this paper, we propose a heterogeneous parallel novel architecture for inverted index generation from the in-memory inversion algorithm.…”

Section: Introductionmentioning

confidence: 99%

Heterogeneous Parallel Architecture for Inverted Index Generation

Silveira

Soares

Brandão

et al. 2019

Anais Do XX Simpósio Em Sistemas Computacionais De Alto Desempenho (SSCAD 2019)

View full text Add to dashboard Cite

The amount of data generated on the Web has increased dramatically, as well as the need for computational power to prepare this information. In particular, indexers process these data to extract terms and their occurrences, storing them in an inverted file, a compact data structure that provides quick search. However, this task involves processing of a large amount of data, requiring high computational power. In this article, we present a heterogeneous parallel architecture that uses CPU and GPU in a cluster to accelerate inverted index generation. Experimental results show that the proposed architecture provides faster execution times, up to 60 times in classification and 23 times in the compression of 1 million elements.

show abstract

A Novel Emergency Cross-Media Information Retrieval Model

Wang

et al. 2013

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

A Parallel Framework for In-Memory Construction of Term-Partitioned Inverted Indexes

Cited by 4 publications

References 23 publications

A term-based inverted index partitioning model for efficient distributed query processing

A term-based inverted index partitioning model for efficient distributed query processing

Heterogeneous Parallel Architecture for Inverted Index Generation

A Novel Emergency Cross-Media Information Retrieval Model

Contact Info

Product

Resources

About