Self-indexing inverted files for fast text retrieval

Abstract

Query processing costs on large text databases are dominated by the need to retrieve and scan the inverted list of each query term. Here we show that query response time for conjunctive Boolean queries and for informal ranked queries can be dramatically reduced, at little cost in terms of storage, by the inclusion of an internal index in each inverted list. This method has been applied in a retrieval system for a collection of nearly two million short documents. Our experimental results show that the selfindexing strategy adds less than 20% to the size of the inverted file, but, for Boolean queries of 5-10 terms, can reduce processing time to under one fifth of the previous cost. Similarly, ranked queries of 40-50 terms can be evaluated in as little as 25% of the previous time, with little or no loss of retrieval effectiveness.

Searching web data: An entity retrieval and high-performance indexing model

“…In the average case, the complexity of an intersection is reduced to sub-linear time with the use of self-indexing [59] over the entity identifiers in order to skip and avoid unnecessary record comparisons.…”

“…Such a join is linear with the number of results in the case of the quad table, and sub-linear in average for the node and field index with the use of the self-indexing method [59]. In contrast, Semplore has often to resort to possibly expensive external sort before merge-join operations.…”

Semplore: An IR Approach to Scalable Hybrid Query of Semantic Web Data

“…As to the physical storage and access of the logical inverted index structure , it has been thoroughly studied in the IR field, which results in many optimized methods, such as byte-aligned index compression [13] and self-indexing [14]. Furthermore, in the proposed PosIdx method, relation objects enjoy the benefit of spatial locality for fast access, because positions of a term are usually physically stored together and continuously in modern IR engines.…”

“…By adding additional indexing structures to the inverted index (e.g., self-indexing [14]), modern IR engines can supply a very efficient stream reader for a posting list AIS.…”

“…The algorithm is called Bit Vector Intersection (BVI) and shown in Algorithm 3. Note that N = |O R | in line 1 can be directly obtained from inverted index as the document frequency of the term R without any computation at run time. Line 4 and 5 can be implemented together on the PosIdx index very efficiently using a sequential scan on the position list during merge-sort, with the help of self-indexing [14].…”