Optimizing scoring functions and indexes for proximity search in type-annotated corpora

Abstract

We introduce a new, powerful class of text proximity queries: find an instance of a given "answer type" (person, place, distance) near "selector" tokens matching given literals or satisfying given ground predicates. An example query is type=distance NEAR Hamburg Munich. Nearness is defined as a flexible, trainable parameterized aggregation function of the selectors, their frequency in the corpus, and their distance from the candidate answer. Such queries provide a key data reduction step for information extraction, data integration, question answering, and other text-processing applications. We describe the architecture of a next-generation information retrieval engine for such applications, and investigate two key technical problems faced in building it. First, we propose a new algorithm that estimates a scoring function from past logs of queries and answer spans. Plugging the scoring function into the query processor gives high accuracy: typically, an answer is found at rank 2-4. Second, we exploit the skew in the distribution over types seen in query logs to optimize the space required by the new index structures required by our system. Extensive performance studies with a 10GB, 2-million document TREC corpus and several hundred TREC queries show both the accuracy and the efficiency of our system. From an initial 4.3GB index using 18,000 types from WordNet, we can discard 88% of the space, while inflating query times by a factor of only 1.9. Our final index overhead is only 20% of the total index space needed.

Annotating and searching web tables using entities, types and relationships

“…This query form is a natural extension of entity search [7,8] to Web table sources. A common special case of the above query form is to look for entities that have a given value of an attribute.…”

“…Such queries seek entities of a specified type that are mentioned significantly often near specified words [7]. This has been further extended [8] to ad-hoc retrieval of multiple entities that appear to form a logical record, e.g., name, affiliation and email of database researchers.…”

Entity-Relationship Queries over Wikipedia

“…Contexts that are likely to be true evidence are given higher importance. Existing systems only exploit proximity feature [10,11] and may not leverage redundancy [10]. In summary, we make the following contributions in this paper:…”

Finding support sentences for entities

“…None of these works deal with the issue of explaining a query-entity relationship. The closest is perhaps [4] which discusses sentence ranking models where the query includes a constraint on a type of entity (e.g. a location, a person).…”