Beyond bag‐of‐words: Bigram‐enhanced context‐dependent term weights

Abstract

While term independence is a widely held assumption in most of the established information retrieval approaches, it is clearly not true and various works in the past have investigated a relaxation of the assumption. One approach is to use n-grams in document representation instead of unigrams. However, the majority of early works on n-grams obtained only modest performance improvement. On the other hand, the use of information based on supporting terms or "contexts" of queries has been found to be promising. In particular, recent studies showed that using new context-dependent term weights improved the performance of relevance feedback (RF) retrieval compared with using traditional bag-of-words BM25 term weights. Calculation of the new term weights requires an estimation of the local probability of relevance of each query term occurrence. In previous studies, the estimation of this probability was based on unigrams that occur in the neighborhood of a query term. We explore an integration of the n-gram and context approaches by computing context-dependent term weights based on a mixture of unigrams and bigrams. Extensive experiments are performed using the title queries of the Text Retrieval Conference (TREC)-6, TREC-7, TREC-8, and TREC-2005 collections, for RF with relevance judgment of either the top 10 or top 20 documents of an initial retrieval. We identify some crucial elements needed in the use of bigrams in our methods, such as proper inverse document frequency (IDF) weighting of the bigrams and noise reduction by pruning bigrams with large document frequency values. We show that enhancing context-dependent term weights with bigrams is effective in further improving retrieval performance.

A context‐dependent relevance model

“…Subsequently, Dang, Luk, and Allan () extended the work to include n ‐grams in the B&D procedure. In particular, they showed that including bigrams ( n = 2) could improve retrieval performance over unigram B&D, while larger values of n did not lead to further improvement.…”

“…In Dang et al. (), a bigram is defined as an ordered pair of words in a document after stop‐word removal. An additional requirement is that the adjacent members ( t i , t i+1 ) should not be separated by any punctuation, excluding hyphens, in the document.…”

“…Dang et al. () identified two ingredients in their bigram B&D method that were needed to attain robust performance improvement over using unigrams only. First, they found that in calculating the idf weighting for bigrams, it was necessary to use a “local” document frequency.…”

“…Inspired by the promising results of the B&D method of Dang et al. (, ), we wish to investigate an adaptation of the method to introduce context‐dependence in the language modeling framework. In the setting of relevance feedback, we apply the B&D procedure to the RM3 relevance model.…”

“…Hence, the significance of our work is the demonstration of the usefulness of context consideration in the language model framework. It provides a new way to go beyond the bag‐of‐words representation in this framework (Dang, Luk, & Allan, ). Furthermore, together with the previous studies within the BM25 framework (Dang et al., ), our results show that the effectiveness of our method for using context information in IR is quite general and not limited to any specific retrieval model.…”