UMass at TREC 2004: Novelty and HARD

Abstract

For the TREC 2004 Novelty track, UMass participated in all four tasks. Although finding relevant sentences was harder this year than last, we continue to show marked improvements over the baseline of calling all sentences relevant, with a variant of tfidf being the most successful approach. We achieve 5-9% improvements over the baseline in locating novel sentences, primarily by looking at the similarity of a sentence to earlier sentences and focusing on named entities.For the High Accuracy Retrieval from Documents (HARD) track, we investigated the use of clarification forms, fixed-and variable-length passage retrieval, and the use of metadata. Clarification form results indicate that passage level feedback can provide improvements comparable to user supplied related-text for document evaluation and outperforms related-text for passage evaluation. Document retrieval methods without a query expansion component show the most gains from related-text. We also found that displaying the top passages for feedback outperformed displaying centroid passages. Named entity feedback resulted in mixed performance. Our primary findings for passage retrieval are that document retrieval methods performed better than passage retrieval methods on the passage evaluation metric of binary preference at 12,000 characters, and that clarification forms improved passage retrieval for every retrieval method explored. We found no benefit to using variable-length passages over fixed-length passages for this corpus. Our use of geography and genre metadata resulted in no significant changes in retrieval performance.

Query Expansion with Locally-Trained Word Embeddings

“…This interpolated language model can then be used with Equation 4 to rank documents (Abdul-Jaleel et al, 2004). We will refer to this as the expanded query score of a document.…”

Promoting Divergent Terms in the Estimation of Relevance Models

“…In [14] they compared five methods to estimate the query language models: RM3 and RM4 [1]; a divergence minimization model (DMM) and a simple mixture model (SMM) [23]; and a regularized mixture model (RMM) [20]. The main finding of this paper was that, in general, RM3 is the best and most stable method among the others.…”

“…The main finding of this paper was that, in general, RM3 is the best and most stable method among the others. RM3 and RM4 [1] are extensions of the originally formulated RM1 and RM2 approximations, respectively. These extensions linearly interpolate the original query with the terms selected for expansion using RM1 or RM2.…”

Online Sentence Novelty Scoring for Topical Document Streams

“…F-score Blott et al (2004) / Gamon (2006 0.622 Tomiyama et al (2004) 0.619 Abdul-Jaleel et al (2004) 0.618 Schiffman and McKeown (2004) 0 …”