Redundant documents and search effectiveness

Bernstein, Yaniv; Zobel, Justin

doi:10.1145/1099554.1099733

Cited by 38 publications

(30 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar work, in a mono-lingual environment, involves the identification of redundant [4] and co-derivative [3] documents, using fingerprinting techniques. Fingerprints are compact representations of text chunks.…”

Section: Related Workmentioning

confidence: 99%

“…In the context of web search, data redundancy in the search results has already been shown to be an issue [4]. For example, even if a document is considered to be relevant to an information need, when shown after a number of redundant documents, it does not provide the user any additional information.…”

Section: Introductionmentioning

confidence: 99%

“…Wiki-based encyclopedias, or other userdriven sites). Previous work in this area has explored both novelty detection for improving search effectiveness, and the use of fingerprinting techniques for identifying redundant documents [4], but mainly in a monolingual environment.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Cross-Lingual Text Fragment Alignment Using Divergence from Randomness

Yahyaei

Bonzanini

Roelleke

2011

String Processing and Information Retrieval

View full text Add to dashboard Cite

Abstract. This paper describes an approach to automatically align fragments of texts of two documents in different languages. A text fragment is a list of continuous sentences and an aligned pair of fragments consists of two fragments in two documents, which are content-wise related. Cross-lingual similarity between fragments of texts is estimated based on models of divergence from randomness. A set of aligned fragments based on the similarity scores are selected to provide an alignment between sections of the two documents. Similarity measures based on divergence show strong performance in the context of cross-lingual fragment alignment in the performed experiments.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Lingual Text Fragment Alignment Using Divergence from Randomness

Yahyaei

Bonzanini

Roelleke

2011

String Processing and Information Retrieval

View full text Add to dashboard Cite

show abstract

“…(3) We suggest a new technique called bridging for estimating the origin of all selected shingles in a document even though only information about a very small number of shingles in the document is available. (4) We perform an extensive analysis of different algorithms on two real datasets and show that (1), (2) and (3) together provide the best solution for our problem.…”

Section: Introductionmentioning

confidence: 99%

“…(3) Semantic duplication, where pages contain (almost) the same content, but different words. Most attention in the past has been given to finding near-duplicate pages [4,6,10,11,12,16,17]. Recently, attention has shifted towards detecting partial replication [7,15,14], but none of the prior work focuses on the origin detection problem.…”

Section: Introductionmentioning

confidence: 99%

Detecting the origin of text segments efficiently

Hamid

Behzadi

Christoph

et al. 2009

Proceedings of the 18th International Conference on World Wide Web

View full text Add to dashboard Cite

In the origin detection problem an algorithm is given a set S of documents, ordered by creation time, and a query document D. It needs to output for every consecutive sequence of k alphanumeric terms in D the earliest document in S in which the sequence appeared (if such a document exists). Algorithms for the origin detection problem can, for example, be used to detect the "origin" of text segments in D and thus to detect novel content in D. They can also find the document from which the author of D has copied the most (or show that D is mostly original.)We propose novel algorithms for this problem and evaluate them together with a large number of previously published algorithms. Our results show that (1) detecting the origin of text segments efficiently can be done with very high accuracy even when the space used is less than 1% of the size of the documents in S, (2) the precision degrades smoothly with the amount of available space, (3) various estimation techniques can be used to increase the performance of the algorithms.

show abstract

The Case of the Duplicate Documents Measurement, Search, and Science

Zobel

Bernstein

2006

Frontiers of WWW Research and Development - APWeb 2006

View full text Add to dashboard Cite

Redundant documents and search effectiveness

Cited by 38 publications

References 15 publications

Cross-Lingual Text Fragment Alignment Using Divergence from Randomness

Cross-Lingual Text Fragment Alignment Using Divergence from Randomness

Detecting the origin of text segments efficiently

The Case of the Duplicate Documents Measurement, Search, and Science

Contact Info

Product

Resources

About