Detecting near-duplicate documents using sentence-level features and supervised learning

Lin, Yung-Shen; Liao, Ting-Yi; Lee, Shie-Jue

doi:10.1016/j.eswa.2012.08.045

Cited by 19 publications

(10 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…← the index of boundary object of matching partition of on (21) for all ( < < ) in do (22) if in ℎ then (23) c o n t i n u e (24) end if (25) ← sigNCD( , )…”

Section: P1: Pruning With Lower Boundmentioning

confidence: 99%

Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics

Zhang

Yao

et al. 2016

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Detecting near duplicates on the web is challenging due to its volume and variety. Most of the previous studies require the setting of input parameters, making it difficult for them to achieve robustness across various scenarios without careful tuning. Recently, a universal and parameter-free similarity metric, the normalized compression distance or NCD, has been employed effectively in diverse applications. Nevertheless, there are problems preventing NCD from being applied to medium-to-large datasets as it lacks efficiency and tends to get skewed by large object size. To make this parameter-free method feasible on a large corpus of web documents, we propose a new method called SigNCD which measures NCD based on lightweight signatures instead of full documents, leading to improved efficiency and stability. We derive various lower bounds of NCD and propose pruning policies to further reduce computational complexity. We evaluate SigNCD on both English and Chinese datasets and show an increase in 1 score compared with the original NCD method and a significant reduction in runtime. Comparisons with other competitive methods also demonstrate the superiority of our method. Moreover, no parameter tuning is required in SigNCD, except a similarity threshold.

show abstract

“…← the index of boundary object of matching partition of on (21) for all ( < < ) in do (22) if in ℎ then (23) c o n t i n u e (24) end if (25) ← sigNCD( , )…”

Section: P1: Pruning With Lower Boundmentioning

confidence: 99%

Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics

Zhang

Yao

et al. 2016

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…For example, if two instances have over 90% similarity, they can arguably be defined as redundant. Duplicate detection often regards such examples as 'near duplicates' (9) or 'approximate duplicates' (10). In bioinformatics, 'redundancy' is commonly used to describe records with sequence similarity over a certain threshold, such as 90% for CD-HIT (11).…”

Section: Kinds Of Duplicatementioning

confidence: 99%

“…Swiss-Prot is expert curated and reviewed, with software support, whereas TrEMBL is curated automatically without review. Here we list the steps of curation in Swiss-Prot, 9 as previously explained elsewhere (38):…”

Section: Quality Control In Uniprotmentioning

confidence: 99%

Benchmarks for Measurement of Duplicate Detection Methods in Nucleotide Databases

Chen

Zobel

Verspoor

2016

Preprint

View full text Add to dashboard Cite

Duplication of information in databases is a major data quality challenge. The presence of duplicates, implying either redundancy or inconsistency, can have a range of impacts on the quality of analyses that use the data. To provide a sound basis for research on this issue in databases of nucleotide sequences, we have developed new, large-scale validated collections of duplicates, which can be used to test the effectiveness of duplicate detection methods. Previous collections were either designed primarily to test efficiency, or contained only a limited number of duplicates of limited kinds. To date, duplicate detection methods have been evaluated on separate, inconsistent benchmarks, leading to results that cannot be compared and, due to limitations of the benchmarks, of questionable generality.In this study we present three nucleotide sequence database benchmarks, based on information drawn from a range of resources, including information derived from mapping to Swiss-Prot and TrEMBL. Each benchmark has distinct characteristics. We quantify these characteristics and argue for their complementary value in evaluation. The benchmarks collectively contain a vast number of validated biological duplicates; the largest has nearly half a billion duplicate pairs (although this is probably only a tiny fraction of the total that is present). They are also the first benchmarks targeting the primary nucleotide databases. The records include the 21 most heavily studied organisms in molecular biology research. Our quantitative analysis shows that duplicates in the different benchmarks, and in different organisms, have different characteristics. It is thus unreliable to evaluate duplicate detection methods against any single benchmark. For example, the benchmark derived from Swiss-Prot mappings identifies more diverse types of duplicates, showing the importance of expert curation, but is limited to coding sequences. Overall, these benchmarks form a resource that we believe will be of great value for development and evaluation of the duplicate detection methods that are required to help maintain these essential resources.Availability: The benchmark data sets are available at https://bitbucket.org/biodbqual/benchmarks.

show abstract

“…In this work, duplicates are typically defined as records with sequence similarity over a certain threshold, and other factors are not considered. These kinds of duplicates are often referred to as approximate duplicates or near duplicates (37), and are interchangeable with redundancies. For instance, one study located all records with over 90% mutual sequence identity (11).…”

Section: Introductionmentioning

confidence: 99%

Duplicates, redundancies and inconsistencies in the primary nucleotide databases: a descriptive study

Chen¹,

Zobel²,

Verspoor³

2017

Database

View full text Add to dashboard Cite

GenBank, the EMBL European Nucleotide Archive and the DNA DataBank of Japan, known collectively as the International Nucleotide Sequence Database Collaboration or INSDC, are the three most significant nucleotide sequence databases. Their records are derived from laboratory work undertaken by different individuals, by different teams, with a range of technologies and assumptions and over a period of decades. As a consequence, they contain a great many duplicates, redundancies and inconsistencies, but neither the prevalence nor the characteristics of various types of duplicates have been rigorously assessed. Existing duplicate detection methods in bioinformatics only address specific duplicate types, with inconsistent assumptions; and the impact of duplicates in bioinformatics databases has not been carefully assessed, making it difficult to judge the value of such methods. Our goal is to assess the scale, kinds and impact of duplicates in bioinformatics databases, through a retrospective analysis of merged groups in INSDC databases. Our outcomes are threefold: (1) We analyse a benchmark dataset consisting of duplicates manually identified in INSDC—a dataset of 67 888 merged groups with 111 823 duplicate pairs across 21 organisms from INSDC databases – in terms of the prevalence, types and impacts of duplicates. (2) We categorize duplicates at both sequence and annotation level, with supporting quantitative statistics, showing that different organisms have different prevalence of distinct kinds of duplicate. (3) We show that the presence of duplicates has practical impact via a simple case study on duplicates, in terms of GC content and melting temperature. We demonstrate that duplicates not only introduce redundancy, but can lead to inconsistent results for certain tasks. Our findings lead to a better understanding of the problem of duplication in biological databases.Database URL: the merged records are available at https://cloudstor.aarnet.edu.au/plus/index.php/s/Xef2fvsebBEAv9w

show abstract

Detecting near-duplicate documents using sentence-level features and supervised learning

Cited by 19 publications

References 32 publications

Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics

Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics

Benchmarks for Measurement of Duplicate Detection Methods in Nucleotide Databases

Duplicates, redundancies and inconsistencies in the primary nucleotide databases: a descriptive study

Contact Info

Product

Resources

About