Duplicates, redundancies and inconsistencies in the primary nucleotide databases: a descriptive study

Abstract

GenBank, the EMBL European Nucleotide Archive and the DNA DataBank of Japan, known collectively as the International Nucleotide Sequence Database Collaboration or INSDC, are the three most significant nucleotide sequence databases. Their records are derived from laboratory work undertaken by different individuals, by different teams, with a range of technologies and assumptions and over a period of decades. As a consequence, they contain a great many duplicates, redundancies and inconsistencies, but neither the prevalence nor the characteristics of various types of duplicates have been rigorously assessed. Existing duplicate detection methods in bioinformatics only address specific duplicate types, with inconsistent assumptions; and the impact of duplicates in bioinformatics databases has not been carefully assessed, making it difficult to judge the value of such methods. Our goal is to assess the scale, kinds and impact of duplicates in bioinformatics databases, through a retrospective analysis of merged groups in INSDC databases. Our outcomes are threefold: (1) We analyse a benchmark dataset consisting of duplicates manually identified in INSDC—a dataset of 67 888 merged groups with 111 823 duplicate pairs across 21 organisms from INSDC databases – in terms of the prevalence, types and impacts of duplicates. (2) We categorize duplicates at both sequence and annotation level, with supporting quantitative statistics, showing that different organisms have different prevalence of distinct kinds of duplicate. (3) We show that the presence of duplicates has practical impact via a simple case study on duplicates, in terms of GC content and melting temperature. We demonstrate that duplicates not only introduce redundancy, but can lead to inconsistent results for certain tasks. Our findings lead to a better understanding of the problem of duplication in biological databases.Database URL: the merged records are available at https://cloudstor.aarnet.edu.au/plus/index.php/s/Xef2fvsebBEAv9w

Integrating eDNA metabarcoding and simultaneous underwater visual surveys to describe complex fish communities in a marine biodiversity hotspot

“…However, such reference databases are often missing for local faunas (Juhel et al, 2020) and their completeness varies for different regions. Thus, the impact of the taxonomic gaps in the reference database on the final taxonomic assignment and on the ecological inferences derived from eDNA data (Chen et al, 2017), and the extent of improvement gained with the development and use of a local database, are relevant aspects needing evaluation in eDNA studies.…”

Ransomware Detection System for Android Applications

“…This could be a result of malware polymorphism technique which is used by attackers where they made some changes to the application in order to derive different hash in order to evade detection by signature based anti-viruses [59]. These apps eventually could have the same feature set for API calls and therefore need to be removed from dataset to avoid duplication [60]. One of the main contributions of this work is offering a clean ransomware dataset without duplicate apps.…”

Search Effectiveness in Nonredundant Sequence Databases: Assessments and Solutions

“…A possible solution is to remove duplicate records. However, the notion of duplication is context-dependent; removal of records that might be regarded as duplicates for one task may be harmful to other tasks (Chen et al, 2017b).…”

“…Such dramatic increases in numbers of sequence records lead to duplication. Duplicate records can be broadly categorized as of two kinds: entity duplicates, which are records belonging to same entities (Chen et al, 2016b(Chen et al, , 2017bYonchev et al, 2018), such as when the same gene records are submitted to the same database (Chen et al, 2017b); and near-duplicates or redundant records (Suzek et al, 2015;Mirdita et al, 2016;Chen et al, 2018), where records share some specified percentage X% similarity defined by users. For example, the Uniclust protein database defines redundant records at sequence similarity thresholds of 30%, 50%, and 90% for different purposes (Mirdita et al, 2016).…”