Comparison of automated and human assignment of MeSH terms on publicly-available molecular datasets

Abstract

Publicly available molecular datasets can be used for independent verification or investigative repurposing, but depends on the presence, consistency and quality of descriptive annotations. Annotation and indexing of molecular datasets using well-defined controlled vocabularies or ontologies enables accurate and systematic data discovery, yet the majority of molecular datasets available through public data repositories lack such annotations. A number of automated annotation methods have been developed; however few systematic evaluations of the quality of annotations supplied by application of these methods have been performed using annotations from standing public data repositories. Here, we compared manually-assigned Medical Subject Heading (MeSH) annotations associated with experiments by data submitters in the PRoteomics IDEntification (PRIDE) proteomics data repository to automated MeSH annotations derived through the National Center for Biomedical Ontology Annotator and National Library of Medicine MetaMap programs. These programs were applied to free-text annotations for experiments in PRIDE. As many submitted datasets were referenced in publications, we used the manually curated MeSH annotations of those linked publications in MEDLINE as “gold standard”. Annotator and MetaMap exhibited recall performance 3-fold greater than that of the manual annotations. We connected PRIDE experiments in a network topology according to shared MeSH annotations and found 373 distinct clusters, many of which were found to be biologically coherent by network analysis. The results of this study suggest that both Annotator and MetaMap are capable of annotating public molecular datasets with a quality comparable, and often exceeding, that of the actual data submitters, highlighting a continuous need to improve and apply automated methods to molecular datasets in public data repositories to maximize their value and utility.

Identifying medical terms in patient-authored text: a crowdsourcing-based approach

“…Ruau et al evaluated automated MeSH annotations on PRoteomics IDEntification (PRIDE) experiment descriptions against manually assigned MeSH annotations. MetaMap achieved precision and recall scores of 15.66% and 79.44%, while OBA achieved 20.97% and 79.48%, respectively 18. Pratt and Yetisgen-Yildiz compare MetaMap's annotations to human annotations on 60 MEDLINE titles: they found that MetaMap achieved exact precision and recall scores of 27.7% and 52.8%, and partial precision and recall scores of 55.2% and 93.3%, respectively.…”

Informatics and Computational Methods in Natural Product Drug Discovery: A Review and Perspectives

“…MeSH can be used to aggregate PubMed articles describing certain types of NPs, and can be refined using additional terms (e.g., “”) or qualifiers (e.g., “”). MeSH terms can link journal entities to structured external databases by either using cross-mappings [including via the NLM's Unified Medical Language System (UMLS)] or annotations in external databases directly to MeSH terms (Ruau et al, 2011). MeSH terms have been used to summarize components of plant genomes (Beissinger and Morota, 2017), demonstrating potential paths forward in discovering novel NPs (rather than using the terms to gather knowledge about known NPs).…”

A conceptualization of a nuclear or radiological emergency

“…Although originally conceived fifty years ago as a terminology from which to retrieve terms for (manually) indexing scientific publications (Rogers, 1963) MeSH has seen extensive use as a lexical resource by the language technology community. The most investigated task is the automatic annotation of medical articles, where a variety of approaches have been proposed that are both fully automated (Stevenson et al, 2012) and based on assisted term selection (Huang et al, 2011;Ruau et al, 2011).…”