MeSHLabeler and DeepMeSH: Recent Progress in Large-Scale MeSH Indexing

Peng, Sijia; Mamitsuka, Hiroshi; Zhu, Shanfeng

doi:10.1007/978-1-4939-8561-6_15

Cited by 4 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides facilitating the retrieval and filtering of articles according to publication type and study design by end-users, the predictive scores should be valuable as features to be incorporated into larger automated machine learning-based efforts. For example, our work is complementary to other efforts to automatically assign MeSH terms to articles [10][11][12][13][14][15][16]; one might extend the Multitagger approach to predicting other types of MeSH terms, or alternatively, predicted PT scores may potentially improve the accuracy of other machine learning approaches [10][11][12][13][14][15][16]. Another potential use of Multi-tagger is to assist in automated ranking of articles for inclusion in systematic reviews [6,24,25].…”

Section: Discussionmentioning

confidence: 98%

“…Huang et al [12] built a MeSH term recommender system that used PubMed Related Articles and a learning-to-rank approach. More recently, a combined MeSHLabeler [13] and DeepMeSH system used learning to rank along with Deep Learning semantic representation to achieve the highest binary prediction scores in the BioASQ2 and BioASQ3 challenges [14,15]. FullMeSH has enhanced these approaches by applying section-based convolutional neural networks to full article text [16].…”

Section: Background and Significancementioning

confidence: 99%

See 1 more Smart Citation

Fifty Ways to Tag your Pubtypes: Multi-Tagger, a Set of Probabilistic Publication Type and Study Design Taggers to Support Biomedical Indexing and Evidence-Based Medicine

Cohen

Schneider

et al. 2021

Preprint

View full text Add to dashboard Cite

Objective: Indexing articles according to publication types (PTs) and study designs can be a great aid to filtering literature for information retrieval, especially for evidence syntheses. In this study, 50 automated machine learning based probabilistic PT and study design taggers were built and applied to all articles in PubMed. Materials and Methods: PubMed article metadata from 1987-2014 were used as training data, with 2015 used for recalibration. The set of articles indexed with a particular study design MeSH term or PT tag was used as positive training sets. For each PT, the rest of the literature from the same time period was used as its negative training set. Multiple features based on each article title, abstract and metadata were used in training the models. Taggers were evaluated on PubMed articles from 2016 and 2019. A manual analysis was also performed. Results: Of the 50 predictive models that we created, 44 of these achieved an AUC of ~0.90 or greater, with many having performance above 0.95. Of the clinically related study designs, the best performing was SYSTEMATIC_REVIEW with an AUC of 0.998; the lowest performing was RANDOM_ALLOCATION, with an AUC of 0.823. Discussion: This work demonstrates that is feasible to build a large set of probabilistic publication type and study design taggers with high accuracy and ranking performance. Automated tagging permits users to identify qualifying articles as soon as they are published, and allows consistent criteria to be applied across different bibliographic databases. Probabilistic predictive scores are more flexible than binary yes/no predictions, since thresholds can be tailored for specific uses such as high recall literature search, user-adjustable retrieval size, and quality improvement of manually annotated databases. Conclusion: The PT predictive probability scores for all PubMed articles are freely downloadable at http://arrowsmith.psych.uic.edu/evidence_based_medicine/mt_download.html for incorporation into user tools and workflows. Users can also perform PubMed queries at our Anne O'Tate value-added PubMed search engine http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/AnneOTate.cgi and filter retrieved articles according to both NLM-annotated and model-predicted publication types and study designs.

show abstract

Section: Discussionmentioning

confidence: 98%

Section: Background and Significancementioning

confidence: 99%

Fifty Ways to Tag your Pubtypes: Multi-Tagger, a Set of Probabilistic Publication Type and Study Design Taggers to Support Biomedical Indexing and Evidence-Based Medicine

Cohen

Schneider

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Online/offline evaluation: BioASQ comprises a typical offline evaluation with predicting held out information. Win-ning systems from 2014 to 2017 have been made available online, comprising a user interface [21]. There, articles can be further manually labeled, allowing submissions by participants to be evaluated if new annotations become available.…”

Section: Bioasqmentioning

confidence: 99%

Evaluation Infrastructures for Academic Shared Tasks

et al. 2020

View full text Add to dashboard Cite

Academic search systems aid users in finding information covering specific topics of scientific interest and have evolved from early catalog-based library systems to modern web-scale systems. However, evaluating the performance of the underlying retrieval approaches remains a challenge. An increasing amount of requirements for producing accurate retrieval results have to be considered, e.g., close integration of the system's users. Due to these requirements, small to mid-size academic search systems cannot evaluate their retrieval system in-house. Evaluation infrastructures for shared tasks alleviate this situation. They allow researchers to experiment with retrieval approaches in specific search and recommendation scenarios without building their own infrastructure. In this paper, we elaborate on the benefits and shortcomings of four state-of-the-art evaluation infrastructures on search and recommendation tasks concerning the following requirements: support for online and offline evaluations, domain specificity of shared tasks, and reproducibility of experiments and results. In addition, we introduce an evaluation infrastructure concept design aiming at reducing the shortcomings in shared tasks for search and recommender systems.

show abstract

“…Machine learning algorithms can infer from the existing documents in a category, which textual features make a document likely to belong to a certain category. New documents can then automatically be classified based on their text (Peng et al, 2018), There are disadvantages to using expert classifications like MeSH, when compared to citations and VCP respectively. First, expert classifications are often one-dimensional, i.e.…”

Section: Related Workmentioning

confidence: 99%

Virtual Citation Proximity (VCP): Learning a Hypothetical In-Text Citation-Proximity Metric For Uncited Documents

Molloy¹,

Beel²,

Aizawa³

2020

Preprint

View full text Add to dashboard Cite

The relatedness of research articles, patents, court rulings, webpages, and other document types is often calculated with citation or hyperlink-based approaches like co-citation (proximity) analysis. The main limitation of citation-based approaches is that they cannot be used for documents that receive little or no citations. We propose Virtual Citation Proximity (VCP), a Siamese Neural Network architecture, which combines the advantages of co-citation proximity analysis (diverse notions of relatedness / high recommendation performance), with the advantage of content-based filtering (high coverage). VCP is trained on a corpus of documents with textual features, and with real citation proximity as ground truth. VCP then predicts for any two documents, based on their title and abstract, in what proximity the two documents would be co-cited, if they were indeed co-cited. The prediction can be used in the same way as real citation proximity to calculate document relatedness, even for uncited documents. In our evaluation with 2 million co-citations from Wikipedia articles, VCP achieves an MAE of 0.0055, i.e. an improvement of 20% over the baseline, though the learning curve suggests that more work is needed.

show abstract

MeSHLabeler and DeepMeSH: Recent Progress in Large-Scale MeSH Indexing

Cited by 4 publications

References 14 publications

Fifty Ways to Tag your Pubtypes: Multi-Tagger, a Set of Probabilistic Publication Type and Study Design Taggers to Support Biomedical Indexing and Evidence-Based Medicine

Fifty Ways to Tag your Pubtypes: Multi-Tagger, a Set of Probabilistic Publication Type and Study Design Taggers to Support Biomedical Indexing and Evidence-Based Medicine

Evaluation Infrastructures for Academic Shared Tasks

Virtual Citation Proximity (VCP): Learning a Hypothetical In-Text Citation-Proximity Metric For Uncited Documents

Contact Info

Product

Resources

About