A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study

Fiol, Guilherme Del; Michelson, Matthew; Iorio, Alfonso; Cotoi, Chris; Haynes, R. Brian

doi:10.2196/10281

Cited by 49 publications

(40 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Machine learning will affect how people are paid, as tasks become automated away, but that is challenging to forecast – therefore, we will focus our analysis on the hours it saves (E hrs ) rather than the cost-per-person (C person ). In a simple argument, one can imagine researchers leveraging tools to automatically screen papers [25] for quality, doing in seconds what used to take hours. As another example, following an approach similar to Ref.…”

Section: Discussionmentioning

confidence: 99%

The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials

Michelson¹,

Reuter

2019

Contemporary Clinical Trials Communications

Self Cite

View full text Add to dashboard Cite

Background More than 90% of clinical-trial compounds fail to demonstrate sufficient efficacy and safety. To help alleviate this issue, systematic literature review and meta-analysis (SLR), which synthesize current evidence for a research question, can be applied to preclinical evidence to identify the most promising therapeutics. However, these methods remain time-consuming and labor-intensive. Here, we introduce an economic formula to estimate the expense of SLR for academic institutions and pharmaceutical companies. Methods We estimate the manual effort involved in SLR by quantifying the amount of labor required and the total associated labor cost. We begin with an empirical estimation and derive a formula that quantifies and describes the cost. Results The formula estimated that each SLR costs approximately $141,194.80. We found that on average, the ten largest pharmaceutical companies publish 118.71 and the ten major academic institutions publish 132.16 SLRs per year. On average, the total cost of all SLRs per year to each academic institution amounts to $18,660,304.77 and for each pharmaceutical company is $16,761,234.71. Discussion It appears that SLR is an important, but costly mechanisms to assess the totality of evidence. Conclusions With the increase in the number of publications, the significant time and cost of SLR may pose a barrier to their consistent application to assess the promise of clinical trials thoroughly. We call on investigators and developers to develop automated solutions to help with the assessment of preclinical evidence particularly. The formula we introduce provides a cost baseline against which the efficiency of automation can be measured.

show abstract

Section: Discussionmentioning

confidence: 99%

The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials

Michelson¹,

Reuter

2019

Contemporary Clinical Trials Communications

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although the comparison is thematic as the overall objective of the mentioned studies is the same, i.e., the identification of high-impact studies in the biomedical literature, it provides a perspective on the superiority of the deep learning model over shallow machine learning methods. From a method perspective, the study conducted by Del Fiol et al [12] is closer to our study as they have also used a deep learning method in their experiments. The difference was that they used deep learning based on CNN, while the proposed method used MLP-a deep learning based on multi-layer feed-forward neural networks.…”

Section: Comparison With Prior Workmentioning

confidence: 65%

“…Initially invented for computer vision, CNN models have subsequently been shown to be useful for NLP and have achieved excellent results in semantic parsing [11]. A CNN-based deep learning model [12] [7]. Afzal et al built compared different machine learning algorithms and learned to choose a support vector machine (SVM) based model due to its higher performance [15].…”

Section: Use Of Machine Learning and Deep Learning For Biomedical Litmentioning

confidence: 99%

Deep Learning Based Biomedical Literature Classification Using Criteria of Scientific Rigor

et al. 2020

View full text Add to dashboard Cite

A major blockade to support the evidence-based clinical decision-making is accurately and efficiently recognizing appropriate and scientifically rigorous studies in the biomedical literature. We trained a multi-layer perceptron (MLP) model on a dataset with two textual features, title and abstract. The dataset consisting of 7958 PubMed citations classified in two classes: scientific rigor and non-rigor, is used to train the proposed model. We compare our model with other promising machine learning models such as Support Vector Machine (SVM), Decision Tree, Random Forest, and Gradient Boosted Tree (GBT) approaches. Based on the higher cumulative score, deep learning was chosen and was tested on test datasets obtained by running a set of domain-specific queries. On the training dataset, the proposed deep learning model obtained significantly higher accuracy and AUC of 97.3% and 0.993, respectively, than the competitors, but was slightly lower in the recall of 95.1% as compared to GBT. The trained model sustained the performance of testing datasets. Unlike previous approaches, the proposed model does not require a human expert to create fresh annotated data; instead, we used studies cited in Cochrane reviews as a surrogate for quality studies in a clinical topic. We learn that deep learning methods are beneficial to use for biomedical literature classification. Not only do such methods minimize the workload in feature engineering, but they also show better performance on large and noisy data.

show abstract

“…Future studies could investigate the utility of free-text features extracted through text mining techniques from the article title and abstract. In addition, a larger gold standard would enable researchers to investigate the effect of other advanced classification models such as deep learning [71] and graphical models such as conditional random fields and hidden Markov models [72–74]. Alternatively, transfer learning methods could be investigated as a part of a deep learning approach.…”

Section: Discussionmentioning

confidence: 99%

Automatic identification of recent high impact clinical articles in PubMed to support clinical decision making using time-agnostic features

Bian

Abdelrahman

Shi

et al. 2019

Journal of Biomedical Informatics

Self Cite

View full text Add to dashboard Cite

Objectives: Finding recent clinical studies that warrant changes in clinical practice (“high impact” clinical studies) in a timely manner is very challenging. We investigated a machine learning approach to find recent studies with high clinical impact to support clinical decision making and literature surveillance. Methods: To identify recent studies, we developed our classification model using time-agnostic features that are available as soon as an article is indexed in PubMed®, such as journal impact factor, author count, and study sample size. Using a gold standard of 541 high impact treatment studies referenced in 11 disease management guidelines, we tested the following null hypotheses: 1) the high impact classifier with time-agnostic features (HI-TA) performs equivalently to PubMed’s Best Match sort and a MeSH-based Naïve Bayes classifier; and 2) HI-TA performs equivalently to the high impact classifier with both time-agnostic and time-sensitive features (HI-TS) enabled in a previous study. The primary outcome for both hypotheses was mean top 20 precision. Results: The differences in mean top 20 precision between HI-TA and three baselines (PubMed’s Best Match, a MeSH-based Naïve Bayes classifier, and HI-TS) were not statistically significant (12% vs. 3%, p=0.101; 12% vs. 11%, p=0.720; 12% vs. 25%, p=0.094, respectively). Recall of HI-TA was low (7%). Conclusion: HI-TA had equivalent performance to state-of-the-art approaches that depend on time-sensitive features. With the advantage of relying only on time-agnostic features, the proposed approach can be used as an adjunct to help clinicians identify recent high impact clinical studies to support clinical decision-making. However, low recall limits the use of HI-TA for literature surveillance.

show abstract

A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study

Cited by 49 publications

References 23 publications

The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials

The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials

Deep Learning Based Biomedical Literature Classification Using Criteria of Scientific Rigor

Automatic identification of recent high impact clinical articles in PubMed to support clinical decision making using time-agnostic features

Contact Info

Product

Resources

About