Why was this cited? Explainable machine learning applied to COVID-19 research literature

Abstract

Multiple studies have investigated bibliometric factors predictive of the citation count a research article will receive. In this article, we go beyond bibliometric data by using a range of machine learning techniques to find patterns predictive of citation count using both article content and available metadata. As the input collection, we use the CORD-19 corpus containing research articles—mostly from biology and medicine—applicable to the COVID-19 crisis. Our study employs a combination of state-of-the-art machine learning techniques for text understanding, including embeddings-based language model BERT, several systems for detection and semantic expansion of entities: ConceptNet, Pubtator and ScispaCy. To interpret the resulting models, we use several explanation algorithms: random forest feature importance, LIME, and Shapley values. We compare the performance and comprehensibility of models obtained by “black-box” machine learning algorithms (neural networks and random forests) with models built with rule learning (CORELS, CBA), which are intrinsically explainable. Multiple rules were discovered, which referred to biomedical entities of potential interest. Of the rules with the highest lift measure, several rules pointed to dipeptidyl peptidase4 (DPP4), a known MERS-CoV receptor and a critical determinant of camel to human transmission of the camel coronavirus (MERS-CoV). Some other interesting patterns related to the type of animal investigated were found. Articles referring to bats and camels tend to draw citations, while articles referring to most other animal species related to coronavirus are lowly cited. Bat coronavirus is the only other virus from a non-human species in the betaB clade along with the SARS-CoV and SARS-CoV-2 viruses. MERS-CoV is in a sister betaC clade, also close to human SARS coronaviruses. Thus both species linked to high citation counts harbor coronaviruses which are more phylogenetically similar to human SARS viruses. On the other hand, feline (FIPV, FCOV) and canine coronaviruses (CCOV) are in the alpha coronavirus clade and more distant from the betaB clade with human SARS viruses. Other results include detection of apparent citation bias favouring authors with western sounding names. Equal performance of TF-IDF weights and binary word incidence matrix was observed, with the latter resulting in better interpretability. The best predictive performance was obtained with a “black-box” method—neural network. The rule-based models led to most insights, especially when coupled with text representation using semantic entity detection methods. Follow-up work should focus on the analysis of citation patterns in the context of phylogenetic trees, as well on patterns referring to DPP4, which is currently considered as a SARS-Cov-2 therapeutic target.

Evaluation of editors’ abilities to predict the citation potential of research manuscripts submitted to The BMJ : a cohort study

“…Computational modelling and machine learning are showing promise at citation prediction. 20 21 While we might not be as good as tomorrow’s machine learning, we are consoled that previously we demonstrated our editorial decision making could not be replaced by a non-qualified but experienced administrator. 28 …”

“…19 Recently machine learning techniques have been used to find patterns that predict citation count with some success. 20 21 …”

“…Few studies have successfully used information available before publication for this purpose 19. Recently machine learning techniques have been used to find patterns that predict citation count with some success 2021…”

Multi-omics and immune cells’ profiling of COVID-19 patients for ICU admission prediction: in silico analysis and an integrated machine learning-based approach in the framework of Predictive, Preventive, and Personalized Medicine

“…Although some researchers suggested that the random forest (RF) model had a better prediction effect than the logistic regression model for predicting COVID-19 severity [ 55 ], a recent study noted that logistic regression is not always inferior to other ML algorithms [ 56 ]. It has better explanatory power than RF and the previously mentioned algorithms because they are a kind of “black-box” ML algorithms [ 57 ]. The Boruta algorithm was especially suitable for feature selection, although it is also a tree-based ML algorithm, which can be applied easily and modeled nonlinear relations well without much tuning [ 58 ].…”

Impact of COVID-19 Research: A Study on Predicting Influential Scholarly Documents and Utilizing a Domain-Independent Knowledge Graph

“…However, these methods require a balanced dataset to perform well in per-class prediction accuracy and overly coarse classifications for domain-independent corpora. Recent research has shown that these methods perform well for domain-dependent tasks, particularly in classifying bio-medical scholarly documents [9]. Deep neural networks [10], such as convolutional and recurrent networks [11], have also been applied to classification tasks.…”