Sensitivity Analysis of a BERT-based scholarly recommendation system

Jie, Zhu; Wu, Hulin; Yaseen, Ashraf

doi:10.32473/flairs.v35i.130595

Cited by 4 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regarding BERT-based systems, Zhu et al [ 15 ] developed a BERT-based recommender to recommend public available papers to researchers. Later Zhu et al [ 17 ] performed a sensitivity analysis on the training class imbalance on BERT-based dataset recommendation system. Bilal et al [ 18 ] used BERT classifier along with three bag-of-words based classifiers to recommend helpful online reviews on Yelp datasets.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A novel NIH research grant recommender using BERT

Jie

Patra

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

Research grants are important for researchers to sustain a good position in academia. There are many grant opportunities available from different funding agencies. However, finding relevant grant announcements is challenging and time-consuming for researchers. To resolve the problem, we proposed a grant announcements recommendation system for the National Institute of Health (NIH) grants using researchers’ publications. We formulated the recommendation as a classification problem and proposed a recommender using state-of-the-art deep learning techniques: i.e. Bidirectional Encoder Representations from Transformers (BERT), to capture intrinsic, non-linear relationship between researchers’ publications and grants announcements. Internal and external evaluations were conducted to assess the system’s usefulness. During internal evaluations, the grant citations were used to establish grant-publication ground truth, and results were evaluated against Recall@k, Precision@k, Mean reciprocal rank (MRR) and Area under the Receiver Operating Characteristic curve (ROC-AUC). During external evaluations, researchers’ publications were clustered using Dirichlet Process Mixture Model (DPMM), recommended grants by our model were then aggregated per cluster through Recency Weight, and finally researchers were invited to provide ratings to recommendations to calculate Precision@k. For comparison, baseline recommenders using Okapi Best Matching (BM25), Term-Frequency Inverse Document Frequency (TF-IDF), doc2vec, and Naïve Bayes (NB) were also developed. Both internal and external evaluations (all metrics) revealed favorable performances of our proposed BERT-based recommender.

show abstract

Section: Related Workmentioning

confidence: 99%

“…In addition to our work in [13][14][15][16][17], we were able to locate studies that focus on other academic recommendations and BERT-based recommenders that are related to our research, Patra et al [16] experimented with information retrieval paradigms (BM25, TF-IDF, etc.) for Gene Expression Omnibus data recommendation to researchers.…”

Section: Related Workmentioning

confidence: 99%

A novel NIH research grant recommender using BERT

Jie

Patra

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Furthermore, Zhu et al [11] developed a recommender of scholarly papers to researchers, based on publicly available database using BERT-based model [11]. Based on BERT-based datasets recommendation system, this group also performed a sensitivity analysis on the training class imbalance [12].…”

Section: Related Workmentioning

confidence: 99%

Scholarly recommendation system for NIH funded grants based on biomedical word embedding models

Zhang,

Yaseen,

2023

Preprint

Self Cite

View full text Add to dashboard Cite

Objective Research grants, which are available from several sources, are essential for scholars to sustain a good standing in academia. Although securing grant funds for research is very competitive, being able to locate and find previously funded grants and projects that are relevant to researchers’ interests would be very helpful. In this work, we developed a funded-grants/projects recommendation system for the National Institute of Health (NIH) grants. Methods Our system aims to recommend funded grants to researchers based on their publications or input keywords. By extracting summary information from funded grants and their associated applications, we employed two embedding models for biomedical words and sentences (Biowordvec and Biosentvec), and compare multiple recommendation methods to recommend the most relevant funded grants for researchers’ input. Results Compared to a baseline method, the recommendation system based on biomedical word embedding models provided higher performance. The system also received an average rate of 3.53 out of 5, based on the relevancy evaluation results from biomedical researchers. Conclusion Both internal and external evaluation results prove the effectiveness of our recommendation system. The system would be helpful for biomedical researchers to locate and find previously funded grants related to their interests.

show abstract

“…Depending on different recommendation tasks, for the grants and datasets (since the grants and datasets followed the same structures, we only presented the results of the grants recommender for conciseness of the content) vs. collaborators, we utilized different sets of data, similar to the previous experiments detailed in [27][28][29][30].…”

Section: Datamentioning

confidence: 99%

“…The experiments were carried out on our current version of VRA [27][28][29][30] where it has different components for recommendations. Specifically, the underlying modelling component for the datasets and grant recommendations is a BERT-based model, and the one for collaborators is a temporal graph network (TGN), see Figure 3.…”

Section: Our Virtual Research Assistant (Vra) Architecturementioning

confidence: 99%

Incorporating Uncertainty Quantification for the Performance Improvement of Academic Recommenders

2023

Self Cite

View full text Add to dashboard Cite

Deep learning is widely used in many real-life applications. Despite their remarkable performance accuracies, deep learning networks are often poorly calibrated, which could be harmful in risk-sensitive scenarios. Uncertainty quantification offers a way to evaluate the reliability and trustworthiness of deep-learning-based model predictions. In this work, we introduced uncertainty quantification to our virtual research assistant recommender platform through both Monte Carlo dropout ensemble techniques. We also proposed a new formula to incorporate the uncertainty estimates into our recommendation models. The experiments were carried out on two different components of the recommender platform (i.e., a BERT-based grant recommender and a temporal graph network (TGN)-based collaborator recommender) using real-life datasets. The recommendation results were compared in terms of both recommender metrics (AUC, AP, etc.) and the calibration/reliability metric (ECE). With uncertainty quantification, we were able to better understand the behavior of our regular recommender outputs; while our BERT-based grant recommender tends to be overconfident with its outputs, our TGN-based collaborator recommender tends to be underconfident in producing matching probabilities. Initial case studies also showed that our proposed model with uncertainty quantification adjustment from ensemble gave the best-calibrated results together with the desirable recommender performance.

show abstract

Sensitivity Analysis of a BERT-based scholarly recommendation system

Cited by 4 publications

References 0 publications

A novel NIH research grant recommender using BERT

A novel NIH research grant recommender using BERT

Scholarly recommendation system for NIH funded grants based on biomedical word embedding models

Incorporating Uncertainty Quantification for the Performance Improvement of Academic Recommenders

Contact Info

Product

Resources

About