Novel Approach to Personalized Physician Recommendations Using Semantic Features and Response Metrics: Model Evaluation Study

Abstract

Background The rapid growth of web-based medical services has highlighted the significance of smart triage systems in helping patients find the most appropriate physicians. However, traditional triage methods often rely on department recommendations and are insufficient to accurately match patients’ textual questions with physicians’ specialties. Therefore, there is an urgent need to develop algorithms for recommending physicians. Objective This study aims to develop and validate a patient-physician hybrid recommendation (PPHR) model with response metrics for better triage performance. Methods A total of 646,383 web-based medical consultation records from the Internet Hospital of the First Affiliated Hospital of Xiamen University were collected. Semantic features representing patients and physicians were developed to identify the set of most similar questions and semantically expand the pool of recommended physician candidates, respectively. The physicians’ response rate feature was designed to improve candidate rankings. These 3 characteristics combine to create the PPHR model. Overall, 5 physicians participated in the evaluation of the efficiency of the PPHR model through multiple metrics and questionnaires as well as the performance of Sentence Bidirectional Encoder Representations from Transformers and Doc2Vec in text embedding. Results The PPHR model reaches the best recommendation performance when the number of recommended physicians is 14. At this point, the model has an F1-score of 76.25%, a proportion of high-quality services of 41.05%, and a rating of 3.90. After removing physicians’ characteristics and response rates from the PPHR model, the F1-score decreased by 12.05%, the proportion of high-quality services fell by 10.87%, the average hit ratio dropped by 1.06%, and the rating declined by 11.43%. According to whether those 5 physicians were recommended by the PPHR model, Sentence Bidirectional Encoder Representations from Transformers achieved an average hit ratio of 88.6%, while Doc2Vec achieved an average hit ratio of 53.4%. Conclusions The PPHR model uses semantic features and response metrics to enable patients to accurately find the physician who best suits their needs.