Evaluation of the portability of computable phenotypes with natural language processing in the eMERGE network

Pacheco, Jennifer A.; Rasmussen, Luke V.; Wiley, Ken; Person, Thomas N.; Cronkite, David; Sohn, Sunghwan; Murphy, Shawn N.; Gundelach, Justin H.; Gainer, Vivian S.; Castro, Víctor M.; Liu, Cong; Mentch, Frank; Lingren, Todd; Sundaresan, Agnes S.; Eickelberg, Garrett; Willis, Valerie; Furmanchuk, Al’ona; Carrell, David; Deng, Yu; Walton, Nephi; Satterfield, Benjamin A.; Kullo, Iftikhar J.; Dikilitas, Ozan; Smith, Joshua; Peterson, Josh F.; Shang, Ning; Kiryluk, Krzysztof; Ni, Yizhao; Li, Yikuan; Nadkarni, Girish N.; Rosenthal, Elisabeth; Walunas, Theresa L.; Williams, Marc S.; Karlson, Elizabeth W.; Linder, Jodell E.; Luo, Yuan; Weng, Chunhua; Wei, Wei‐Qi

doi:10.1038/s41598-023-27481-y

Cited by 7 publications

(4 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The research methodology adopted in this study builds upon the integration of text mining and deep learning techniques for the automated discovery of biomedical materials in healthcare applications. The methodology encompasses two key components: (1) The exploration of research questions related to the domain, as exemplified by the graphical representation using bar charts, line charts, and scatter plots; and (2) The evaluation of the performance of different models through the visualization of precision, recall, and F1-score (Pacheco, J. A., et al, 2023).…”

Section: Methodsmentioning

confidence: 99%

Empowering healthcare with NLP-driven deep learning unveiling biomedical materials through text mining

Dileep Pulugu,

Shaik K. Ahamed,

Senthil Vadivu

et al. 2024

The Scientific Temper

View full text Add to dashboard Cite

This study presents a comprehensive approach to automated biomedical materials discovery in healthcare applications by integrating text mining and deep learning techniques. The research methodology encompasses two main components: exploration of key research questions through graphical representation and evaluation of model performance using precision, recall, and F1-score metrics. The identification of pertinent research questions is visualized using bar charts, offering insights into the distribution of studies across domains such as data harmonization, heterogeneity, industrial textual data, and sequential data performance. The precision comparison chart highlights the strengths and weaknesses of different models, with model 1 demonstrating notable precision. The recall comparison chart emphasizes model 2’s outstanding performance in capturing relevant information, while the f1-score comparison chart showcases the balanced metrics of model 2 and 4. These visual analyses contribute to a nuanced understanding of the research landscape and guide the development of the proposed NLP-ML pipeline. The study’s findings underscore the significance of addressing data harmonization challenges and extracting insights from industrial textual data in advancing biomedical materials discovery. Overall, this research amalgamates exploratory data analysis and quantitative model evaluation to contribute to the evolving field of text mining and deep learning applications in biomedical material discovery for healthcare applications.

show abstract

Section: Methodsmentioning

confidence: 99%

Empowering healthcare with NLP-driven deep learning unveiling biomedical materials through text mining

Dileep Pulugu,

Shaik K. Ahamed,

Senthil Vadivu

et al. 2024

The Scientific Temper

View full text Add to dashboard Cite

show abstract

“…Detailed guidelines for expert evaluation can be found in Supplemental Table 2. Experts assigned categorical scores ("Good [3]," "Medium [2]," or "Poor [1]") for each axis based on predefined criteria, providing justifications accordingly. Interrater reliability was assessed using the weighted Cohen's Kappa score 28 .…”

Section: Evaluating the Quality Of Llm-generated Phenotyping Algorithmsmentioning

confidence: 99%

“…Electronic health record (EHR) phenotyping, which involves creating algorithms to identify and correctly classify a patient's observable characteristics by integrating complex clinical data, has become pivotal in observational health research 1 . Developing EHR phenotypes is an intricate and labor-intensive process that demands extensive expertise in both the clinical and informatics domains 2,3 . While phenotyping includes the identification of individuals with specific characteristics, it also necessitates the selection of suitable controls for meaningful comparisons with the identified cases 4 .…”

Section: Introductionmentioning

confidence: 99%

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Yan,

Ong,

Grabowska

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

ObjectivesPhenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm typically demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating drafts of high-quality algorithms.Materials and MethodsWe prompted four LLMs—ChatGPT-4, ChatGPT-3.5, Claude 2, and Bard—in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model for three clinical phenotypes (i.e., type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms from each LLM and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.ResultsChatGPT-4 and ChatGPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although ChatGPT-4 and ChatGPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).ConclusionBoth ChatGPT versions 3.5 and 4 demonstrate the capability to enhance EHR phenotyping efficiency by drafting algorithms of reasonable quality. However, the optimal performance of these algorithms necessitates the involvement of domain experts.

show abstract

“…легчения идентификации фенотипа применяются различные стандарты представления данных (часть из них описаны ниже): SNOMED-CT (Systemized Nomenclature of Medicine -Clinical Terms, систематизированная номенклатура в медицине -клинические термины), CPT (Current Procedural Terminology, современная терминология процедур), LOINC (Logical Observation Identifiers Names and Codes, имена и коды идентификаторов логических наблюдений), FHIR (Fast Health Interoperability Resources, ресурсы быстрого взаимодействия в сфере здравоохранения) и др., а также системы обработки естественного языка, например, "MedLEE", "MetaMap", "KnowledgeMap" и "cTAKES" [22][23][24]. Подобный подход позволяет успешно проводить анализ данных как в рамках одного клинического центра, так и в многоцентровых исследованиях, даже при наличии различий в клинической практике между разными учреждениями [20,24].…”

Section: ресурсы: данные эмк и эиб кодификаторыunclassified

Specifics of creating clinical abstract of biospecimens

Kopylova,

Ershova,

Borisova

et al. 2023

Cardiovasc Ther Prev

View full text Add to dashboard Cite

One technology that helps overcome the problem of low research reproducibility is biobanking, which involves maintaining strict quality standards at all stages. In addition to data on the biosample (detailed documentation on sampling, transportation, preparation and subsequent storage), one of the key points is the availability of information about the donor (patient). The aim of this article was to analyze creating clinical abstract of biospecimens, combining data from various biobanks and assessing the possibilities of electronic medical records and other modern technologies for this. The search for publications was carried out in the PUBMED, eLIBRARY.RU, RSCI databases. One approach to creating a clinical description is the targeted collection of information by a specially trained employee. Primary information is most often taken from the individual records of the study participant, which are developed and approved when planning work. An alternative method is the use of electronic medical records and other documents that collect information during the assessment and treatment of patients. There are also mixed types of clinical data collection, a prime example of which is the UK Biobank. Completeness, structure, and standardization are essential characteristics of clinical description associated with biospecimens. Various standards are currently being developed to unify clinical description, making biobanks and collections more available to external researchers and organizations, which is necessary for collaboration and more efficient use of stored biospecimens. Harmonization of clinical description methodology between different biobanks open up broad boundaries for large- scale research within personalized and translational medicine.

show abstract

Evaluation of the portability of computable phenotypes with natural language processing in the eMERGE network

Cited by 7 publications

References 49 publications

Empowering healthcare with NLP-driven deep learning unveiling biomedical materials through text mining

Empowering healthcare with NLP-driven deep learning unveiling biomedical materials through text mining

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Specifics of creating clinical abstract of biospecimens

Contact Info

Product

Resources

About