Evaluating resources composing the PheMAP knowledge base to enhance high-throughput phenotyping

Wan, Nicholas C; Yaqoob, Ali; Ong, Helena; Zhao, Juan; Wei, Wei‐Qi

doi:10.1093/jamia/ocac234

Cited by 4 publications

(2 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We deployed the highest-rated phenotyping algorithms for each phenotype in a research cohort at VUMC (n=84,821). This cohort, extensively utilized in phenotyping research, has been a significant resource for phenotypic studies 5,31 . We summarized implementation details in Supplemental Section S.3, where we list the general edits we made as well as specific changes for each implemented algorithm (Supplemental Table 3 ).…”

Section: Methodsmentioning

confidence: 99%

“…We deployed the highest-rated phenotyping algorithms for each phenotype in a research cohort at VUMC (n=84,821). This cohort, extensively utilized in phenotyping research, has been a significant resource for phenotypic studies 30,31 . As a benchmark, we implemented three eMERGE algorithms updated with current ICD-10-CM codes 31 to identify phenotype cases and controls.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Yan,

Ong,

Grabowska

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

ObjectivesPhenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm typically demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating drafts of high-quality algorithms.Materials and MethodsWe prompted four LLMs—ChatGPT-4, ChatGPT-3.5, Claude 2, and Bard—in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model for three clinical phenotypes (i.e., type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms from each LLM and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.ResultsChatGPT-4 and ChatGPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although ChatGPT-4 and ChatGPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).ConclusionBoth ChatGPT versions 3.5 and 4 demonstrate the capability to enhance EHR phenotyping efficiency by drafting algorithms of reasonable quality. However, the optimal performance of these algorithms necessitates the involvement of domain experts.

show abstract

Section: Methodsmentioning

confidence: 99%

“…We deployed the highest-rated phenotyping algorithms for each phenotype in a research cohort at VUMC (n=84,821). This cohort, extensively utilized in phenotyping research, has been a significant resource for phenotypic studies 30,31 . As a benchmark, we implemented three eMERGE algorithms updated with current ICD-10-CM codes 31 to identify phenotype cases and controls.…”

Section: Methodsmentioning

confidence: 99%

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Yan,

Ong,

Grabowska

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Dementia and electronic health record phenotypes: a scoping review of available phenotypes and opportunities for future research

Walling

Pevnick

Bennett

et al. 2023

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objective We performed a scoping review of algorithms using electronic health record (EHR) data to identify patients with Alzheimer’s disease and related dementias (ADRD), to advance their use in research and clinical care. Materials and Methods Starting with a previous scoping review of EHR phenotypes, we performed a cumulative update (April 2020 through March 1, 2023) using Pubmed, PheKB, and expert review with exclusive focus on ADRD identification. We included algorithms using EHR data alone or in combination with non-EHR data and characterized whether they identified patients at high risk of or with a current diagnosis of ADRD. Results For our cumulative focused update, we reviewed 271 titles meeting our search criteria, 49 abstracts, and 26 full text papers. We identified 8 articles from the original systematic review, 8 from our new search, and 4 recommended by an expert. We identified 20 papers describing 19 unique EHR phenotypes for ADRD: 7 algorithms identifying patients with diagnosed dementia and 12 algorithms identifying patients at high risk of dementia that prioritize sensitivity over specificity. Reference standards range from only using other EHR data to in-person cognitive screening. Conclusion A variety of EHR-based phenotypes are available for use in identifying populations with or at high-risk of developing ADRD. This review provides comparative detail to aid in choosing the best algorithm for research, clinical care, and population health projects based on the use case and available data. Future research may further improve the design and use of algorithms by considering EHR data provenance.

show abstract

Improving reporting standards for phenotyping algorithm in biomedical research: 5 fundamental dimensions

Wei,

Rowley,

Wood

et al. 2024

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Introduction Phenotyping algorithms enable the interpretation of complex health data and definition of clinically relevant phenotypes; they have become crucial in biomedical research. However, the lack of standardization and transparency inhibits the cross-comparison of findings among different studies, limits large scale meta-analyses, confuses the research community, and prevents the reuse of algorithms, which results in duplication of efforts and the waste of valuable resources. Recommendations Here, we propose five independent fundamental dimensions of phenotyping algorithms—complexity, performance, efficiency, implementability, and maintenance—through which researchers can describe, measure, and deploy any algorithms efficiently and effectively. These dimensions must be considered in the context of explicit use cases and transparent methods to ensure that they do not reflect unexpected biases or exacerbate inequities.

show abstract

Evaluating resources composing the PheMAP knowledge base to enhance high-throughput phenotyping

Cited by 4 publications

References 26 publications

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Dementia and electronic health record phenotypes: a scoping review of available phenotypes and opportunities for future research

Improving reporting standards for phenotyping algorithm in biomedical research: 5 fundamental dimensions

Contact Info

Product

Resources

About