Key Concept Identification for Medical Information Retrieval

Zheng, Jiaping; Yu, Hong

doi:10.18653/v1/d15-1069

Cited by 6 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One study has shown that linking medical terms in EHR notes to Wikipedia pages can improve patient’s comprehension [ 51 ]. Moreover, many methods have been proposed to identify important or potentially unfamiliar medical terms [ 52 , 53 ].…”

Section: Methodsmentioning

confidence: 99%

Readability Formulas and User Perceptions of Electronic Health Records Difficulty: A Corpus Study

Zheng¹,

Yu²

2017

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

BackgroundElectronic health records (EHRs) are a rich resource for developing applications to engage patients and foster patient activation, thus holding a strong potential to enhance patient-centered care. Studies have shown that providing patients with access to their own EHR notes may improve the understanding of their own clinical conditions and treatments, leading to improved health care outcomes. However, the highly technical language in EHR notes impedes patients’ comprehension. Numerous studies have evaluated the difficulty of health-related text using readability formulas such as Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), and Gunning-Fog Index (GFI). They conclude that the materials are often written at a grade level higher than common recommendations.ObjectiveThe objective of our study was to explore the relationship between the aforementioned readability formulas and the laypeople’s perceived difficulty on 2 genres of text: general health information and EHR notes. We also validated the formulas’ appropriateness and generalizability on predicting difficulty levels of highly complex technical documents.MethodsWe collected 140 Wikipedia articles on diabetes and 242 EHR notes with diabetes International Classification of Diseases, Ninth Revision code. We recruited 15 Amazon Mechanical Turk (AMT) users to rate difficulty levels of the documents. Correlations between laypeople’s perceived difficulty levels and readability formula scores were measured, and their difference was tested. We also compared word usage and the impact of medical concepts of the 2 genres of text.ResultsThe distributions of both readability formulas’ scores (P<.001) and laypeople’s perceptions (P=.002) on the 2 genres were different. Correlations of readability predictions and laypeople’s perceptions were weak. Furthermore, despite being graded at similar levels, documents of different genres were still perceived with different difficulty (P<.001). Word usage in the 2 related genres still differed significantly (P<.001).ConclusionsOur findings suggested that the readability formulas’ predictions did not align with perceived difficulty in either text genre. The widely used readability formulas were highly correlated with each other but did not show adequate correlation with readers’ perceived difficulty. Therefore, they were not appropriate to assess the readability of EHR notes.

show abstract

Section: Methodsmentioning

confidence: 99%

Readability Formulas and User Perceptions of Electronic Health Records Difficulty: A Corpus Study

Zheng¹,

Yu²

2017

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…Concept identification indexing approaches have been used for information retrieval in different domains. Zheng and Yu (2015) developed a system for medical information retrieval. The medical terms in electronic health records (EHRs) are often hard for patients to understand.…”

Section: Concept Identification and Retrievalmentioning

confidence: 99%

Knowledge management: library and information science writ large

Zhang¹,

Koenig²

2021

A Research Agenda for Knowledge Management and Analytics

View full text Add to dashboard Cite

In light of the rapidly evolving capabilities of large language models (LLMs), it becomes imperative to develop rigorous domain-specific evaluation benchmarks to accurately assess their capabilities. In response to this need, this paper introduces ArcMMLU, a specialized benchmark tailored for the Library & Information Science (LIS) domain in Chinese. This benchmark aims to measure the knowledge and reasoning capability of LLMs within four key sub-domains: Archival Science, Data Science, Library Science, and Information Science. Following the format of MMLU/CMMLU, we collected over 6,000 high-quality questions for the compilation of ArcMMLU. This extensive compilation can reflect the diverse nature of the LIS domain and offer a robust foundation for LLM evaluation. Our comprehensive evaluation reveals that while most mainstream LLMs achieve an average accuracy rate above 50% on ArcMMLU, there remains a notable performance gap, suggesting substantial headroom for refinement in LLM capabilities within the LIS domain. Further analysis explores the effectiveness of few-shot examples on model performance and highlights challenging questions where models consistently underperform, providing valuable insights for targeted improvements. ArcMMLU fills a critical gap in LLM evaluations within the Chinese LIS domain and paves the way for future development of LLMs tailored to this specialized area 1 .

show abstract

“…Furthermore, their processing through statistical techniques alone appears insufficient since they encompass several domain-specific medical concepts [2] that require making use of extrinsic knowledge for their deciphering [3]. This forms a crucial challenge for Medical Information Retrieval (MIR) systems that aim to find matches between medical documents and their corresponding queries in the same domain [4][5][6] and motivates the use for language resources when further expanding these queries [7,8]. Recently, MIR systems have shifted to exploiting medical semantic resources and ontologies in an attempt to capture knowledge in this domain through formally and explicitly defining medical concepts, instances, as well as semantic and taxonomic relations that link related concepts.…”

Section: Introductionmentioning

confidence: 99%

Head Concepts Selection for Verbose Medical Queries Expansion

et al. 2020

View full text Add to dashboard Cite

Semantic concepts and relations encoded in domain-specific ontologies and other medical semantic resources play a crucial role in deciphering terms in medical queries and documents. The exploitation of these resources for tackling the semantic gap issue has been widely studied in the literature. However, there are challenges that hinder their widespread use in real-world applications. Among these challenges is the insufficient knowledge individually encoded in existing medical ontologies, which is magnified when users express their information needs using long-winded natural language queries. In this context, many of the users' query terms are either unrecognized by the used ontologies, or cause retrieving false positives that degrade the quality of current medical information search approaches. In this article, we explore the combination of multiple extrinsic semantic resources in the development of a full-fledged medical information search framework to: i) highlight and expand head medical concepts in verbose medical queries (i.e. concepts among query terms that significantly contribute to the informativeness and intent of a given query), ii) build semantically-enhanced inverted index documents, iii) contribute to a heuristical weighting technique in the query-document matching process. To demonstrate the effectiveness of the proposed approach, we conducted several experiments over the CLEF e-Health 2014 dataset. Findings indicate that the proposed method combining several extrinsic semantic resources proved to be more effective than related approaches in terms of precision measure.

show abstract

Key Concept Identification for Medical Information Retrieval

Cited by 6 publications

References 18 publications

Readability Formulas and User Perceptions of Electronic Health Records Difficulty: A Corpus Study

Readability Formulas and User Perceptions of Electronic Health Records Difficulty: A Corpus Study

Knowledge management: library and information science writ large

Head Concepts Selection for Verbose Medical Queries Expansion

Contact Info

Product

Resources

About