Clinical Named Entity Recognition using Contextualized Token Representations

Abstract

The clinical named entity recognition (CNER) task seeks to locate and classify clinical terminologies into predefined categories, such as diagnostic procedure, disease disorder, severity, medication, medication dosage, and sign symptom. CNER facilitates the study of side-effect on medications including identification of novel phenomena and human-focused information extraction. Existing approaches in extracting the entities of interests focus on using static word embeddings to represent each word. However, one word can have different interpretations that depend on the context of the sentences. Evidently, static word embeddings are insufficient to integrate the diverse interpretation of a word. To overcome this challenge, the technique of contextualized word embedding has been introduced to better capture the semantic meaning of each word based on its context. Two of these language models, ELMo and Flair, have been widely used in the field of Natural Language Processing to generate the contextualized word embeddings on domain-generic documents. However, these embeddings are usually too general to capture the proximity among vocabularies of specific domains. To facilitate various downstream applications using clinical case reports (CCRs), we pre-train two deep contextualized language models, Clinical Embeddings from Language Model (C-ELMo) and Clinical Contextual String Embeddings (C-Flair) using the clinical-related corpus from the PubMed Central. Explicit experiments show that our models gain dramatic improvements compared to both static word embeddings and domain-generic language models. The pre-trained embeddings of these two models will be available soon.

Information extraction from electronic medical documents: state of the art and future research directions

“…Zhou et al [163] have pre-trained two deep contextualized language models on clinical corpus from the PubMed Central (PMC): Clinical Embeddings from Language Model (C-ELMo) for word-level features and C-FLAIR clinical contextual string embeddings for characterlevel features. Then, each of the two embeddings is concatenated with Glove embedding and passed to BiLSTM-CRF model to extract entity types.…”

“…However, the method may miss some medical entities in the non-medical terms filtering step. The method of Zhou et al [163] can solve the ambiguity problem by making two types of embeddings for more context, which are C-ELMo for word-level features and C-Flair -Filtering may miss some medical entities.…”

“…The method of Deng et al [33] is based on the character embedding which can avoid the variability problem while it is not restricted by a vocabulary of words. Zhou et al [163] can solve the variability by using character embedding to handle out-of-vocabulary words. Zhao et al [161] can avoid the name variation problem where they define different types of rules which considers the lexical, contextual and syntax information based on the clues to find entities.…”