Multi-layer Representation Learning for Medical Concepts

Abstract

Learning efficient representations for concepts has been proven to be an important basis for many applications such as machine translation or document classification. Proper representations of medical concepts such as diagnosis, medication, procedure codes and visits will have broad applications in healthcare analytics. However, in Electronic Health Records (EHR) the visit sequences of patients include multiple concepts (diagnosis, procedure, and medication codes) per visit. This structure provides two types of relational information, namely sequential order of visits and co-occurrence of the codes within each visit. In this work, we propose Med2Vec, which not only learns distributed representations for both medical codes and visits from a large EHR dataset with over 3 million visits, but also allows us to interpret the learned representations confirmed positively by clinical experts. In the experiments, Med2Vec displays significant improvement in key medical applications compared to popular baselines such as Skipgram, GloVe and stacked autoencoder, while providing clinically meaningful interpretation.

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval

“…There are two key differences between all these close previous work [3,4,14,25] and ours. First, unlike these past approaches, we tackle the readability of word senses and polysemy problems by learning document representations that leverage semantics inventoried in both text corpora and knowledge resources through fine-grained elements including words and concepts in a joint learning process.…”

“…The second and recent line of work aims at refining word embedding using relational constraints to better discriminate word senses by simultaneously learning the concept representations and inferring word senses, and accordingly tackling the polysemy issue [3,4,14,25]. Mancini et al [14] simultaneously learn embeddings for both words and their senses via a semantic network based on the CBOW architecture.…”

“…First, unlike these past approaches, we tackle the readability of word senses and polysemy problems by learning document representations that leverage semantics inventoried in both text corpora and knowledge resources through fine-grained elements including words and concepts in a joint learning process. Moreover, in contrast to [4] that considers the document context as a temporal feature directly injected in the objective function for learning representations of fine-grained elements, we assume that there is a mutual benefit to learning simultaneously document, word, and concept embeddings to better capture the semantics at global and local levels. Second, we also tackle the data sparsity problem and show the quality of the learned representations of documents as well as related concepts and words used as auxiliary information to enhance the query-document matching while most of previous work focused on the polysemy problem within NLP tasks.…”

Privacy-Preserving Predictive Modeling: Harmonization of Contextual Embeddings From Different Sources (Preprint)

“…kidney failure). Models that utilize such representation have shown higher prediction performance than previous models that do not [20,21].…”

“…Recently, there has been considerable attention in the application of neural networks to represent medical concepts as multi-dimensional and continuous vectors [20,21]. A process called contextual embedding, commonly used in natural language processing, maps each word from a corpus of text to a hyper-dimensional space where similar words in terms of meaning and/or distributed usage would be located nearby (e.g., short cosine distance).…”