Adapting Pre-trained Word Embeddings For Use In Medical Coding

Patel, Kevin; Patel, Divya; Golakiya, Mansi; Bhattacharyya, Pushpak; Birari, Nilesh

doi:10.18653/v1/w17-2338

Cited by 18 publications

(13 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From these results we can conclude that pre-trained embeddings can improve the language modeling quality, but only if training content is aligned with the task at hand. Similar results are reported by Kim [29] and Patel et al [30]: refining the Google embeddings for the described task and data improves results.…”

Section: Resultssupporting

confidence: 87%

Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

Verwimp

Bellegarda

2019

Interspeech 2019

View full text Add to dashboard Cite

Natural language processing (NLP) tasks tend to suffer from a paucity of suitably annotated training data, hence the recent success of transfer learning across a wide variety of them. The typical recipe involves: (i) training a deep, possibly bidirectional, neural network with an objective related to language modeling, for which training data is plentiful; and (ii) using the trained network to derive contextual representations that are far richer than standard linear word embeddings such as word2vec, and thus result in important gains. In this work, we wonder whether the opposite perspective is also true: can contextual representations trained for different NLP tasks improve language modeling itself? Since language models (LMs) are predominantly locally optimized, other NLP tasks may help them make better predictions based on the entire semantic fabric of a document. We test the performance of several types of pre-trained embeddings in neural LMs, and we investigate whether it is possible to make the LM more aware of global semantic information through embeddings pre-trained with a domain classification model. Initial experiments suggest that as long as the proper objective criterion is used during training, pre-trained embeddings are likely to be beneficial for neural language modeling.

show abstract

Section: Resultssupporting

confidence: 87%

Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

Verwimp

Bellegarda

2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…This is done by back‐propagating the training errors to the embedding level, as suggested in Collobert et al (2011). For example, for the task of medical coding in Patel et al (2017), embeddings were improved with the addition of information from ICD‐10 codes.…”

Section: Discussionmentioning

confidence: 99%

Word embeddings for biomedical natural language processing: A survey

Chiu

Baker

2020

Language and Linguist. Compass

View full text Add to dashboard Cite

Word representations are mathematical objects that capture the semantic and syntactic properties of words in a way that is interpretable by machines. Recently, encoding word properties into low‐dimensional vector spaces using neural networks has become increasingly popular. Word embeddings are now used as the main input to natural language processing (NLP) applications, achieving cutting‐edge results. Nevertheless, most word‐embedding studies are carried out with general‐domain text and evaluation datasets, and their results do not necessarily apply to text from other domains (e.g., biomedicine) that are linguistically distinct from general English. To achieve maximum benefit when using word embeddings for biomedical NLP tasks, they need to be induced and evaluated using in‐domain resources. Thus, it is essential to create a detailed review of biomedical embeddings that can be used as a reference for researchers to train in‐domain models. In this paper, we review biomedical word embedding studies from three key aspects: the corpora, models and evaluation methods. We first describe the characteristics of various biomedical corpora, and then compare popular embedding models. After that, we discuss different evaluation methods for biomedical embeddings. For each aspect, we summarize the various challenges discussed in the literature. Finally, we conclude the paper by proposing future directions that will help advance research into biomedical embeddings.

show abstract

“…For example, the original BERT took 4 days to pre-train on 4 TPUs. Furthermore, there are few limitations of using a non-medical corpus to train a model for medical tasks (Patel et al, 2017). There are medical-specific terms that do not usually exist in general corpora such as news or Wikipedia.…”

Section: Problem Statementmentioning

confidence: 99%

Assessment of DistilBERT performance on Named Entity Recognition task for the detection of Protected Health Information and medical concepts

Abadeer¹

2020

Proceedings of the 3rd Clinical Natural Language Processing Workshop

View full text Add to dashboard Cite

Bidirectional Encoder Representations from Transformers (BERT) models achieve state-ofthe-art performance on a number of Natural Language Processing tasks. However, their model size on disk often exceeds 1 GB and the process of fine-tuning them and using them to run inference consumes significant hardware resources and runtime. This makes them hard to deploy to production environments. This paper fine-tunes DistilBERT, a lightweight deep learning model, on medical text for the named entity recognition task of Protected Health Information (PHI) and medical concepts. This work provides a full assessment of the performance of DistilBERT in comparison with BERT models that were pre-trained on medical text. For Named Entity Recognition task of PHI, DistilBERT achieved almost the same results as medical versions of BERT in terms of F 1 score at almost half the runtime and consuming approximately half the disk space. On the other hand, for the detection of medical concepts, DistilBERT's F 1 score was lower by 4 points on average than medical BERT variants.

show abstract

Adapting Pre-trained Word Embeddings For Use In Medical Coding

Cited by 18 publications

References 15 publications

Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

Word embeddings for biomedical natural language processing: A survey

Assessment of DistilBERT performance on Named Entity Recognition task for the detection of Protected Health Information and medical concepts

Contact Info

Product

Resources

About