Automated Lexicon and Feature Construction Using Word Embedding and Clustering for Classification of ASD Diagnoses Using EHR

Leroy, Gondy; Gu, Yang; Pettygrove, Sydney; Kurzius-Spencer, Margaret

doi:10.1007/978-3-319-59569-6_4

Cited by 8 publications

(8 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Text classification assigns a label or a class to a document. In the medical domain, this can be used to determine if a text refers to a positive instance of a particular medical condition [16,17]. Support vector machines (SVM) is a popular algorithm that generally performs well in a variety of tasks, including clinical applications [13,14,18].…”

Section: Nlp and Healthcare Applicationsmentioning

confidence: 99%

Use of Conventional Machine Learning to Optimize Deep Learning Hyper-parameters for NLP Labeling Tasks

Gu¹,

Leroy²

2020

Proceedings of the Annual Hawaii International Conference on System Sciences

Self Cite

View full text Add to dashboard Cite

Deep learning delivers good performance in classification tasks, but is suboptimal with small and unbalanced datasets, which are common in many domains. To address this limitation, we use conventional machine learning, i.e., support vector machines (SVM) to tune deep learning hyperparameters. We evaluated our approach using mental health electronic health records in which diagnostic criteria needed to be extracted. A bidirectional Long Short-Term Memory network (BI-LSTM) could not learn the labels for the seven scarcest classes, but saw an increase in performance after training with optimal weights learned from tuning SVMs. With these customized class weights, the F1 scores for rare classes rose from 0 to values ranging from 18% to 57%. Overall, the BI-LSTM with SVM-customized class weights achieved a micro-average of 47.1% for F1 across all classes, an improvement over the regular BI-LSTM's 45.9%. The main contribution lies in avoiding null performance for rare classes.

show abstract

Section: Nlp and Healthcare Applicationsmentioning

confidence: 99%

Use of Conventional Machine Learning to Optimize Deep Learning Hyper-parameters for NLP Labeling Tasks

Gu¹,

Leroy²

2020

Proceedings of the Annual Hawaii International Conference on System Sciences

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, Lee et al [ 16 ] proposed BioBERT by pretraining BERT on a large biomedical corpus of PubMed abstracts, and demonstrated that it outperforms BERT on three representative biomedical text mining tasks. Alsentzer et al [ 17 ] attempted to adapt pretrained models for clinical text by training BioBERT on clinical notes, resulting in the creation of BioClinical_BERT [ 18 ]. Gururangan et al [ 19 ] illustrated the usefulness of DAPT by continuing training of pretrained models on domain-specific data from four different domains (biomedical and computer science publications, news, and reviews).…”

Section: Introductionmentioning

confidence: 99%

“…A summary of the specific contributions of this paper are as follows: We compare the performances of six models pretrained with texts from different domains and sourcesBERT [ 14 ] and RoBERTa [ 15 ] (generic text), BERTweet [ 22 ], and Twitter BERT (social media text, specifically Twitter) [ 20 ], BioClinical_BERT [ 18 ] (clinical text), and BioBERT [ 16 ] (biomedical literature text)—on 22 social media-based health-related text classification tasks. We perform TSPT using the masked language model (MLM) [ 40 ], and assess its impact on classification performance compared to other pretraining strategies for three tasks.…”

Section: Introductionmentioning

confidence: 99%

Comparison of Pretraining Models and Strategies for Health-Related Social Media Text Classification

Guo

Yang

et al. 2022

Healthcare

View full text Add to dashboard Cite

Pretrained contextual language models proposed in the recent past have been reported to achieve state-of-the-art performances in many natural language processing (NLP) tasks, including those involving health-related social media data. We sought to evaluate the effectiveness of different pretrained transformer-based models for social media-based health-related text classification tasks. An additional objective was to explore and propose effective pretraining strategies to improve machine learning performance on such datasets and tasks. We benchmarked six transformer-based models that were pretrained with texts from different domains and sources—BERT, RoBERTa, BERTweet, TwitterBERT, BioClinical_BERT, and BioBERT—on 22 social media-based health-related text classification tasks. For the top-performing models, we explored the possibility of further boosting performance by comparing several pretraining strategies: domain-adaptive pretraining (DAPT), source-adaptive pretraining (SAPT), and a novel approach called topic specific pretraining (TSPT). We also attempted to interpret the impacts of distinct pretraining strategies by visualizing document-level embeddings at different stages of the training process. RoBERTa outperformed BERTweet on most tasks, and better than others. BERT, TwitterBERT, BioClinical_BERT and BioBERT consistently underperformed. For pretraining strategies, SAPT performed better or comparable to the off-the-shelf models, and significantly outperformed DAPT. SAPT+TSPT showed consistently high performance, with statistically significant improvement in three tasks. Our findings demonstrate that RoBERTa and BERTweet are excellent off-the-shelf models for health-related social media text classification, and extended pretraining using SAPT and TSPT can further improve performance.

show abstract

“…Alsentzer et al (2019) 17 attempted to adapt pretrained models for clinical text by training BioBERT on clinical notes, resulting in the creation of BioClinical_BERT. 18 Gururangan et al (2020) 19 illustrated the usefulness of DAPT by continuing training of pretrained models on domain-specific data from four different domains (biomedical and computer science publications, news, and reviews). However, some studies, including our own pilot, demonstrated that DAPT is not guaranteed to achieve SOTA results for health-related NLP tasks involving social media data.…”

Section: Introductionmentioning

confidence: 99%

Comparison of pretraining models and strategies for health-related social media text classification

Guo

Yang

et al. 2021

Preprint

View full text Add to dashboard Cite

Motivation Pretrained contextual language models proposed in the recent past have been reported to achieve state-of-the-art performances in many natural language processing (NLP) tasks. There is a need to benchmark such models for targeted NLP tasks, and to explore effective pretraining strategies to improve machine learning performance. Results In this work, we addressed the task of health-related social media text classification. We benchmarked five models-RoBERTa, BERTweet, TwitterBERT, BioClinical_BERT, and BioBERT on 22 tasks. We attempted to boost performance for the best models by comparing distinct pretraining strategies-domain-adaptive pretraining (DAPT), source-adaptive pretraining (SAPT), and topic-specific pretraining (TSPT). RoBERTa and BERTweet performed comparably in most tasks, and better than others. For pretraining strategies, SAPT performed better or comparable to the off-the-shelf models, and significantly outperformed DAPT. SAPT+TSPT showed consistently high performance, with statistically significant improvement in one task. Our findings demonstrate that RoBERTa and BERTweet are excellent off-the-shelf models for health-related social media text classification, and extended pretraining using SAPT and TSPT can further improve performance.

show abstract

Automated Lexicon and Feature Construction Using Word Embedding and Clustering for Classification of ASD Diagnoses Using EHR

Cited by 8 publications

References 3 publications

Use of Conventional Machine Learning to Optimize Deep Learning Hyper-parameters for NLP Labeling Tasks

Use of Conventional Machine Learning to Optimize Deep Learning Hyper-parameters for NLP Labeling Tasks

Comparison of Pretraining Models and Strategies for Health-Related Social Media Text Classification

Comparison of pretraining models and strategies for health-related social media text classification

Contact Info

Product

Resources

About