Comparison of Pretraining Models and Strategies for Health-Related Social Media Text Classification

Guo, Yuting; Ge, Yufeng; Yang, Yuan-Chi; Al-Garadi, Mohammed Ali; Sarker, Abeed

doi:10.3390/healthcare10081478

Cited by 12 publications

(10 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of included papers used more than one dataset to evaluate their classifiers in parallel, showing differences between the evaluations scores of the same algorithm or architecture and therefore making it hard to estimate how each algorithm would perform in real-life, with potentially new or evolving data. [57][58][59] In total, 15 papers described qualitative or practical evaluations of their algorithms. In the most cases, the qualitative evaluation completely replaced quantitative analysis.…”

Section: Evaluation Of Algorithmsmentioning

confidence: 99%

Automated data analysis of unstructured grey literature in health research: A mapping review

Schmidt,

Mohamed,

Meader

et al. 2023

Research Synthesis Methods

View full text Add to dashboard Cite

The amount of grey literature and ‘softer’ intelligence from social media or websites is vast. Given the long lead‐times of producing high‐quality peer‐reviewed health information, this is causing a demand for new ways to provide prompt input for secondary research. To our knowledge, this is the first review of automated data extraction methods or tools for health‐related grey literature and soft data, with a focus on (semi)automating horizon scans, health technology assessments (HTA), evidence maps, or other literature reviews. We searched six databases to cover both health‐ and computer‐science literature. After deduplication, 10% of the search results were screened by two reviewers, the remainder was single‐screened up to an estimated 95% sensitivity; screening was stopped early after screening an additional 1000 results with no new includes. All full texts were retrieved, screened, and extracted by a single reviewer and 10% were checked in duplicate. We included 84 papers covering automation for health‐related social media, internet fora, news, patents, government agencies and charities, or trial registers. From each paper, we extracted data about important functionalities for users of the tool or method; information about the level of support and reliability; and about practical challenges and research gaps. Poor availability of code, data, and usable tools leads to low transparency regarding performance and duplication of work. Financial implications, scalability, integration into downstream workflows, and meaningful evaluations should be carefully planned before starting to develop a tool, given the vast amounts of data and opportunities those tools offer to expedite research.

show abstract

Section: Evaluation Of Algorithmsmentioning

confidence: 99%

Automated data analysis of unstructured grey literature in health research: A mapping review

Schmidt,

Mohamed,

Meader

et al. 2023

Research Synthesis Methods

View full text Add to dashboard Cite

show abstract

“…As an example, Qasim et al 15 detected the fake news of COVID-19 by tuning BERT, RoBERTa, and XLNet. Guo et al 16 evaluated six types of PLMs on social media health-related text classification tasks. An elaborate experiment conducted by Lu et al 40 verified that BERT performed best in all scenarios for classifying the presence or the absence of 16 diseases from patient discharge summaries.…”

Section: Related Workmentioning

confidence: 99%

“… 13 , 14 In the early stages of exploiting such knowledge for the MTC, researchers added additional classifiers on the top of PLMs and fine-tuned both of them using task-specific objective functions. 15 , 16 However, one major problem in such a “fine-tuning” paradigm is that additional parameters are introduced when tuning the extra classifiers. Therefore, how to bridge the gap between pre-training objectives and classification tasks has been considered as a critical factor in making a PLM more suitable for MTC.…”

Section: Introductionmentioning

confidence: 99%

Medical text classification based on the discriminative pre-training model and prompt-tuning

Wang

Peng

et al. 2023

DIGITAL HEALTH

View full text Add to dashboard Cite

Medical text classification, as a fundamental medical natural language processing task, aims to identify the categories to which a short medical text belongs. Current research has focused on performing the medical text classification task using a pre-training language model through fine-tuning. However, this paradigm introduces additional parameters when training extra classifiers. Recent studies have shown that the “prompt-tuning” paradigm induces better performance in many natural language processing tasks because it bridges the gap between pre-training goals and downstream tasks. The main idea of prompt-tuning is to transform binary or multi-classification tasks into mask prediction tasks by fully exploiting the features learned by pre-training language models. This study explores, for the first time, how to classify medical texts using a discriminative pre-training language model called ERNIE-Health through prompt-tuning. Specifically, we attempt to perform prompt-tuning based on the multi-token selection task, which is a pre-training task of ERNIE-Health. The raw text is wrapped into a new sequence with a template in which the category label is replaced by a [UNK] token. The model is then trained to calculate the probability distribution of the candidate categories. Our method is tested on the KUAKE-Question Intention Classification and CHiP-Clinical Trial Criterion datasets and obtains the accuracy values of 0.866 and 0.861. In addition, the loss values of our model decrease faster throughout the training period compared to the fine-tuning. The experimental results provide valuable insights to the community and suggest that prompt-tuning can be a promising approach to improve the performance of pre-training models in domain-specific tasks.

show abstract

“…In other words, a weight tells how much the input affects the output. Biases, which are always the same, are an extra input for the next layer that will always be 1 [33] [34].…”

Section: Weighted Average or Weighted Sum Ensemblementioning

confidence: 99%

“…www.ijacsa.thesai.org transferring knowledge from distinct but related source domains. ResNet50V2, ResNet101V2', 'MobileNetV3Large', 'MobileNetV3Small', 'MobileNet', 'EfficientNetB0', 'EfficientNetB1', and 'EfficientNetB2' are examples of pretrained networks [33]…”

mentioning

confidence: 99%

Multi Oral Disease Classification from Panoramic Radiograph using Transfer Learning and XGBoost

Jaiswal¹,

Katkar²,

Bhirud³

2022

IJACSA

View full text Add to dashboard Cite

The subject of oral healthcare is a crucial research field with significant technological development. This research examines the field of oral health care known as dentistry, a branch of medicine concerned with the anatomy, development, and disorders of the teeth. Good oral health is essential for speaking, smiling, testing, touching, digesting food, swallowing, and many other aspects, such as expressing a variety of emotions through facial expressions. Comfort in doing all these activities contributes to a person's self-confidence. For diagnosing multiple oral diseases at a time panoramic radiograph is used. Oral healthcare experts are important to appropriately detect and classify disorders. This automated approach was developed to eliminate the overhead of experts and the time required for diagnosis. This research is based on a self-created dataset of 500 images representing six distinct diseases in 46 possible combinations. Tooth wear, periapical, periodontitis, tooth decay, missing tooth, and impacted tooth are all examples of diseases. This system is developed using the concept of transfer learning with the use of a different pre-trained network such as "ResNet50V2", "ResNet101V2", "MobileNetV3Large", "MobileNetV3Small", "MobileNet", "EfficientNetB0", "EfficientNetB1", and "EfficientNetB2" with XGBoost and to get the final prediction The images in the dataset were divided into 80% training and 20% images for testing. To assess the performance of this system, various measuring metrics are used. Experiments revealed that the proposed model detected Tooth wear, periapical, periodontitis, tooth decay, missing tooth, and impacted tooth with an accuracy of 91.8%, 92.2%, 92.4%, 93.2%, 91.6%, and 90.8%, respectively.

show abstract

Comparison of Pretraining Models and Strategies for Health-Related Social Media Text Classification

Cited by 12 publications

References 52 publications

Automated data analysis of unstructured grey literature in health research: A mapping review

Automated data analysis of unstructured grey literature in health research: A mapping review

Medical text classification based on the discriminative pre-training model and prompt-tuning

Multi Oral Disease Classification from Panoramic Radiograph using Transfer Learning and XGBoost

Contact Info

Product

Resources

About