A Survey on Text Classification: From Shallow to Deep Learning

Li, Qian; Peng, Hao; Li, Jianxin; Xia, Congying; Yang, Renyu; Sun, Lichao; Yu, Philip S.; He, Lifang

doi:10.48550/arxiv.2008.00364

Cited by 45 publications

(47 citation statements)

References 136 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We estimate costs for each task in Table 3 (Sun et al, 2020a) that leverages the gradients of the gold labels w.r.t the embeddings of the input tokens to find the most informative tokens, which have the largest gradients among all positions within a sentence. Then we corrupt the selected tokens with one of the following typos: 1) Insertion; 2) Deletion; 3) Swap; 4) Mistype: Mistyping a word though keyboard, such as "oh" → "0h"; 5) Pronounce: Wrongly typing due to the close pronounce of the word, such as "egg" → "agg"; 6) Replace-W: Replace the word by the frequent human behavioural keyboard typo based on the Wikipedia statistics (Sun, 2020).…”

Section: Mea Setup and Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!

He¹,

Lyu²,

Sun³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Natural language processing (NLP) tasks, ranging from text classification to text generation, have been revolutionised by the pretrained language models, such as BERT. This allows corporations to easily build powerful APIs by encapsulating fine-tuned BERT models for downstream tasks. However, when a fine-tuned BERT model is deployed as a service, it may suffer from different attacks launched by the malicious users. In this work, we first present how an adversary can steal a BERT-based API service (the victim/target model) on multiple benchmark datasets with limited prior knowledge and queries. We further show that the extracted model can lead to highly transferable adversarial attacks against the victim model. Our studies indicate that the potential vulnerabilities of BERT-based API services still hold, even when there is an architectural mismatch between the victim model and the attack model. Finally, we investigate two defence strategies to protect the victim model, and find that unless the performance of the victim model is sacrificed, both model extraction and adversarial transferability can effectively compromise the target models.

show abstract

Section: Mea Setup and Resultsmentioning

confidence: 99%

“…To evaluate the efficacy of the proposed attacks, we select four NLP datasets covering two main tasks, i) sentiment analysis and ii) topic classification (Li et al, 2020). We use TP-US from Trustpilot Sentiment dataset (Hovy et al, 2015) and YELP dataset (Zhang et al, 2015) for sentiment analysis.…”

Section: Nlp Tasks and Datasetsmentioning

confidence: 99%

Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!

He¹,

Lyu²,

Sun³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

show abstract

“…These days, unstructured text is everywhere, from our conversations and comments on social media to emails, websites, etc. Their processing by means of artificial intelligence (AI) techniques and Natural Language Processing (NLP) is at a high level [1], including such an important type of text as medical records. Extracting useful information from medical texts and reports automatically plays a pivotal and important role in supporting medical decision making [2,3,4].…”

Section: Introductionmentioning

confidence: 99%

New Arabic Medical Dataset for Diseases Classification

Hammoud¹,

Vatian²,

Dobrenko³

et al. 2021

Intelligent Data Engineering and Automated Learning – IDEAL 2021

View full text Add to dashboard Cite

The Arabic language suffers from a great shortage of datasets suitable for training deep learning models, and the existing ones include general non-specialized classifications. In this work, we introduce a new Arab medical dataset, which includes two thousand medical documents collected from several Arabic medical websites, in addition to the Arab Medical Encyclopedia. The dataset was built for the task of classifying texts and includes 10 classes (Blood, Bone, Cardiovascular, Ear, Endocrine, Eye, Gastrointestinal, Immune, Liver and Nephrological) diseases. Experiments on the dataset were performed by fine-tuning three pre-trained models: BERT from Google, Arabert that based on BERT with large Arabic corpus, and AraBioNER that based on Arabert with Arabic medical corpus.

show abstract

“…We have used Hierarchical Attention Networks (HAN) from this family. The related works in literature have considered different datasets and algorithms thus making it difficult to have a holistic view [19]. In this work, we provide a comparative view of different families of algorithms on a range of datasets.…”

Section: Introductionmentioning

confidence: 99%

Comparative Study of Long Document Classification

Wagh,

Khandve,

Joshi

et al. 2021

Preprint

View full text Add to dashboard Cite

The amount of information stored in the form of documents on the internet has been increasing rapidly. Thus it has become a necessity to organize and maintain these documents in an optimum manner. Text classification algorithms study the complex relationships between words in a text and try to interpret the semantics of the document. These algorithms have evolved significantly in the past few years. There has been a lot of progress from simple machine learning algorithms to transformer-based architectures. However, existing literature has analyzed different approaches on different data sets thus making it difficult to compare the performance of machine learning algorithms. In this work, we revisit long document classification using standard machine learning approaches. We benchmark approaches ranging from simple Naive Bayes to complex BERT on six standard text classification datasets. We present an exhaustive comparison of different algorithms on a range of long document datasets. We re-iterate that long document classification is a simpler task and even basic algorithms perform competitively with BERT-based approaches on most of the datasets. The BERT-based models perform consistently well on all the datasets and can be blindly used for the document classification task when the computations cost is not a concern. In the shallow model's category, we suggest the usage of raw BiLSTM + Max architecture which performs decently across all the datasets. Even simpler Glove + Attention bag of words model can be utilized for simpler use cases. The importance of using sophisticated models is clearly visible in the IMDB sentiment dataset which is a comparatively harder task.

show abstract

A Survey on Text Classification: From Shallow to Deep Learning

Cited by 45 publications

References 136 publications

Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!

Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!

New Arabic Medical Dataset for Diseases Classification

Comparative Study of Long Document Classification

Contact Info

Product

Resources

About