ParsBERT: Transformer-based Model for Persian Language Understanding

Farahani, Mehrdad; Gharachorloo, Mohammad; Farahani, Marzieh; Manthouri, Mohammad

doi:10.48550/arxiv.2005.12515

Cited by 6 publications

(8 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This model achieved an F1-score over 83%. Farahani et al [21] investigated BERT's performance in NLP for Persian due to its success and growing popularity in English. By training the BERT model for Persian language, they showed that it outperformed previous studies at operations such as text classification and sentiment analysis.…”

Section: Bert Based Techniquesmentioning

confidence: 99%

An Attack Detection Framework Based on BERT and Deep Learning

2022

View full text Add to dashboard Cite

Deep Learning (DL) and Natural Language Processing (NLP) techniques are improving and enriching with a rapid pace. Furthermore, we witness that the use of web applications is increasing in almost every direction in parallel with the related technologies. Web applications encompass a wide array of use cases utilizing personal, financial, defense, and political information (e.g., wikileaks incident). Indeed, to access and to manipulate such information are among the primary goals of attackers. Thus, vulnerability of the information targeted by adversaries is a vital problem and if such information is captured then the consequences can be devastating, which can, potentially, become national security risks in the extreme cases. In this study, as a remedy to this problem, we propose a novel model that is capable of distinguishing normal HTTP requests and anomalous HTTP requests. Our model employs NLP techniques, Bidirectional Encoder Representations from Transformers (BERT) model, and DL techniques. Our experimental results reveal that the proposed approach achieves a success rate over 99.98% and an F1 score over 98.70% in the classification of anomalous and normal requests. Furthermore, web attack detection time of our model is significantly lower (i.e., 0.4 ms) than the other approaches presented in the literature.

show abstract

Section: Bert Based Techniquesmentioning

confidence: 99%

An Attack Detection Framework Based on BERT and Deep Learning

2022

View full text Add to dashboard Cite

show abstract

“…ParsBERT [7] is a monolingual BERT model trained by Hooshvare Lab. 2 It is trained on monolingual corpus of Persian text and has achieved state-of-the-art results by outperforming previous mBERT-based and recurrent neural network-based models in several benchmarks.…”

Section: Parsbertmentioning

confidence: 99%

“…In addition, we train our models using the ParsBERT model [7], which is a monolingual BERT model for the Persian language that has outperformed other architectures and multilingual BERT models. We also include earlier sequence taggers, for example, bidirectional long short-term memory (BiLSTM) [8] and CRF to our pretrained models to provide a comprehensive comparative study of the impacts of text representation and learning models in NER.…”

mentioning

confidence: 99%

Comparative study of text representation and learning for Persian named entity recognition

Pour

Momtazi

2022

ETRI Journal

View full text Add to dashboard Cite

Transformer models have had a great impact on natural language processing (NLP) in recent years by realizing outstanding and efficient contextualized language models. Recent studies have used transformer-based language models for various NLP tasks, including Persian named entity recognition (NER). However, in complex tasks, for example, NER, it is difficult to determine which contextualized embedding will produce the best representation for the tasks. Considering the lack of comparative studies to investigate the use of different contextualized pretrained models with sequence modeling classifiers, we conducted a comparative study about using different classifiers and embedding models. In this paper, we use different transformer-based language models tuned with different classifiers, and we evaluate these models on the Persian NER task. We perform a comparative analysis to assess the impact of text representation and text classification methods on Persian NER performance. We train and evaluate the models on three different Persian NER datasets, that is, MoNa, Peyma, and Arman. Experimental results demonstrate that XLM-R with a linear layer and conditional random field (CRF) layer exhibited the best performance. This model achieved phrase-based F-measures of 70.04, 86.37, and 79.25 and word-based F scores of 78, 84.02, and 89.73 on the MoNa, Peyma, and Arman datasets, respectively. These results represent state-of-the-art performance on the Persian NER task.Named entity recognition (NER) detects and classifies named entities in text into categories, for example, persons, organizations, locations, and time expressions. NER is a fundamental task in many natural language processing (NLP) applications, for example, information extraction, text summarization, and machine translation, especially for low-resource languages, for example, Persian, which requires additional features to process the text effectively.NER is applied to various domains, and the set of entities is selected based on the given domain, for example, NER has been used to extract drug and disease mentions in English and Russian by Miftahutdinov and others [1]. In addition, Chen and others [2] investigated

show abstract

“…Multilingual BERT (mBERT) is pretrained on the masked LM task over 104 languages. Additionally, we use two specialized variants of BERT for Persian: wikiBERT 20 which is trained on Persian Wikipedia and ParsBERT (Farahani et al, 2020). 21 Finally, we use mT5 (Xue et al, 2020), which is a multilingual variant of the T5 architec-ture.…”

Section: Machine Translationmentioning

confidence: 99%

ParsiNLU: A Suite of Language Understanding Challenges for Persian

Khashabi

Cohan

Shakeri

et al. 2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this language. The availability of high-quality evaluation datasets is a necessity for reliable assessment of the progress on different NLU tasks and domains. We introduce ParsiNLU, the first benchmark in Persian language that includes a range of language understanding tasks—reading comprehension, textual entailment, and so on. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers. This results in over 14.5k new instances across 6 distinct NLU tasks. Additionally, we present the first results on state-of-the-art monolingual and multilingual pre-trained language models on this benchmark and compare them with human performance, which provides valuable insights into our ability to tackle natural language understanding challenges in Persian. We hope ParsiNLU fosters further research and advances in Persian language understanding.1

show abstract

ParsBERT: Transformer-based Model for Persian Language Understanding

Cited by 6 publications

References 17 publications

An Attack Detection Framework Based on BERT and Deep Learning

An Attack Detection Framework Based on BERT and Deep Learning

Comparative study of text representation and learning for Persian named entity recognition

ParsiNLU: A Suite of Language Understanding Challenges for Persian

Contact Info

Product

Resources

About