Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding

Abstract

The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT a , a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%. a https://huggingface.co/Kowsher/bangla-bert INDEX TERMS Bangla NLP, BERT-Base, Large Corpus,Transformer I. INTRODUCTION P RE-TRAINED language models based on the transformer architecture have become an absolute standard for state-of-the-art performance on a wide variety of natural language processing applications [1].BERT, a renowned transformer-based technique, brought a great revolution that had huge impacts on the evolution of NLP [2]. Since its release as an academic research paper, this technologically pioneering NLP model has amazed the AI world. It's the firstever deeply bidirectional and fully unsupervised technique

Terrorism Attack Classification Using Machine Learning: The Effectiveness of Using Textual Features Extracted from GTD Dataset

“…The model receives an input sentence X represented as a sequence of tokens X = x 1 , ..., x s and outputs a single semantic vector [CLS] of each sentence. Before processing the raw sentence having tokens, DistilBERT applies a sequence of sub-word tokenization and word-piece tokenization [68] to produce a set of embedding vectors named input embedding (S1). The tokenization maps each token to three different embeddings: word, segment, and positional.…”

An ensemble novel architecture for Bangla Mathematical Entity Recognition (MER) using transformer based learning

“…The architecture is able to improve 2.50% accuracy on PolitiFact dataset, and the improvement for GossipCop dataset is 1.10%. To break BERT's language constraints in [13] authors proposed Bangla-BERT. Here, they are able to increase the results by at most 5.3%.…”

Machine Learning for Millimeter Wave and Terahertz Beam Management: A Survey and Open Challenges

“…However, in contrast to an RNN, it can process the entire input all at once resulting in an increased training speed. Another distinguishing aspect of transformers is their ability to quickly adapt to other tasks they have not been trained on, i.e., transfer learning [145]. Due to the promising capabilities of learning long-term dependencies and transfer learning, transformers can be leveraged to deal with the high mobility scenario of BM.…”