IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

Abstract

Although the Indonesian language is spoken by almost 200 million people and the 10th mostspoken language in the world, 1 it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the INDOLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release INDOBERT, a new pre-trained language model for Indonesian, and evaluate it over INDOLEM, in addition to benchmarking it against existing resources. Our experiments show that INDOBERT achieves state-of-the-art performance over most of the tasks in INDOLEM.

Out-of-Scope Intent Detection on A Knowledge-Based Chatbot

“…Even two versions of IndoBERT models, [28] and [30], managed to classify IS queries with a 100% F1 score. Meanwhile, [29] got the highest F1 score of 98% when using a learning rate of 5e-5 and a batch size of 16. Thus, we employed this learning rate and batch size value for subsequent evaluations.…”

“…6. Model of [28,30] converged faster than [29]. At the epoch 10 and 12, the accuracy of [28,30] yielded 100%, respectively.…”

“…IndoNLU model trained 12 corpus collections, both formal and colloquial Indonesian sentences, containing 23.43 GB of texts, four billion words, with around 250 million sentences [28]. Moreover, [29] trained three primary sources containing 220 million sentences. Meanwhile, [30] trained 16 GB of raw texts, around two billion words.…”

Indonesian Information Extraction : Challenges and Opportunities

“…Also, the studies produced several datasets that can support the information extraction process, such as the dataset from Gultom's research [7]and Syaifudin [14], which used a dataset that has been labeled tagging for the Named Entity Recognition (NER). Also, research builds Corpus for NLP Indonesian to support information extraction, namely Fajri Koto [15], which m arouses IndoLEM, dataset NLP for Indonesian and mele n gkapi dataset BERT for Language Indonesia, IndoBERT.…”

SkoltechNLP at SemEval-2021 Task 5: Leveraging Sentence-level Pre-training for Toxic Span Detection

“…Word-level toxicity classification can be formulated as a sequence labeling task, which also actively uses the pre-trained models mentioned above. BERT comprises the versatile information on words and their context, which allows to successfully use it for sequence labeling tasks of different levels: part-of-speech tagging and syntactic parsing (Koto et al, 2020), named entity recognition (Hakala and Pyysalo, 2019), semantic role labeling (He et al, 2019), detection of Machine Translation errors (Moura et al, 2020).…”