MACEDONIZER - The Macedonian Transformer Language Model

Dobreva, Jovana; Pavlov, Tashko; Mishev, Kostadin; Simjanoska, Monika; Tudzarski, Stojancho; Trajanov, Dimitar; Kocarev, Lj.

doi:10.1007/978-3-031-22792-9_5

Cited by 2 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jacob Devlin and his colleagues from Google developed BERT and released it in 2018. Google announced in 2019 that it started using BERT in its search engine and by the end of 2020, BERT was being utilized in nearly all English-language queries [6]. The ParsBERT model, which was presented in [5], is a language model for a single language that employs Google's BERT architecture.…”

Section: History Of Bert and Parsbertmentioning

confidence: 99%

Evaluating Bert and Parsbert for Analyzing Persian Advertisement Data

Mehrban¹,

Ahadian²

2023

IJNLC

View full text Add to dashboard Cite

This paper discusses the impact of the Internet on modern trading and the importance of data generated from these transactions for organizations to improve their marketing efforts. The paper uses the example of Divar, an online marketplace for buying and selling products and services in Iran, and presents a competition to predict the percentage of a car sales ad that would be published on the Divar website. Since the dataset provides a rich source of Persian text data, the authors use the Hazm library, a Python library designed for processing Persian text, and two state-of-the-art language models, mBERT and ParsBERT, to analyze it. The paper's primary objective is to compare the performance of mBERT and ParsBERT on the Divar dataset. The authors provide some background on data mining, Persian language, and the two language models, examine the dataset's composition and statistical features, and provide details on their fine-tuning and training configurations for both approaches. They present the results of their analysis and highlight the strengths and weaknesses of the two language models when applied to Persian text data. The paper offers valuable insights into the challenges and opportunities of working with low-resource languages such as Persian and the potential of advanced language models like BERT for analyzing such data. The paper also explains the data mining process, including steps such as data cleaning and normalization techniques. Finally, the paper discusses the types of machine learning problems, such as supervised, unsupervised, and reinforcement learning, and the pattern evaluation techniques, such as confusion matrix. Overall, the paper provides an informative overview of the use of language models and data mining techniques for analyzing text data in low-resource languages, using the example of the Divar dataset.

show abstract

Section: History Of Bert and Parsbertmentioning

confidence: 99%

Evaluating Bert and Parsbert for Analyzing Persian Advertisement Data

Mehrban¹,

Ahadian²

2023

IJNLC

View full text Add to dashboard Cite

show abstract

“…Fast forward to today, decoder-based language models are most prominent in the field, with the OpenAI GPT models (now in the fourth generation) being especially popular for instruction tuning [10]. However, their last model published in open code (and also the latest one available for Serbian) is still GPT-2 [11], with the efforts still being focused mainly on the development of encoder-based models both for Serbian [12] and similar Slavic languages [13][14][15][16].…”

Section: State-of-the-artmentioning

confidence: 99%

Transformer-Based Composite Language Models for Text Evaluation and Classification

Škorić,

Utvić,

Stanković

2023

Mathematics

View full text Add to dashboard Cite

Parallel natural language processing systems were previously successfully tested on the tasks of part-of-speech tagging and authorship attribution through mini-language modeling, for which they achieved significantly better results than independent methods in the cases of seven European languages. The aim of this paper is to present the advantages of using composite language models in the processing and evaluation of texts written in arbitrary highly inflective and morphology-rich natural language, particularly Serbian. A perplexity-based dataset, the main asset for the methodology assessment, was created using a series of generative pre-trained transformers trained on different representations of the Serbian language corpus and a set of sentences classified into three groups (expert translations, corrupted translations, and machine translations). The paper describes a comparative analysis of calculated perplexities in order to measure the classification capability of different models on two binary classification tasks. In the course of the experiment, we tested three standalone language models (baseline) and two composite language models (which are based on perplexities outputted by all three standalone models). The presented results single out a complex stacked classifier using a multitude of features extracted from perplexity vectors as the optimal architecture of composite language models for both tasks.

show abstract

MACEDONIZER - The Macedonian Transformer Language Model

Cited by 2 publications

References 13 publications

Evaluating Bert and Parsbert for Analyzing Persian Advertisement Data

Evaluating Bert and Parsbert for Analyzing Persian Advertisement Data

Transformer-Based Composite Language Models for Text Evaluation and Classification

Contact Info

Product

Resources

About