Fine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing

Abstract

With the advent of pre-trained language models, many natural language processing tasks in various languages have achieved great success. Although some research has been conducted on fine-tuning BERT-based models for syntactic parsing, and several Arabic pre-trained models have been developed, no attention has been paid to Arabic dependency parsing. In this study, we attempt to fill this gap and compare nine Arabic models, fine-tuning strategies, and encoding methods for dependency parsing. We evaluated three treebanks to highlight the best options and methods for fine-tuning Arabic BERT-based models to capture syntactic dependencies in the data. Our exploratory results show that the AraBERTv2 model provides the best scores for all treebanks and confirm that fine-tuning to the higher layers of pre-trained models is required. However, adding additional neural network layers to those models drops the accuracy. Additionally, we found that the treebanks have differences in the encoding techniques that give the highest scores. The analysis of the errors obtained by the test examples highlights four issues that have an important effect on the results: parse tree post-processing, contextualized embeddings, erroneous tokenization, and erroneous annotation. This study reveals a direction for future research to achieve enhanced Arabic BERT-based syntactic parsing.

CamelParser2.0: A State-of-the-Art Dependency Parser for Arabic

“…There have been previous developments in Arabic dependency parsing Marton et al, 2013;Zhang et al, 2015;Shahrour et al, 2016;Al-Ghamdi et al, 2023). However, they are not based on state-of-the-art (SOTA) developments in neural dependency parsing and pre- 'w+ ' 'hl' 's+' 'yšrHwn' '+hA' '?…”

Natural Language Processing: Recent Development and Applications

“…In terms of NLP applications, Alrumayyan and Al-Yahya [7] utilized language modeling and neural embeddings to support the task of jurisprudence principles. Al-Ghamdi et al [8] employed the fine-tuning of Arabic bidirectional encoder representations from transformer-based models to develop Arabic pre-trained models. Alanazi [9] used cryptocurrency-related Twitter text to classify pure and compound sentiments.…”