Assessing BERT’s ability to learn Italian syntax: a study on null-subject and agreement phenomena

Guarasci, Raffaele; Silvestri, Stefano; Pietro, Giuseppe De; Fujita, Hamido; Esposito, Massimo

doi:10.1007/s12652-021-03297-4

Cited by 34 publications

(7 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The previous research has provided us with references regarding the research subjects (pre-trained language models for ancient Chinese) and methods (complex networks). However, most studies [6][7][8][9][10][11][12] on probing language models have not comprehensively elucidated the patterns and principles about the organization of linguistic elements within the models. Therefore, we use complex networks to comprehensively study how these pre-trained language models organize linguistic elements at different levels.…”

Section: Complex Network In Language Systemmentioning

confidence: 99%

“…Thus, it is necessary to investigate the mechanisms by which these models ‘comprehend’ ancient Chinese. In fact, previous work has explored how pre-trained language models ‘learn’ linguistic knowledge, including probing lexical [6,7], syntactic [8,9] and semantic [10–12] knowledge encoded in the models. However, most of these works focus on probing the knowledge at certain aspects, without adopting a holistic perspective to study how the models simulate human language, or to discern potential patterns in how they organize the elements of ancient Chinese.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A complex network approach to analyse pre-trained language models for ancient Chinese

Zheng,

Xiao

2024

R. Soc. Open Sci.

View full text Add to dashboard Cite

Ancient Chinese is a splendid treasure within Chinese culture. To facilitate its compilation, pre-trained language models for ancient Chinese are developed. After that, researchers are actively exploring the factors contributing to their success. However, previous work did not study how language models organized the elements of ancient Chinese from a holistic perspective. Hence, we adopt complex networks to explore how language models organize the elements in ancient Chinese system. Specifically, we first analyse the characters’ and words’ co-occurrence networks in ancient Chinese. Then, we study characters’ and words’ attention networks, generated by attention heads within SikuBERT from two aspects: static and dynamic network analysis. In the static network analysis, we find that (i) most of attention networks exhibit small-world properties and scale-free behaviour, (ii) over 80% of attention networks exhibit high similarity with the corresponding co-occurrence networks, (iii) there exists a noticeable gap between characters’ and words’ attention networks across layers, while their fluctuations remain relatively consistent, and (iv) the attention networks generated by SikuBERT tend to be sparser compared with those from Chinese BERT. In dynamic network analysis, we find that the sentence segmentation task does not significantly affect network metrics, while the part-of-speech tagging task makes attention networks sparser.

show abstract

Section: Complex Network In Language Systemmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A complex network approach to analyse pre-trained language models for ancient Chinese

Zheng,

Xiao

2024

R. Soc. Open Sci.

View full text Add to dashboard Cite

show abstract

“…For instance, [23] uses it for sentiment analysis. [16] evaluates a Transformer's ability to learn Italian syntax. Finally, [6] proposes a chatbot that helps detect and classify fraud in a finance context.…”

Section: Introductionmentioning

confidence: 99%

AI2: The next leap toward native language based and explainable machine learning framework

Dessureault¹,

Massicotte²

2023

Preprint

View full text Add to dashboard Cite

The machine learning frameworks flourished in the last decades, allowing artificial intelligence to get out of academic circles to be applied to enterprise domains. This field has significantly advanced, but there is still some meaningful improvement to reach the subsequent expectations. The proposed framework, named AI 2 , uses a natural language interface that allows a non-specialist to benefit from machine learning algorithms without necessarily knowing how to program with a programming language. The primary contribution of the AI 2 framework allows a user to call the machine learning algorithms in English, making its interface usage easier. The second contribution is greenhouse gas (GHG) awareness. It has some strategies to evaluate the GHG generated by the algorithm to be called and to propose alternatives to find a solution without executing the energy-intensive algorithm. Another contribution is a preprocessing module that helps to describe and to load data properly. Using an English text-based chatbot, this module guides the user to define every dataset so that it can be described, normalized, loaded and divided appropriately. The last contribution of this paper is about explainability. For decades, the scientific community has known that machine learning algorithms imply the famous black-box problem. Traditional machine learning methods convert an input into an output without being able to justify this result. The proposed framework explains the algorithm's process with the proper texts, graphics and tables. The results, declined in five cases, present usage applications from the user's English command to the explained output. Ultimately, the AI 2 framework represents the next leap toward native language-based, human-oriented concerns about machine learning framework.

show abstract

“…The study by (de Vries, van Cranenburgh, and Nissim 2020) represents an exception in this context: the authors applied the probing task approach to compare the linguistic competence encoded by a Dutch BERT-based model and multilingual BERT (mBERT), showing that earlier layers of mBERT are consistently more informative that earlier layers of the monolingual model. (Guarasci et al 2021) applied instead the structural probe originally defined by (Hewitt and Manning 2019) on the representations of a pre-trained Italian BERT. Testing their approach on different subsets of the Italian Universal Dependency Treebank (IUDT), they showed on the one hand that the model is able to encode properties of syntax especially in its central-upper layers; on the other hand, that such embedded syntactic information can be used to successfully perform two specific syntactic tasks, i.e.…”

Section: Introductionmentioning

confidence: 99%

Probing Linguistic Knowledge in Italian Neural Language Models across Language Varieties

Miaschi¹,

Sarti²,

Brunato³

et al. 2022

ijcol

View full text Add to dashboard Cite

In this paper, we present an in-depth investigation of the linguistic knowledge encoded by the transformer models currently available for the Italian language. In particular, we investigate how the complexity of two different architectures of probing models affects the performance of the Transformers in encoding a wide spectrum of linguistic features. Moreover, we explore how this implicit knowledge varies according to different textual genres and language varieties.

show abstract

Assessing BERT’s ability to learn Italian syntax: a study on null-subject and agreement phenomena

Cited by 34 publications

References 43 publications

A complex network approach to analyse pre-trained language models for ancient Chinese

A complex network approach to analyse pre-trained language models for ancient Chinese

AI2: The next leap toward native language based and explainable machine learning framework

Probing Linguistic Knowledge in Italian Neural Language Models across Language Varieties

Contact Info

Product

Resources

About