Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers

Repo, Liina; Skantsi, Valtteri; Rönnqvist, Samuel; Hellström, Saara; Oinonen, Miika; Salmela, Anna; Biber, Douglas; Egbert, Jesse; Pyysalo, Sampo; Laippala, Veronika

doi:10.18653/v1/2021.eacl-srw.24

Cited by 8 publications

(20 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3 The micro average F1-score was a good 68% on average. Similar to previous studies (Repo et al, 2021;Biber and Egbert, 2016), we observed a large variation among classes, ranging from an F1-score of 44% (Informational persuasion) to 81% (Lyrical). Our method was able to extract stable keywords for all the classes expect for Spoken, where no keyword candidate passed the selection frequency threshold.…”

Section: Predictive Performancesupporting

confidence: 90%

“…As a classifier, we use the XLM-R deep language model (Conneau et al, 2020) because of its strong ability to model multiple languages, both in monolingual and cross-lingual settings. We opt for the base size rather than the large, due to its relatively frugal use of resource and comparable predictive performance on CORE (Repo et al, 2021). The task is modeled as a multilabel classification task using a sequence classification head, binary crossentry with sigmoid loss and a fixed prediction threshold.…”

Section: Classifier and Attribution Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Explaining Classes through Stable Word Attributions

Rönnqvist¹,

Kyröläinen²,

Myntti³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

Input saliency methods have recently become a popular tool for explaining predictions of deep learning models in NLP. Nevertheless, there has been little work investigating methods for aggregating prediction-level explanations to the class level, nor has a framework for evaluating such class explanations been established. We explore explanations based on XLM-R and the Integrated Gradients input attribution method, and propose 1) the Stable Attribution Class Explanation method (SACX) to extract keyword lists of classes in text classification tasks, and 2) a framework for the systematic evaluation of the keyword lists. We find that explanations of individual predictions are prone to noise, but that stable explanations can be effectively identified through repeated training and explanation. We evaluate on web register data and show that the class explanations are linguistically meaningful and distinguishing of the classes.

show abstract

Section: Predictive Performancesupporting

confidence: 90%

Section: Classifier and Attribution Methodsmentioning

confidence: 99%

Explaining Classes through Stable Word Attributions

Rönnqvist¹,

Kyröläinen²,

Myntti³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

show abstract

“…In this regard, further studies are needed to develop resources for Web registers in languages other than English. Indeed, recent studies suggest that cross-lingual modeling of Web registers is achievable (Repo et al, 2021;Rönnqvist et al, 2021), but it is unclear to what extent cultural differences affect the representation of Web registers and even the very existence of specific registers. Therefore, in the future, extending the research of Web registers to a widely multilingual setting would be greatly beneficial.…”

Section: General Discussion and Conclusionmentioning

confidence: 99%

“…For instance, Multilingual BERT (Devlin et al, 2018) and XLM-R (Conneau et al, 2020) provide cross-lingual language models that can be fine-tuned to model data in multilingual settings. These have also been applied successfully to register identification (Repo et al, 2021;Rönnqvist et al, 2021). Another line of modeling endeavor has focused on smaller and faster models, such as DistilBERT, which have been created to overcome the computational challenges related to the original version (Sanh et al, 2019).…”

Section: Pretrained Language Models and Transfer Learningmentioning

confidence: 99%

“…This also ensured that the model performance reported in this study is comparable across languages when modeling register categories based on the taxonomy presented in CORE. Currently, as part of an on-going project, corpora has been released for three languages that follow the register taxonomy presented in CORE (Repo et al, 2021;Rönnqvist et al, 2021). Additionally, we are in the process of releasing corpora for over 10 languages.…”

Section: Classifiersmentioning

confidence: 99%

See 1 more Smart Citation

Register identification from the unrestricted open Web using the Corpus of Online Registers of English

Laippala

Rönnqvist

Oinonen

et al. 2022

Lang Resources & Evaluation

View full text Add to dashboard Cite

This article examines the automatic identification of Web registers, that is, text varieties such as news articles and reviews. Most studies have focused on corpora restricted to include only preselected classes with well-defined characteristics. These corpora feature only a subset of documents found on the unrestricted open Web, for which register identification has been particularly difficult because the range of linguistic variation on the Web is known to be substantial. As part of this study, we present the first open release of the Corpus of Online Registers of English (CORE), which is drawn from the unrestricted open Web and, currently, is the largest collection of manually annotated Web registers. Furthermore, we demonstrate that the CORE registers can be automatically identified with competitive results, with the best performance being an F1-score of 68% with the deep learning model BERT. The best performance was achieved using two modeling strategies. The first one involved modeling the registers using propagated register labels, that is, repeating the main register label along with its corresponding subregister label in a multilabel model. In the second one, we explored how the length of the document affects model performance, discovering that the beginning provided superior classification accuracy. Overall, the current study presents a systematic approach for the automatic identification of a large number of Web registers from the unrestricted Web, hence providing new pathways for future studies.

show abstract

Automatic genre identification: a survey

Kuzman,

Ljubešić

2023

Lang Resources & Evaluation

View full text Add to dashboard Cite

Automatic genre identification (AGI) is a text classification task focused on genres, i.e., text categories defined by the author’s purpose, common function of the text, and the text’s conventional form. Obtaining genre information has been shown to be beneficial for a wide range of disciplines, including linguistics, corpus linguistics, computational linguistics, natural language processing, information retrieval and information security. Consequently, in the past 20 years, numerous researchers have collected genre datasets with the aim to develop an efficient genre classifier. However, their approaches to the definition of genre schemata, data collection and manual annotation vary substantially, resulting in significantly different datasets. As most AGI experiments are dataset-dependent, a sufficient understanding of the differences between the available genre datasets is of great importance for the researchers venturing into this area. In this paper, we present a detailed overview of different approaches to each of the steps of the AGI task, from the definition of the genre concept and the genre schema, to the dataset collection and annotation methods, and, finally, to machine learning strategies. Special focus is dedicated to the description of the most relevant genre schemata and datasets, and details on the availability of all of the datasets are provided. In addition, the paper presents the recent advances in machine learning approaches to automatic genre identification, and concludes with proposing the directions towards developing a stable multilingual genre classifier.

show abstract

Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers

Cited by 8 publications

References 22 publications

Explaining Classes through Stable Word Attributions

Explaining Classes through Stable Word Attributions

Register identification from the unrestricted open Web using the Corpus of Online Registers of English

Automatic genre identification: a survey

Contact Info

Product

Resources

About