SIGMORPHON–UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection

Kodner, Jordan; Khalifa, Salam; Batsuren, Khuyagbaatar; Dolatian, Hossep; Cotterell, Ryan; Akkuş, Faruk; Anastasopoulos, Antonios; Taras, Andrushko,; Arora, Aryaman; Atanalov, Nona; Bella, Gábor; Elena, Budianskaya,; Ate, Yustinus Ghanggo; Goldman, Omer; David, Guriel,; Guriel, Simon; Guriel-Agiashvili, Silvia; Kieraś, Witold; Krizhanovsky, Andrew; Krizhanovsky, Natalia; Marchenko, I. G.; Markowska, Magdalena; Polina, Mashkovtseva,; Nepomniashchaya, Maria; Rodionova, Daria; Scheifer, Karina; Sorova, Alexandra; Anastasia, Yemelina,; Young, Jeremiah; Vylomova, Ekaterina

doi:10.18653/v1/2022.sigmorphon-1.19

Cited by 3 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A breakdown of BLIND test triple by type in Table 2 replicates prior work (Kodner et al, 2022(Kodner et al, , 2023b demonstrating that generalization to unseen feature sets is particularly challenging. All systems showed lower accuracy on OOV feature sets (fsOOV & bothOOV) than on other triples.…”

Section: Feature Set Generalization In Blindsupporting

confidence: 58%

“…All data was adapted from Kodner et al (2023b), 1 which was in turn extracted from Uni-Morph 3 and 4 (McCarthy et al, 2020;Batsuren et al, 2022). The data was subjected to additional processing as described below.…”

Section: Languagesmentioning

confidence: 99%

“…BLIND: Following Kodner et al (2023b), this splitting strategy ensures that approximately 50% of test items contain OOV feature sets but is otherwise blind to the identity of those feature sets. This was shown in the 2022 SIGMORPHON-UniMorph shared task to create more opportunities for testing dimensions of generalization across feature sets than more traditional uniform random sampling (Kodner et al, 2022). en-PRS: This probe tests the implications of UniMorph's design choice of annotating the non-3rd singular present with NFIN rather than PRS.…”

Section: Data Splitsmentioning

confidence: 99%

“…The current practice in the evaluation of computational morphological inflection models, such as that employed in the SIGMORPHON, CoNLL-SIGMORPHON and SIGMORPHON-UniMorph shared tasks (Cotterell et al, 2016(Cotterell et al, , 2017McCarthy et al, 2019;Vylomova et al, 2020;Pimentel et al, 2021;Kodner et al, 2022) as well as in more targeted studies focused on specific languages or the generalization behavior of computational models (Goldman et al, 2022;Wiemerslage et al, 2022;Kodner et al, 2023b;Guriel et al, 2023;Kodner et al, 2023a), is to train on (lemma, inflection, features) triples and predict inflected forms from held-out (lemma, features) pairs. The algorithm for generating train-test splits is both random and language-independent, which has proven successful in distinguishing morphological inflection models at the gross quantitative level.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Exploring Linguistic Probes for Morphological Inflection

Kodner,

Khalifa,

Payne

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Modern work on the cross-linguistic computational modeling of morphological inflection has typically employed language-independent data splitting algorithms. In this paper, we supplement that approach with language-specific probes designed to test aspects of morphological generalization. Testing these probes on three morphologically distinct languages, English, Spanish, and Swahili, we find evidence that three leading morphological inflection systems employ distinct generalization strategies over conjugational classes and feature sets on both orthographic and phonologically transcribed inputs.

show abstract

Section: Feature Set Generalization In Blindsupporting

confidence: 58%

Section: Languagesmentioning

confidence: 99%

Section: Data Splitsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Exploring Linguistic Probes for Morphological Inflection

Kodner,

Khalifa,

Payne

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Full shifts may occur in language modelling tasks, where changes in the p(x) directly translate into changes in p(y|x) 12 , or when adapting to new language pairs in multi-lingual experiments (e.g. Costa-jussà et al, 2022;Kodner et al, 2022). Another case of full shift is the one in which entirely different types of data are used either for pretraining (e.g.…”

Section: Label Shiftmentioning

confidence: 99%

State-of-the-art generalisation research in NLP: A taxonomy and review

Hupkes¹,

Giulianelli²,

Dankers³

et al. 2022

Preprint

View full text Add to dashboard Cite

The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not well understood, nor are there any common standards to evaluate it. In this paper, we aim to lay the groundwork to improve both of these issues. We present a taxonomy for characterising and understanding generalisation research in NLP, we use that taxonomy to present a comprehensive map of published generalisation studies, and we make recommendations for which areas might deserve attention in the future. Our taxonomy is based on an extensive literature review of generalisation research, and contains five axes along which studies can differ: their main motivation, the type of generalisation they aim to solve, the type of data shift they consider, the source by which this data shift is obtained, and the locus of the shift within the modelling pipeline. We use our taxonomy to classify over 400 previous papers that test generalisation, for a total of more than 600 individual experiments. Considering the results of this review, we present an in-depth analysis of the current state of generalisation research in NLP, and make recommendations for the future. Along with this paper, we release a webpage where the results of our review can be dynamically explored, and which we intend to update as new NLP generalisation studies are published. With this work, we aim to make steps towards making state-of-the-art generalisation testing the new status quo in NLP.

show abstract

A taxonomy and review of generalization research in NLP

Hupkes,

Giulianelli,

Dankers

et al. 2023

Nat Mach Intell

View full text Add to dashboard Cite

The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

show abstract

SIGMORPHON–UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection

Abstract: The 2022 SIGMORPHON-UniMorph shared task on large scale morphological inflection generation included a wide range of typologically diverse languages: 33 languages from 11 top-level language families: Arabic (Modern Standard), Assamese, Braj, Chukchi, East-

Cited by 3 publications

References 23 publications

Exploring Linguistic Probes for Morphological Inflection

Exploring Linguistic Probes for Morphological Inflection

State-of-the-art generalisation research in NLP: A taxonomy and review

A taxonomy and review of generalization research in NLP

Contact Info

Product

Resources

About