Cross-Lingual Dependency Parsing with Unlabeled Auxiliary Languages

Abstract

Cross-lingual transfer learning has become an important weapon to battle the unavailability of annotated resources for low-resource languages. One of the fundamental techniques to transfer across languages is learning language-agnostic representations, in the form of word embeddings or contextual encodings. In this work, we propose to leverage unannotated sentences from auxiliary languages to help learning language-agnostic representations. Specifically, we explore adversarial training for learning contextual encoders that produce invariant representations across languages to facilitate cross-lingual transfer. We conduct experiments on cross-lingual dependency parsing where we train a dependency parser on a source language and transfer it to a wide range of target languages. Experiments on 28 target languages demonstrate that adversarial training significantly improves the overall transfer performances under several different settings. We conduct a careful analysis to evaluate the language-agnostic representations resulted from adversarial training.

PPT: Parsimonious Parser Transfer for Unsupervised Cross-Lingual Adaptation

“…This finding means that if the target language has a closely related high-resource language, it may be better to transfer from that language as the source and use PPT for adaptation. Against AHMAD (Ahmad et al, 2019b), PPT performs better on 4 out of 6 distant languages. On nearby languages, the average UAS of PPT is higher, and the average LAS is on par.…”

“…First, HE is a neural lexicalised DMV parser with normalising flow that uses a language modelling objective when fine-tuning on the unlabelled target language data (He et al, 2019). Second, AHMAD is an adversarial training method that attempts to learn language-agnostic representations (Ahmad et al, 2019b). Lastly, MENG is a constrained inference method that derives constraints from the target corpus statistics to aid inference (Meng et al, 2019).…”

“…Thus, this technique also allows for source-free transfer. A different method is proposed by Ahmad et al (2019b) who explored the use of unlabelled data from an auxiliary language, which can be different from the target language. They employed adversarial training to learn language-agnostic representations.…”

“…Direct transfer is attractive as it does not require labelled target language data, rendering the approach fully unsupervised. 2 Recent work has shown that it is possible to outperform direct transfer if unlabelled data, either in the target lan-guage or a different auxiliary language, is available (He et al, 2019;Meng et al, 2019;Ahmad et al, 2019b). Here, we focus on the former setting and present flexible methods that can adapt a pre-trained parser given unlabelled target data.…”

Alignment-free Cross-lingual Semantic Role Labeling

“…Model transfer does not require parallel corpora or word alignment tools; nevertheless, it relies on accurate features such as POS tags (McDonald et al, 2013) or syntactic parse trees (Kozhevnikov and Titov, 2013) to enhance the ability to generalize across languages. Adversarial training is commonly used to extract language-agnostic features thereby improving the performance of cross-lingual systems (Chen et al, 2019;Ahmad et al, 2019b).…”