Many Languages, One Parser

Abstract

We train one multilingual model for dependency parsing and use it to parse sentences in several languages. The parsing model uses (i) multilingual word clusters and embeddings; (ii) token-level language information; and (iii) language-specific features (finegrained POS tags). This input representation enables the parser not only to parse effectively in multiple languages, but also to generalize across languages based on linguistic universals and typological similarities, making it more effective to learn from limited annotations. Our parser's performance compares favorably to strong baselines in a range of data scenarios, including when the target language has a large treebank, a small treebank, or no treebank for training.

Choosing Transfer Languages for Cross-Lingual Learning

“…Tsvetkov et al (2016b) used typological information in the target language as additional input to their model for phonetic representation learning. Ammar et al (2016) and Although not for cross-lingual transfer, there has been prior work on data selection for training models. Tsvetkov et al (2016a) and Ruder and Plank (2017) use Bayesian optimization for data selection.…”

“…A common challenge in applying natural language processing (NLP) techniques to low-resource languages is the lack of training data in the languages in question. It has been demonstrated that through cross-lingual transfer, it is possible to leverage one or more similar high-resource languages to improve the performance on the low-resource languages in several NLP tasks, including machine score(L tf,1 , L tk ) score (L tf,2 , L tk translation (Zoph et al, 2016;Johnson et al, 2017;Nguyen and Chiang, 2017;Neubig and Hu, 2018), parsing (Täckström et al, 2012;Ammar et al, 2016;Ahmad et al, 2019;, partof-speech or morphological tagging (Täckström et al, 2013;Cotterell and Heigold, 2017;Malaviya et al, 2018;Plank and Agić, 2018), named entity recognition (Zhang et al, 2016;Mayhew et al, 2017;Xie et al, 2018), and entity linking (Tsai and Roth, 2016;Rijhwani et al, 2019). There are many methods for performing this transfer, including joint training (Ammar et al, 2016;Tsai and Roth, 2016;Cotterell and Heigold, 2017;Johnson et al, 2017;Malaviya et al, 2018), annotation projection (Täckström et al, 2012;Täckström et al, 2013;Zhang et al, 2016;Plank and Agić, 2018), fine-tuning (Zoph et al, 2016;Neubig and Hu, 2018), data augmentation (Mayhew et al, 2017), or zero-shot transfer (Ahmad et al, 2019;Xie et al, 2018;Neubig and Hu, 2018;Rijhwani et al, 2019).…”

“…However, it is not always true that all languages in a single language family share the same linguistic properties (Ahmad et al, 2019). Therefore, another strategy is to select transfer languages based on the typological properties that are relevant to the specific NLP task, such as word ordering for parsing tasks (Ammar et al, 2016;Ahmad et al, 2019). With several heuristics available for selecting a transfer language, it is unclear a priori if any single attribute of a language will be the most reliable criterion in determining whether cross-lingual learning is likely to work for a specific NLP task.…”

Working Hard or Hardly Working: Challenges of Integrating Typology into Neural Dependency Parsers

“…Typology as Quantization: Adding simple, discrete language identifiers to the input has been shown to be useful in multi-task multi-lingual settings (Ammar et al, 2016;Johnson et al, 2017).…”

Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing

“…Finally, prior studies have noticed that the word order information is significant for parsing and use it as features (Ammar et al, 2016;Naseem et al, 2012;Rasooli and Collins, 2017;Zhang and Barzilay, 2015;Dryer, 2007). further propose to decompose these features from models for adapting target languages.…”