Crosslingual Document Embedding as Reduced-Rank Ridge Regression

Abstract

There has recently been much interest in extending vector-based word representations to multiple languages, such that words can be compared across languages. In this paper, we shift the focus from words to documents and introduce a method for embedding documents written in any language into a single, language-independent vector space. For training, our approach leverages a multilingual corpus where the same concept is covered in multiple languages (but not necessarily via exact translations), such as Wikipedia. Our method, Cr5 (Crosslingual reduced-rank ridge regression), starts by training a ridge-regression-based classifier that uses languagespecific bag-of-word features in order to predict the concept that a given document is about. We show that, when constraining the learned weight matrix to be of low rank, it can be factored to obtain the desired mappings from language-specific bags-of-words to language-independent embeddings. As opposed to most prior methods, which use pretrained monolingual word vectors, postprocess them to make them crosslingual, and finally average word vectors to obtain document vectors, Cr5 is trained end-to-end and is thus natively crosslingual as well as document-level. Moreover, since our algorithm uses the singular value decomposition as its core operation, it is highly scalable. Experiments show that our method achieves state-of-the-art performance on a crosslingual document retrieval task. Finally, although not trained for embedding sentences and words, it also achieves competitive performance on crosslingual sentence and word retrieval tasks.

NEMO: Frequentist Inference Approach to Constrained Linguistic Typology Feature Prediction in SIGTYP 2020 Shared Task

“…We further note that the relatively good performance of ridge regression and random forest classifiers on the development data without any hyperparameter fine-tuning may possibly be explained by the relative robustness of both methods to sparseness and collinearity effects (which severely affect other types of parametric and non-parametric predictors in our experiment), as previously analyzed in detail by Tomaschek et al (2018) and observed by others in a typological setting (Burdick et al, 2020) and elsewhere (Josifoski et al, 2019). We briefly reevaluate these assumptions on the released test set in Section 4.6.…”

NEMO: Frequentist Inference Approach to Constrained Linguistic Typology Feature Prediction in SIGTYP 2020 Shared Task

“…We further note that the relatively good per formance of ridge regression and random forest classifiers on the development data without any hyperparameter finetuning may possibly be ex plained by the relative robustness of both meth ods to sparseness and collinearity effects (which severely affect other types of parametric and non parametric predictors in our experiment), as pre viously analyzed in detail by Tomaschek et al (2018) and observed by others in a typological set ting (Burdick et al, 2020) and elsewhere (Josifoski et al, 2019). We briefly reevaluate these assump tions on the released test set in Section 4.6.…”

Crosslingual Topic Modeling with WikiPDA

“…We presented Wikipedia-based Polyglot Dirichlet Allocation (WikiPDA), a novel crosslingual topic model for Wikipedia. Our human evaluation showed that the topics learned from 28 languages are as coherent as those learned from English alone, and more coherent than those from text-based LDA on English, a noteworthy finding, given that other crosslingual tasks have suffered by adding languages to the training set (Josifoski et al, 2019). We demonstrated WikiPDA's practical utility in several example applications and highlighted its capability for zero-shot language transfer.…”

“…Comparing model classes 1 and 3, we find that the dense WikiPDA model for 28 languages performed indistinguishably from the dense model for English only; i.e., adding more languages did not make the topics less coherent. This outcome is noteworthy, since on other crosslingual tasks (e.g., document retrieval), performance on a fixed testing language decreased when adding languages to the training set (Josifoski et al, 2019).…”