Finding next of kin: Cross-lingual embedding spaces for related languages

Sharoff, Serge

doi:10.1017/s1351324919000354

Cited by 8 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It will also lead to integrating training of the transformer with the semantic classification task on a deeper level, which can be accomplished by customizing its pre-training (weight-initialization) algorithm to include word semantic information available from existing taxonomies, which we are planning to undertake in future, along with experimenting with cross-lingual knowledge transfer (e.g. [22]), when a model uses English data to predict semantic relations in other, less resourced, languages.…”

Section: Discussionmentioning

confidence: 99%

Recognizing Semantic Relations: Attention-Based Transformers vs. Recurrent Models

Roussinov

Sharoff

Puchnina

2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Automatically recognizing an existing semantic relation (such as "is a", "part of", "property of", "opposite of" etc.) between two arbitrary words (phrases, concepts, etc.) is an important task affecting many information retrieval and artificial intelligence tasks including query expansion, common-sense reasoning, question answering, and database federation. Currently, two classes of approaches exist to classify a relation between words (concepts) X and Y: (1) path-based and (2) distributional. While the path-based approaches look at word-paths connecting X and Y in text, the distributional approaches look at statistical properties of X and Y separately, not necessary in the proximity of each other. Here, we suggest how both types can be improved and empirically compare them using several standard benchmarking datasets. For our distributional approach, we are suggesting using an attentionbased transformer. While they are known to be capable of supporting knowledge transfer between different tasks, and recently set a number of benchmarking records in various applications, we are the first to successfully apply them to the task of recognizing semantic relations. To improve a path-based approach, we are suggesting our original neural word path model that combines useful properties of convolutional and recurrent networks, and thus addressing several shortcomings from the prior path-based models. Both our models significantly outperforms the state-of-the-art within its type accordingly. Our transformer-based approach outperforms current state-of-the-art by 1-12% points on 4 out of 6 standard benchmarking datasets. This results in 15-40% error reduction and is closing the gap between the automated and human performance by up to 50%. It also needs much less training data than prior approaches. For the ease of re-producing our results, we make our source code and trained models publicly available.During the last few years, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have resulted in major breakthroughs and are behind

show abstract

Section: Discussionmentioning

confidence: 99%

Recognizing Semantic Relations: Attention-Based Transformers vs. Recurrent Models

Roussinov

Sharoff

Puchnina

2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…The pretrained Rusyn embeddings still need to be projected into the Panslav5 embedding space. To this end, we use a simpler procedure than Sharoff (2018) and take the intersection of word forms of the two files as the seed lexicon (i.e., we assume that words spelled the same should occur in the same area of the vector space). Almost half of the forms present in the Rusyn Fasttext embeddings are covered by this lexicon.…”

Section: Pretrained Rusyn Embeddingsmentioning

confidence: 99%

“…Most approaches start with a seed dictionary containing word pairs of both languages, but some variants merely rely on identical tokens such as punctuation signs, numerals, or named entities (Artetxe, Labaka and Agirre 2017). More relevant to our setting is the work by Sharoff (2018) on related languages: he starts by automatically extracting a seed dictionary from Wikipedia page titles and uses these entries to determine edit distance weights for the language pair in question, assuming that most word pairs are cognates. Weighted Levenshtein distance is then included as a factor in the word embedding projection algorithm.…”

Section: Previous Workmentioning

confidence: 99%

“…Weighted Levenshtein distance is then included as a factor in the word embedding projection algorithm. In this way, Sharoff (2018) creates a “Panslavonic” word embedding space for seven Slavic languages, among which are Czech, Polish, Russian, Slovak, and Ukrainian.…”

Section: Previous Workmentioning

confidence: 99%

“…Instead of training the word embeddings on the rather small tagging data set, we can take advantage of word embeddings that have been pretrained on large raw corpora of the source languages. Sharoff (2018) takes the Fasttext (Bojanowski et al 2017) embeddings as a starting point for his work; these embeddings have been independently trained on lowercased Wikipedia data and contain 300 dimensions. Sharoff (2018) converts the independently trained embeddings of all Slavonic languages into a shared vector space using known conversion methods and automatically induced bilingual dictionaries.…”

Section: Word Embeddingsmentioning

confidence: 99%

See 2 more Smart Citations

Neural morphosyntactic tagging for Rusyn

Scherrer

Rabus

2019

Nat. Lang. Eng.

View full text Add to dashboard Cite

The paper presents experiments on part-of-speech and full morphological tagging of the Slavic minority language Rusyn. The proposed approach relies on transfer learning and uses only annotated resources from related Slavic languages, namely Russian, Ukrainian, Slovak, Polish, and Czech. It does not require any annotated Rusyn training data, nor parallel data or bilingual dictionaries involving Rusyn. Compared to earlier work, we improve tagging performance by using a neural network tagger and larger training data from the neighboring Slavic languages. We experiment with various data preprocessing and sampling strategies and evaluate the impact of multitask learning strategies and of pretrained word embeddings. Overall, while genre discrepancies between training and test data have a negative impact, we improve full morphological tagging by 9% absolute micro-averaged F1 as compared to previous research.

show abstract