Align and Copy: UZH at SIGMORPHON 2017 Shared Task for
            Morphological Reinflection

Makarov, Peter; Ruzsics, Tatiana; Clematide, Simon

doi:10.18653/v1/k17-2004

Cited by 53 publications

(63 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another idea relevant to explore in future work is to consider the networks that are designed to be strong at character copying which is the most common operation in string transduction tasks such as morphological segmentation, morphological reinflection and normalization (Gu et al, 2016;See et al, 2017;Makarov et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

Neural Sequence-to-sequence Learning of Internal Word Structure

Ruzsics¹,

Samardžić

2017

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)

View full text Add to dashboard Cite

Learning internal word structure has recently been recognized as an important step in various multilingual processing tasks and in theoretical language comparison. In this paper, we present a neural encoder-decoder model for learning canonical morphological segmentation. Our model combines character-level sequence-to-sequence transformation with a language model over canonical segments. We obtain up to 4% improvement over a strong character-level encoderdecoder baseline for three languages. Our model outperforms the previous state-ofthe-art for two languages, while eliminating the need for external resources such as large dictionaries. Finally, by comparing the performance of encoder-decoder and classical statistical machine translation systems trained with and without corpus counts, we show that including corpus counts is beneficial to both approaches.

show abstract

Section: Discussionmentioning

confidence: 99%

Neural Sequence-to-sequence Learning of Internal Word Structure

Ruzsics¹,

Samardžić

2017

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)

View full text Add to dashboard Cite

show abstract

“…For the SIGMORPHON 2016 and the CoNLL-SIGMORPHON 2017 shared tasks (Cotterell et al, , 2017, multiple MRI systems were developed, e.g., (Nicolai et al, 2016;Taji et al, 2016;Kann and Schütze, 2016;Östling, 2016;Makarov et al, 2017). Encoder-decoder neural networks (Cho et al, 2014a; performed best, such that we extend them in this work.…”

Section: Related Workmentioning

confidence: 91%

Proceedings of the First Workshop on Subword and Character Level Models in NLP

2017

View full text Add to dashboard Cite

IntroductionTraditional NLP starts with a hand-engineered layer of representation, the level of tokens or words. A tokenization component first breaks up the text into units using manually designed rules. Tokens are then processed by components such as word segmentation, morphological analysis and multiword recognition. The heterogeneity of these components makes it hard to create integrated models of both structure within tokens (e.g., morphology) and structure across multiple tokens (e.g., multi-word expressions). This approach can perform poorly (i) for morphologically rich languages, (ii) for noisy text, (iii) for languages in which the recognition of words is difficult and (iv) for adaptation to new domains; and (v) it can impede the optimization of preprocessing in end-to-end learning.The workshop provides a forum for discussing recent advances as well as future directions on sub-word and character-level natural language processing and representation learning that address these problems.We received 37 submissions, out of which we accepted 24 as papers and 4 as extended abstracts. AbstractMost of neural language models use different kinds of embeddings for word prediction. While word embeddings can be associated to each word in the vocabulary or derived from characters as well as factored morphological decomposition, these word representations are mainly used to parametrize the input, i.e. the context of prediction. This work investigates the effect of using subword units (character and factored morphological decomposition) to build output representations for neural language modeling. We present a case study on Czech, a morphologically-rich language, experimenting with different input and output representations. When working with the full training vocabulary, despite unstable training, our experiments show that augmenting the output word representations with character-based embeddings can significantly improve the performance of the model. Moreover, reducing the size of the output look-up table, to let the character-based embeddings represent rare words, brings further improvement. IntroductionMost of neural language models, such as n-gram models (Bengio et al., 2003) are word based and rely on the definition of a finite vocabulary V. Therefore, a look-up table maps each wordw ∈ V to a vector of real features, and is stored in a matrix. While this approach yields significant improvement for a variety of tasks and languages, see for instance (Schwenk, 2007) in speech recognition and (Le et al., 2012; Devlin et al., 2014; in machine translation, it induces several limitations.For morphologically-rich languages, like Czech or German, the lexical coverage is still an important issue, since there is a combinatorial explosion of word forms, most of which are hardly observed on training data. On the one hand, growing the look-up table is not a solution, since it would increase the number of parameters without having enough training examples for a proper estimation. On the other hand, rare words can be replaced...

show abstract

“…The choice of attention has a smaller impact. In both data settings, our best model on the validation set outperforms all submissions from the 2018 shared task except for UZH (Makarov and Clematide, 2018), which uses a more involved imitation learning approach and larger ensembles. In contrast, our only departure from standard seq2seq training is the drop-in replacement of softmax by entmax.…”

Section: Morphological Inflectionmentioning

confidence: 94%

Sparse Sequence-to-Sequence Models

Peters¹,

Niculae²,

Martins³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

131

112

View full text Add to dashboard Cite

Sequence-to-sequence models are a powerful workhorse of NLP. Most variants employ a softmax transformation in both their attention mechanism and output layer, leading to dense alignments and strictly positive output probabilities. This density is wasteful, making models less interpretable and assigning probability mass to many implausible outputs. In this paper, we propose sparse sequence-to-sequence models, rooted in a new family of α-entmax transformations, which includes softmax and sparsemax as particular cases, and is sparse for any α > 1. We provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. Our models are able to produce sparse alignments and to assign nonzero probability to a short list of plausible outputs, sometimes rendering beam search exact. Experiments on morphological inflection and machine translation reveal consistent gains over dense models. arXiv:1905.05702v2 [cs.CL] 12 Jun 2019 the encodings [h 1 , . . . , h J ], using s t as a query vector. This is done by computing token-level scores z j := s t W (z) h j , then taking a weighted averageπ j h j , where π := softmax(z).(1) the anonymous reviewers, for helpful discussion and feedback.

show abstract

Align and Copy: UZH at SIGMORPHON 2017 Shared Task for Morphological Reinflection

Cited by 53 publications

References 14 publications

Neural Sequence-to-sequence Learning of Internal Word Structure

Neural Sequence-to-sequence Learning of Internal Word Structure

Proceedings of the First Workshop on Subword and Character Level Models in NLP

Sparse Sequence-to-Sequence Models

Contact Info

Product

Resources

About