On Biasing Transformer Attention Towards Monotonicity

Rios, Annette; Amrhein, Chantal; Aepli, Noëmi; Sennrich, Rico

doi:10.18653/v1/2021.naacl-main.354

Cited by 7 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we have evaluated the most common and most popular model architectures, but it would be interesting to test specific model architectures for character transduction tasks, e.g. models that put some monotonicity constraint on the attention mechanism (Wu and Cotterell, 2019;Rios et al, 2021). We defer this to future work.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation

Kuparinen,

Miletić,

Scherrer

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Text normalization methods have been commonly applied to historical language or usergenerated content, but less often to dialectal transcriptions. In this paper, we introduce dialect-to-standard normalization -i.e., mapping phonetic transcriptions from different dialects to the orthographic norm of the standard variety -as a distinct sentence-level character transduction task and provide a large-scale analysis of dialect-to-standard normalization methods. To this end, we compile a multilingual dataset covering four languages: Finnish, Norwegian, Swiss German and Slovene. For the two biggest corpora, we provide three different data splits corresponding to different use cases for automatic normalization. We evaluate the most successful sequence-to-sequence model architectures proposed for text normalization tasks using different tokenization approaches and context sizes. We find that a characterlevel Transformer trained on sliding windows of three words works best for Finnish, Swiss German and Slovene, whereas the pre-trained byT5 model using full sentences obtains the best results for Norwegian. Finally, we perform an error analysis to evaluate the effect of different data splits on model performance.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Furthermore, specific model architectures for character transduction tasks have been proposed, e.g. constraining the attention to be monotonic (Wu and Cotterell, 2019;Rios et al, 2021). We did not include such architectures in our experiments since they generally only showed marginal improvements.…”

Section: Limitationsmentioning

confidence: 99%

Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation

Kuparinen,

Miletić,

Scherrer

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…Another modification to the transformer architecture, which can improve performance on morphology tasks, is to add a so-called monotonicity loss (Rios et al, 2021). This can bias the transformer toward near-monotonic alignment between the input and output sequence, which is often the case in inflection.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

2022

View full text Add to dashboard Cite

Speech consists of a continuously-varying acoustic signal. Yet human listeners experience it as sequences of discrete speech sounds, which are used to recognise words. To examine how the human brain appropriately sequences the speech signal, we recorded two-hour magnetoencephalograms from 21 subjects listening to short narratives. Our analyses show that the brain continuously encodes the three most recently heard speech sounds in parallel, and maintains this information long past the sensory input. Each speech sound has a representation that evolves over time, jointly encoding both its phonetic features and time elapsed since onset. This allows the brain to represent the relative order and phonetic content of the phonetic sequence. These dynamic representations are active earlier when phonemes are more predictable, and are sustained longer when lexical identity is uncertain. The flexibility in the dynamics of these representations paves the way for further understanding of how such sequences may be used to interface with higher order structure such as morphemes and words.Bio: Laura Gwilliams received her PhD in Psychology with a focus in Cognitive Neuroscience from New York University in May 2020. Currently she is a post-doctoral researcher at UCSF, using MEG and ECoG data to understand how linguistic structures are parsed and composed while listening to continuous speech. The ultimate goal of Laura's research is to describe speech comprehension in terms of what operations are applied to the acoustic signal; which representational formats are generated and manipulated (e.g. phonetic, syllabic, morphological), and under what processing architecture.

show abstract

“…Isolated morphological analysis and reinflection have long been of interest to the NLP community, with yearly shared tasks (Kurimo et al, 2010;Vylomova et al, 2020) that result in dedicated architectures that perform well for many languages (Aharoni and Goldberg, 2017;Makarov and Clematide, 2018;Wu and Cotterell, 2019;Rios et al, 2021). However, despite the large morphological diversity of natural languages, many approaches for sentence-level sequence-to-sequence tasks are often only tested on a subset of (morphologically similar) languages and then adopted without much questioning (Bender, 2011).…”

Section: Related Workmentioning

confidence: 99%

How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?

Amrhein¹,

Sennrich²

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

Self Cite

View full text Add to dashboard Cite

Data-driven subword segmentation has become the default strategy for open-vocabulary machine translation and other NLP tasks, but may not be sufficiently generic for optimal learning of non-concatenative morphology. We design a test suite to evaluate segmentation strategies on different types of morphological phenomena in a controlled, semisynthetic setting. In our experiments, we compare how well machine translation models trained on subword-and character-level can translate these morphological phenomena. We find that learning to analyse and generate morphologically complex surface representations is still challenging, especially for nonconcatenative morphological phenomena like reduplication or vowel harmony and for rare word stems. Based on our results, we recommend that novel text representation strategies be tested on a range of typologically diverse languages to minimise the risk of adopting a strategy that inadvertently disadvantages certain languages. 1 Compounding Circumfixation Infixation Vowel Harmony Reduplication (German) (Chickasaw) (Bontoc) (Turkish) (Itza') Schild / Kröte lakna fikas üzüldünüz tz'eek 'shield' / 'toad' 'it is yellow' 'strong' üz-ül-dü-nüz 'few' Schildkröte iklakno fumikas sadden-PASS-PAST-2PL tz'eek-tz'eek 'turtle' 'it isn't yellow' 'to be strong' 'You became sad.' 'very few'

show abstract

On Biasing Transformer Attention Towards Monotonicity

Cited by 7 publications

References 23 publications

Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation

Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation

Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?

Contact Info

Product

Resources

About