In Neural Machine Translation, What Does Transfer Learning Transfer?

Aji, Alham Fikri; Bogoychev, Nikolay; Heafield, Kenneth; Sennrich, Rico

doi:10.18653/v1/2020.acl-main.688

Cited by 38 publications

(22 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Advances in 'transfer learning' may help here (Nguyen & Chiang 2017;Aji et al 2020), as well as less supervised MT (Artetxe el al. 2018).…”

Section: Translation Of Textsmentioning

confidence: 99%

The Dawn of the Human-Machine Era: A forecast of new and emerging language technologies

Sayers¹,

Sousa‐Silva²,

Höhn³

et al. 2021

View full text Add to dashboard Cite

New language technologies are coming, thanks to the huge and competing private investment fuelling rapid progress; we can either understand and foresee their effects, or be taken by surprise and spend our time trying to catch up. This report scketches out some transformative new technologies that are likely to fundamentally change our use of language. Some of these may feel unrealistically futuristic or far-fetched, but a central purpose of this report - and the wider LITHME network - is to illustrate that these are mostly just the logical development and maturation of technologies currently in prototype. But will everyone benefit from all these shiny new gadgets? Throughout this report we emphasise a range of groups who will be disadvantaged and issues of inequality. Important issues of security and privacy will accompany new language technologies. A further caution is to re-emphasise the current limitations of AI. Looking ahead, we see many intriguing opportunities and new capabilities, but a range of other uncertainties and inequalities. New devices will enable new ways to talk, to translate, to remember, and to learn. But advances in technology will reproduce existing inequalities among those who cannot afford these devices, among the world’s smaller languages, and especially for sign language. Debates over privacy and security will flare and crackle with every new immersive gadget. We will move together into this curious new world with a mix of excitement and apprehension - reacting, debating, sharing and disagreeing as we always do. Plug in, as the human-machine era dawns.

show abstract

“…Advances in 'transfer learning' may help here (Nguyen & Chiang 2017;Aji et al 2020), as well as less supervised MT (Artetxe el al. 2018).…”

Section: Translation Of Textsmentioning

confidence: 99%

The Dawn of the Human-Machine Era: A forecast of new and emerging language technologies

Sayers¹,

Sousa‐Silva²,

Höhn³

et al. 2021

View full text Add to dashboard Cite

show abstract

“…For our transfer baseline without romanization, we merge our bilingual baseline vocabulary with that of the parent model following previous work (Aji et al, 2020;Kocmi and Bojar, 2020 (Papineni et al, 2002) of the multilingual pretrained models trained on original scripts (orig), romanized with uroman and uconv.…”

Section: Vocabulary Transfermentioning

confidence: 99%

“…While transfer learning has shown great promise, the transfer between languages with different scripts brings additional challenges. For a successful transfer of the embedding layer, both the parent and the child model should use the same or a partially overlapping vocabulary (Aji et al, 2020). It is common to merge the two vocabularies by aligning identical subwords and randomly assigning the remaining subwords from the child vocabulary to positions in the parent vocabulary (Lakew et al, 2018(Lakew et al, , 2019Kocmi and Bojar, 2020).…”

Section: Introductionmentioning

confidence: 99%

On Romanization for Model Transfer Between Scripts in Neural Machine Translation

Amrhein¹,

Sennrich²

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

Self Cite

View full text Add to dashboard Cite

Transfer learning is a popular strategy to improve the quality of low-resource machine translation. For an optimal transfer of the embedding layer, the child and parent model should share a substantial part of the vocabulary. This is not the case when transferring to languages with a different script. We explore the benefit of romanization in this scenario. Our results show that romanization entails information loss and is thus not always superior to simpler vocabulary transfer methods, but can improve the transfer between related languages with different scripts. We compare two romanization tools and find that they exhibit different degrees of information loss, which affects translation quality. Finally, we extend romanization to the target side, showing that this can be a successful strategy when coupled with a simple deromanization model.

show abstract

“…This make the model lack of rationality and interpretability to a certain extent. For each prescription, a set of symptoms is regarded as a source sentence to be translated to the target sequence that is a group of herbs, via the RNN based Seq2Seq model, which achieves tremendous performance in the task of machine translation [37]. This model is consisting of an encoder and a decoder based on RNN which encodes the source sequence of tokens into the latent space and then iteratively decoded to each word in the target sentence.…”

Section: Introductionmentioning

confidence: 99%

TPGen: Prescription Generation Using Knowledge-guided Translator

Ruan¹,

Hua²,

Wu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: Prescriptions contain a lot of clinical information and play a pivotal role in the clinical diagnosis of Traditional Chinese Medicine (TCM), which is a combination of herb to treat the symptoms of a patient from decision-making of doctors. In the process of clinical decision-making, a large number of prescriptions have been invented and accumulated based on TCM theories. Mining complex and the regular relationships between symptoms and herbs in the prescriptions are important for both clinical practice and novel prescription development. Previous work used several machine learning methods to discover regularities and generate prescriptions but rarely used TCM knowledge to guide prescription generation and described why each herb is predicted for treating a symptom. Methods: In this work, we employ a machine translation mechanism and propose a novel sequence-to-sequence (seq2seq) architecture termed TPGen to generate prescriptions. TPGen consisting of an encoder and a decoder is a well-known framework for resolving the machine translation problem in the natural language processing (NLP) domain. We use the lite transformer and Bi-directional Gate Recurrent Units(Bi-GRUS) as a fundamental model in TPGen, and integrate TCM clinical knowledge to guide the model improvement termed TPGen+. Results: We conduct extensive experiments on a public TCM dataset and clinical data. The experimental results demonstrate that our proposed model is effective and outperforms other state-of-the-art methods in TCM expert evaluation. The approach will be beneficial for clinical prescription discovery and diagnosis

show abstract

In Neural Machine Translation, What Does Transfer Learning Transfer?

Cited by 38 publications

References 23 publications

The Dawn of the Human-Machine Era: A forecast of new and emerging language technologies

The Dawn of the Human-Machine Era: A forecast of new and emerging language technologies

On Romanization for Model Transfer Between Scripts in Neural Machine Translation

TPGen: Prescription Generation Using Knowledge-guided Translator

Contact Info

Product

Resources

About