Scheduled Multi-Task Learning: From Syntax to Translation

Kiperwasser, Eliyahu; Ballesteros, Miguel

doi:10.1162/tacl_a_00017

Cited by 72 publications

(79 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The papers mentioned so far targeted primarily the quality of MT (as measured by BLEU), not the secondary tasks. [13] note that their system performs reasonably well in both tagging and parsing. [31] present an in-depth analysis of the syntactic knowledge learned by the recurrent sequence-to-sequence NMT.…”

Section: Related Workmentioning

confidence: 92%

“…In parallel to our work, [13] examined various scheduling strategies for a very simple approach to multi-tasking: representing all the tasks converted to a common format of source and target sequences of symbols from a joint vocabulary and training one sequence-to-sequence system on the mix of training examples from the different tasks.…”

Section: Related Workmentioning

confidence: 99%

“…The scheduling strategy specified the proportion of the tasks in training batches in time. [13] report improvements in BLEU score [26] in small-data setting for German-to-English translation for all multi-task setups (translation combined with POS tagging and/or source-side 1 parsing).…”

Section: Related Workmentioning

confidence: 99%

“…5 pLabels" task is tagging with dependency labels, and "DepHeads+DepLabels" is an interleaved combination of the two. The training data in multitasking are selected by constant scheduler as in [13], with parameter 0.5, which means the trainer alternates between the tasks, in average, after every training step. As [13] reminds, this is different from [20] and [43] where the mixing happens at the level of batches and not individual examples.…”

Section: Approachmentioning

confidence: 99%

“…A possible reason may be the vector concatenation and layer normalization after each multi-head attention layer in the Transformer. [13] suggest another representation of "Dep-Heads", which doesn't suffer on unknown words and repeated words. They indicate the governing node as an offset from the node's position represented as decimal number, positive to the right, negative to the left.…”

Section: Self-attention Patterns In the Encodermentioning

confidence: 99%

See 4 more Smart Citations

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed

Pham¹,

Macháček²,

Bojar³

2019

CyS

View full text Add to dashboard Cite

The utility of linguistic annotation in neural machine translation seemed to had been established in past papers. The experiments were however limited to recurrent sequence-to-sequence architectures and relatively small data settings.We focus on the state-of-the-art Transformer model and use comparably larger corpora. Specifically, we try to promote the knowledge of source-side syntax using multi-task learning either through simple data manipulation techniques or through a dedicated model component. In particular, we train one of Transformer attention heads to produce source-side dependency tree.Overall, our results cast some doubt on the utility of multi-task setups with linguistic information. The data manipulation techniques, recommended in previous works, prove ineffective in large data settings.The treatment of self-attention as dependencies seems much more promising: it helps in translation and reveals that Transformer model can very easily grasp the syntactic structure. An important but curious result is, however, that identical gains are obtained by using trivial "linear trees" instead of true dependencies. The reason for the gain thus may not be coming from the added linguistic knowledge but from some simpler regularizing effect we induced on self-attention matrices.

show abstract

Section: Related Workmentioning

confidence: 92%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Approachmentioning

confidence: 99%

Section: Self-attention Patterns In the Encodermentioning

confidence: 99%

See 3 more Smart Citations

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed

Pham¹,

Macháček²,

Bojar³

2019

CyS

View full text Add to dashboard Cite

show abstract

Quality of Syntactic Implication of RL-Based Sentence Summarization

Cerisara

Gardent

2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

Work on summarization has explored both reinforcement learning (RL) optimization using ROUGE as a reward and syntax-aware models, such as models whose input is enriched with part-of-speech (POS)-tags and dependency information. However, it is not clear what is the respective impact of these approaches beyond the standard ROUGE evaluation metric. Especially, RL-based for summarization is becoming more and more popular. In this paper, we provide a detailed comparison of these two approaches and of their combination along several dimensions that relate to the perceived quality of the generated summaries: number of repeated words, distribution of part-of-speech tags, impact of sentence length, relevance and grammaticality. Using the standard Gigaword sentence summarization task, we compare an RL self-critical sequence training (SCST) method with syntax-aware models that leverage POS tags and Dependency information. We show that on all qualitative evaluations, the combined model gives the best results, but also that only training with RL and without any syntactic information already gives nearly as good results as syntax-aware models with less parameters and faster training convergence.

show abstract

Deep Sarcasm Detection with Sememe and Syntax Knowledge

Zhang,

Shan,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Scheduled Multi-Task Learning: From Syntax to Translation

Cited by 72 publications

References 31 publications

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed

Quality of Syntactic Implication of RL-Based Sentence Summarization

Deep Sarcasm Detection with Sememe and Syntax Knowledge

Contact Info

Product

Resources

About