Cutting-off Redundant Repeating Generations for Neural Abstractive
            Summarization

Suzuki, Jun; Nagata, Michio

doi:10.18653/v1/e17-2047

Cited by 47 publications

(36 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[T] indicates a token-level objective and [S] indicates a sequence level objectives. ABS+ refers to Rush et al (2015), RNN MLE/MRT (Ayana et al, 2016), WFE (Suzuki and Nagata, 2017), SEASS , DRGD . ong et al, 2015).…”

Section: Comparison To Beam-search Optimizationmentioning

confidence: 99%

Classical Structured Prediction Losses for Sequence to Sequence Learning

Edunov¹,

Ott²,

Auli³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

134

164

View full text Add to dashboard Cite

There has been much recent work on training neural attention models at the sequencelevel using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT'14 German-English translation as well as Gigaword abstractive summarization. On the large WMT'14 English-French task, sequence-level training achieves 41.5 BLEU which is on par with the state of the art.

show abstract

Section: Comparison To Beam-search Optimizationmentioning

confidence: 99%

Classical Structured Prediction Losses for Sequence to Sequence Learning

Edunov¹,

Ott²,

Auli³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

134

164

View full text Add to dashboard Cite

show abstract

“…Furthermore, we compare the performance of MC-RNN with other recent methods on this task. We find that both ROUGE-1 and ROUGE-2 of our model outperform all current top systems, including (Shen et al 2016;Gehring et al 2017;Suzuki and Nagata 2017).…”

Section: Experimental Results Frommentioning

confidence: 51%

“…Recurrent neural networks (RNNs), designed with recurrent units and parameter sharing, have demonstrated outstanding ability in modeling sequential data and achieved success in various Nature Language Processing (NLP) tasks, such as language modeling (Merity, Keskar, and Socher 2018), machine translation (Bahdanau et al 2017;Huang et al 2018), abstractive summarization (Suzuki and Nagata 2017), and dialog systems (Asri, He, and Suleman 2016). Traditional RNNs produce hidden state vectors one by one through recurrent computations, treating all tokens in the sequence uniformly and equally.…”

Section: Introductionmentioning

confidence: 99%

Modeling Local Dependence in Natural Language with Multi-Channel Recurrent Neural Networks

Huang

Wang

et al. 2019

AAAI

View full text Add to dashboard Cite

Recurrent Neural Networks (RNNs) have been widely used in processing natural language tasks and achieve huge success. Traditional RNNs usually treat each token in a sentence uniformly and equally. However, this may miss the rich semantic structure information of a sentence, which is useful for understanding natural languages. Since semantic structures such as word dependence patterns are not parameterized, it is a challenge to capture and leverage structure information. In this paper, we propose an improved variant of RNN, Multi-Channel RNN (MC-RNN), to dynamically capture and leverage local semantic structure information. Concretely, MC-RNN contains multiple channels, each of which represents a local dependence pattern at a time. An attention mechanism is introduced to combine these patterns at each step, according to the semantic information. Then we parameterize structure information by adaptively selecting the most appropriate connection structures among channels. In this way, diverse local structures and dependence patterns in sentences can be well captured by MC-RNN. To verify the effectiveness of MC-RNN, we conduct extensive experiments on typical natural language processing tasks, including neural machine translation, abstractive summarization, and language modeling. Experimental results on these tasks all show significant improvements of MC-RNN over current top systems.

show abstract

“…Our re-ranking strategy selects a headline that contains source-side words the most. Table 3 shows that Transformer+LRP E+P E with this reranking (+Re-ranking) achieved better scores than the state-of-the-art (Suzuki and Nagata, 2017).…”

Section: Resultsmentioning

confidence: 99%

Positional Encoding to Control Output Sequence Length

Takase¹,

Okazaki²

2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Neural encoder-decoder models have been successful in natural language generation tasks. However, real applications of abstractive summarization must consider additional constraint that a generated summary should not exceed a desired length. In this paper, we propose a simple but effective extension of a sinusoidal positional encoding (Vaswani et al., 2017) to enable neural encoder-decoder model to preserves the length constraint. Unlike in previous studies where that learn embeddings representing each length, the proposed method can generate a text of any length even if the target length is not present in training data. The experimental results show that the proposed method can not only control the generation length but also improve the ROUGE scores.

show abstract

Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization

Cited by 47 publications

References 17 publications

Classical Structured Prediction Losses for Sequence to Sequence Learning

Classical Structured Prediction Losses for Sequence to Sequence Learning

Modeling Local Dependence in Natural Language with Multi-Channel Recurrent Neural Networks

Positional Encoding to Control Output Sequence Length

Contact Info

Product

Resources

About