Deep Copycat Networks for Text-to-Text Generation

Ive, Julia; Madhyastha, Pranava; Specia, Lucia

doi:10.18653/v1/d19-1318

Cited by 4 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-like our Graformer -instead average attention weights over all heads. Other approaches compute their copy scores independently from encoder-decoder attention weights -either based on the output (Cai and Lam, 2020) or an intermediate representation of the last decoder layer (Ive et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs

Schmitt¹,

Ribeiro²,

Dufter³

et al. 2021

Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)

View full text Add to dashboard Cite

We present a novel encoder-decoder architecture for graph-to-text generation based on Transformer, called the Graformer. With our novel graph self-attention, every node in the input graph is taken into account for the encoding of every other node -not only direct neighbors, facilitating the detection of global patterns. For this, the relation between any two nodes is characterized by the length of the shortest path between them, including the special case when there is no such path. The Graformer learns to weigh these node-node relations differently for different attention heads, thus virtually learning differently connected views of the input graph. We evaluate the Graformer on two graph-to-text generation benchmarks, the AGENDA dataset and the WebNLG challenge dataset, where it achieves strong performance while using significantly less parameters than other approaches. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs

Schmitt¹,

Ribeiro²,

Dufter³

et al. 2021

Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)

View full text Add to dashboard Cite

show abstract

“…In Table 5.2, the results of our systems and other state-of-the-art systems for the CNN/DailyMail corpus are shown 13 . It can be seen how our systems obtain better results than other widely used systems PGen+Cov [291], CopyCat [317] and SummaRunner [290]. The obtained results are worse in comparison to other extractive systems that use oracles, especially in the case of BertSumEXT [294], in spite of our systems share the same backbone architecture (transformer encoders).…”

Section: Discussionmentioning

confidence: 76%

“…Regarding the abstractive systems, all of them are based on encoder-decoder models with attention mechanisms [95,96,269,289,291,296,297,312,317]. First approaches suffered from known problems of the traditional sequence-to-sequence approaches: repetitions, grammatically incorrect generations, lack of coherence, coverage, hallucination (especially factual inconsistency), and the inability of producing words out of the training vocabulary [269,289,312].…”

Section: State Of the Artmentioning

confidence: 99%

“…First approaches suffered from known problems of the traditional sequence-to-sequence approaches: repetitions, grammatically incorrect generations, lack of coherence, coverage, hallucination (especially factual inconsistency), and the inability of producing words out of the training vocabulary [269,289,312]. To address these issues, some systems, capable of selecting or generating a new word at each timestep, were proposed [291,317] (they can also be seen as a kind of hybrid end-to-end systems). The most relevant example is [291], where an approach based on Pointer Networks and encoder-decoder models with attention mechanisms is proposed.…”

Section: State Of the Artmentioning

confidence: 99%

“…In order to address the repetition problem, the authors enrich their system by using a coverage mechanism based on the attentions of previous timesteps, for each decoder timestep (PointerGen+Cov). This system has been modified by the authors of [317], replacing the backbone architecture (Long Short-Term Memories [10]) by transformers. Most of these issues (grammatically incorrect generations, repetitions, and the inability of producing out of vocabulary words) have been recently overcome by transfer learning, being now more competitive against the extractive approaches.…”

Section: State Of the Artmentioning

confidence: 99%

See 2 more Smart Citations

Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization

Barba¹

View full text Add to dashboard Cite

Nowadays, society has access, and the possibility to contribute, to large amounts of the content present on the internet, such as social networks, online newspapers, forums, blogs, or multimedia content platforms. These platforms have had, during the last years, an overwhelming impact on the daily life of individuals and organizations, becoming the predominant ways for sharing, discussing, and analyzing online content. Therefore, it is very interesting to work with these platforms, from different points of view, under the umbrella of Natural Language Processing. In this thesis, we focus on two broad areas inside this field, applied to analyze online content: text analytics in social media and automatic summarization. Neural networks are also a central topic in this thesis, where all the experimentation has been performed by using deep learning approaches, mainly based on attention mechanisms. Besides, we mostly work with the Spanish language, due to it is an interesting and underexplored language with a great interest in the research projects we participated in.On the one hand, for text analytics in social media, we focused on affective analysis tasks, including sentiment analysis and emotion detection, along with the analysis of the irony. In this regard, an approach based on Transformer Encoders, based on contextualizing pretrained Spanish word embeddings from Twitter, to address sentiment analysis and irony detection tasks, is presented. We also propose the use of evaluation metrics as loss functions, in order to train neural networks for reducing the impact of the class imbalance in multi-class and multi-label emotion detection tasks. Additionally, a specialization of BERT both for the Spanish language and the Twitter domain, that takes into account inter-sentence coherence in Twitter conversation flows, is presented. The performance of all these approaches has been tested with different corpora, from several reference evaluation benchmarks, showing very competitive results in all the tasks addressed.On the other hand, we focused on extractive summarization of news articles and TV talk shows. Regarding the summarization of news articles, a theoretical framework for extractive summarization, based on siamese hierarchical networks with attention mechanisms, is presented. Also, we present two instantiations of this framework: Siamese Hierarchical Attention Networks and Siamese Hierarchical Transformer Encoders. These systems were ResumenHoy en día, la sociedad tiene acceso y posibilidad de contribuir a grandes cantidades de contenidos presentes en Internet, como redes sociales, periódicos online, foros, blogs o plataformas de contenido multimedia. Todo este tipo de medios han tenido, durante los últimos años, un impacto abrumador en el día a día de individuos y organizaciones, siendo actualmente medios predominantes para compartir, debatir y analizar contenidos online. Por este motivo, resulta de interés trabajar sobre este tipo de plataformas, desde diferentes puntos de vista, bajo el paraguas del Procesamiento...

show abstract

DACSA: A large-scale Dataset for Automatic summarization of Catalan and Spanish newspaper Articles

Soriano¹,

Ahuir²,

Hurtado³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

The application of supervised methods to automatic summarization requires the availability of adequate corpora consisting of a set of document-summary pairs. As in most Natural Language Processing tasks, the great majority of available datasets for summarization are in English, making it difficult to develop automatic summarization models for other languages. Although Spanish is gradually forming part of some recent summarization corpora, it is not the same for minority languages such as Catalan. In this work, we describe the construction of a corpus of Catalan and Spanish newspapers, the Dataset for Automatic summarization of Catalan and Spanish newspaper Articles (DACSA) corpus. It is a high-quality large-scale corpus that can be used to train summarization models for Catalan and Spanish. We have carried out an analysis of the corpus, both in terms of the style of the summaries and the difficulty of the summarization task. In particular, we have used a set of well-known metrics in the summarization field in order to characterize the corpus. Additionally, we have evaluated the performance of some extractive and abstractive summarization systems on the DACSA corpus for benchmarking purposes.

show abstract

Deep Copycat Networks for Text-to-Text Generation

Cited by 4 publications

References 19 publications

Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs

Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs

Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization

DACSA: A large-scale Dataset for Automatic summarization of Catalan and Spanish newspaper Articles

Contact Info

Product

Resources

About