What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment

Mei, Hongyuan; Bansal, Mohit; Walter, Matthew R.

doi:10.48550/arxiv.1509.00838

Cited by 23 publications

(31 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It takes into account the semantic relationship between records and reference words and experiment results verify the effectiveness of this improvement on both document-and sentence-level datasets. Furthermore, our work is similar but different from datato-text generation studies (Mei, Bansal, and Walter 2015;Nie et al 2018;Puduppully, Dong, and Lapata 2018;Bao et al 2018;Sun et al 2018;Gong et al 2019a;Chen et al 2019;Gong et al 2019b). This series of work focuses on generating more accurate descriptions of given data, rather than studying the writing content of control output.…”

Section: Related Workcontrasting

confidence: 72%

“…Similarly, the model also need to select the suitable style words according to the input records. One straightforward way to address this problem is to use the relevant algorithms in datato-text generation, such as pre-selector (Mei, Bansal, and Walter 2015) and content selector (Puduppully, Dong, and Lapata 2018). However, these supervised methods cannot be directly transferred considering that we impose an additional goal of preserving the style words, which lacks of parallel data and explicit training objective.…”

Section: Desired Outputmentioning

confidence: 99%

“…As shown in the empirical study (see Table 2), the hierarchical encoder can gain significant improvements compared with the standard MLP-based datato-text model (Wiseman, Shieber, and Rush 2017). Each word and figure are represented as a low dimensional, continuous and real-valued vector, also known as word embedding (Mikolov et al 2013;Pennington, Socher, and Manning 2014). All vectors are stacked in a word embedding matrix L w ∈ R d×|V | , where d is the dimension of the word vector and |V | is the vocabulary size.…”

Section: Hierarchical Record Encodermentioning

confidence: 99%

See 2 more Smart Citations

Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation

Feng

Sun

Qin

et al. 2020

AAAI

View full text Add to dashboard Cite

In this paper, we focus on a new practical task, document-scale text content manipulation, which is the opposite of text style transfer and aims to preserve text styles while altering the content. In detail, the input is a set of structured records and a reference text for describing another recordset. The output is a summary that accurately describes the partial content in the source recordset with the same writing style of the reference. The task is unsupervised due to lack of parallel data, and is challenging to select suitable records and style words from bi-aspect inputs respectively and generate a high-fidelity long document. To tackle those problems, we first build a dataset based on a basketball game report corpus as our testbed, and present an unsupervised neural model with interactive attention mechanism, which is used for learning the semantic relationship between records and reference texts to achieve better content transfer and better style preservation. In addition, we also explore the effectiveness of the back-translation in our task for constructing some pseudo-training pairs. Empirical results show superiority of our approaches over competitive methods, and the models also yield a new state-of-the-art result on a sentence-level dataset. 1

show abstract

Section: Related Workcontrasting

confidence: 72%

Section: Desired Outputmentioning

confidence: 99%

Section: Hierarchical Record Encodermentioning

confidence: 99%

See 1 more Smart Citation

Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation

Feng

Sun

Qin

et al. 2020

AAAI

View full text Add to dashboard Cite

show abstract

“…End to end learning approaches Several recent deep learning approaches propose to learn a mapping directly from inputs to actions, whether structured observations are provided [22,33] or the agent deals with raw visual observations [25,43]. Cross modal grounding of language instructions to visual observations is often used in several works, via e.g., reinforcement learning [38,37], autoencoder architectures that impose a language instructions-based heat map on the visual observations (using U-net architectures [24], attention mechanisms [49], or implementation of non linear differentiable filters [2]).…”

Section: Related Workmentioning

confidence: 99%

“…Several recent deep learning approaches propose to learn a mapping directly from inputs to actions, whether structured observations are provided [22,33] or the agent deals with raw visual observations [25,43]. Proponents of this approach argue about the simplicity of these models in outperforming a combination of models that require engineering hand crafted representations.…”

Section: End To End Learning Approachesmentioning

confidence: 99%

A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment

Saha¹,

Fotouhif²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper we propose a new framework -MoVi-Lan (Modular Vision and Language) for execution of visually grounded natural language instructions for day to day indoor household tasks. While several data-driven, endto-end learning frameworks have been proposed for targeted navigation tasks based on the vision and language modalities, performance on recent benchmark data sets revealed the gap in developing comprehensive techniques for long horizon, compositional tasks (involving manipulation and navigation) with diverse object categories, realistic instructions and visual scenarios with non-reversible state changes. We propose a modular approach to deal with the combined navigation and object interaction problem without the need for strictly aligned vision and language training data (e.g., in the form of expert demonstrated trajectories). Such an approach is a significant departure from the traditional end-to-end techniques in this space and allows for a more tractable training process with separate vision and language data sets. Specifically, we propose a novel geometry-aware mapping technique for cluttered indoor environments, and a language understanding model generalized for household instruction following. We demonstrate a significant increase in success rates for long-horizon, compositional tasks over the baseline on the recently released benchmark data set-ALFRED.

show abstract

TrICy: Trigger-Guided Data-to-Text Generation With Intent Aware Attention-Copy

Agarwal,

Ghosh,

BSS

et al. 2024

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

generation is a crucial task in many natural language understanding (NLU) applications and forms the foundation of task-oriented dialog systems. In the context of conversational AI solutions that can work directly with local data on the user's device, architectures utilizing large pre-trained language models (PLMs) are impractical for on-device deployment due to a high memory footprint. To this end, we propose TrICy, a novel lightweight framework for an enhanced D2T task that generates text sequences based on the intent in context and may further be guided by userprovided triggers. We leverage an attention-copy mechanism to predict out-of-vocabulary (OOV) words accurately. Performance analyses on E2E NLG dataset [1] (BLEU: 66.43%, ROUGE-L: 70.14%), WebNLG dataset [2] (BLEU: Seen 64.08%, Unseen 52.35%), and our Custom dataset related to text messaging applications, showcase our architecture's effectiveness. Moreover, we show that by leveraging an optional trigger input, data-to-text generation quality increases significantly and achieves the new SOTA score of 69.29% BLEU for E2E NLG. Furthermore, our analyses show that TrICy achieves at least 24% and 3% improvement in BLEU and METEOR respectively over LLMs like GPT-3, ChatGPT, and Llama 2. We also demonstrate that in some scenarios, performance improvement due to triggers is observed even when they are absent in training.

show abstract

What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment

Cited by 23 publications

References 13 publications

Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation

Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation

A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment

TrICy: Trigger-Guided Data-to-Text Generation With Intent Aware Attention-Copy

Contact Info

Product

Resources

About