Effective Use of Transformer Networks for Entity Tracking

Gupta, Aditya; Durrett, Greg

doi:10.18653/v1/d19-1070

Cited by 17 publications

(22 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is "pretrained dynamics," we also consider a version without a randomly initialized dynamics model. e. (Gupta and Durrett, 2019)-style. Thiso paper proposes using Transformers to model physical state, for tasks like entity tracking in recipes.…”

Section: Pigpen-nlu Resultsmentioning

confidence: 99%

“…PIGLeT also outperforms 'BERT style' approaches that control for the same language model architecture, but perform the physical reasoning inside the language transformer rather than as a separate model. Performance drops when the physical decoder must be learned from few paired examples (as in Gupta and Durrett (2019)); it drops even further when neither model is given access to our pretrained dynamics model, with both baselines then underperforming 'No Change.' This suggests that our approach of having a physical reasoning model outside of an LM is a good inductive bias.…”

Section: Pigpen-nlu Resultsmentioning

confidence: 99%

See 1 more Smart Citation

PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World

Zellers¹,

Holtzman²,

Peters³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do: glass cups break when thrown, plastic ones don't. We then use it as the interface to our language model, giving us a unified model of linguistic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic representation, or natural language.Experimental results show that our model effectively learns world dynamics, along with how to communicate them. It is able to correctly forecast "what happens next" given an English sentence over 80% of the time, outperforming a 100x larger, text-to-text approach by over 10%. Likewise, its natural language summaries of physical interactions are also judged by humans as more accurate than LM alternatives. We present comprehensive analysis showing room for future work. 2041The robot throws the vase onto the coffee table.The robot is holding a vase, and there is a laptop on the coffee table that is on.The laptop and the vase both break, with the vase shattering into smaller pieces, and the laptop powers off.

show abstract

Section: Pigpen-nlu Resultsmentioning

confidence: 99%

Section: Pigpen-nlu Resultsmentioning

confidence: 99%

PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World

Zellers¹,

Holtzman²,

Peters³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Transformer architectures trained on language modeling have been recently adapted to downstream tasks demonstrating state-of-the-art performance (Weller and Seppi, 2019;Gupta and Durrett, 2019;Maronikolakis et al, 2020). In this paper, we adapt and subsequently combine transformers with external linguistic information for complaint prediction.…”

Section: Transformer-based Modelsmentioning

confidence: 99%

Complaint Identification in Social Media with Transformer Networks

Jin¹,

Αλέτρας²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Complaining is a speech act extensively used by humans to communicate a negative inconsistency between reality and expectations. Previous work on automatically identifying complaints in social media has focused on using feature-based and task-specific neural network models. Adapting state-of-the-art pre-trained neural language models and their combinations with other linguistic information from topics or sentiment for complaint prediction has yet to be explored. In this paper, we evaluate a battery of neural models underpinned by transformer networks which we subsequently combine with linguistic information. Experiments on a publicly available data set of complaints demonstrate that our models outperform previous state-of-the-art methods by a large margin achieving a macro F1 up to 87.

show abstract

“…Thus, newer models are not particularly different from the perspective of the NLI task itself. Following existing work [19,49], we include a comparison of models trained from the same BERT BAS E checkpoint. Table 4 shows the accuracy of the classification-only model and our multi-task trained models on the MNLI dataset, all having the same BERT BAS E as a starting point.…”

Section: Performance On Original Nli Taskmentioning

confidence: 99%

Explaining Text Matching on Neural Natural Language Inference

Kim

Jang

Allan

2020

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Natural language inference (NLI) is the task of detecting the existence of entailment or contradiction in a given sentence pair. Although NLI techniques could help numerous information retrieval tasks, most solutions for NLI are neural approaches whose lack of interpretability prohibits both straightforward integration and diagnosis for further improvement. We target the task of generating token-level explanations for NLI from a neural model. Many existing approaches for token-level explanation are either computationally costly or require additional annotations for training. In this article, we first introduce a novel method for training an explanation generator that does not require additional human labels. Instead, the explanation generator is trained with the objective of predicting how the model's classification output will change when parts of the inputs are modified. Second, we propose to build an explanation generator in a multi-task learning setting along with the original NLI task so the explanation generator can utilize the model's internal behavior. The experiment results suggest that the proposed explanation generator outperforms numerous strong baselines. In addition, our method does not require excessive additional computation at prediction time, which renders it an order of magnitude faster than the best-performing baseline.

show abstract

Effective Use of Transformer Networks for Entity Tracking

Cited by 17 publications

References 16 publications

PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World

PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World

Complaint Identification in Social Media with Transformer Networks

Explaining Text Matching on Neural Natural Language Inference

Contact Info

Product

Resources

About