Compositional Generalization by Learning Analytical Expressions

Liu, Qian; An, Shengnan; Lou, Jian–Guang; Chen, Bei; Lin, Zeqi; Gao, Yan; Zhou, Bin; Zheng, Nanning; Zhang, Dongmei

doi:10.48550/arxiv.2006.10627

Cited by 4 publications

(1 citation statement)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different compositional generalization approaches have been investigated, such as architecture design [1,2], independence assumption [3,4], data augmentation [5,6], causality [7,8], reinforcement learning [9], group theory [10] and meta-learning [11]. There are also general discussions [12,13].…”

Section: Introductionmentioning

confidence: 99%

Concepts, Properties and an Approach for Compositional Generalization

Li¹

2021

Preprint

View full text Add to dashboard Cite

Compositional generalization is the capacity to recognize and imagine a large amount of novel combinations from known components. It is a key in human intelligence, but current neural networks generally lack such ability. This report connects a series of our work for compositional generalization, and summarizes an approach. The first part contains concepts and properties. The second part looks into a machine learning approach. The approach uses architecture design and regularization to regulate information of representations. This report focuses on basic ideas with intuitive and illustrative explanations. We hope this work would be helpful to clarify fundamentals of compositional generalization and lead to advance artificial intelligence.

show abstract

Section: Introductionmentioning

confidence: 99%

Concepts, Properties and an Approach for Compositional Generalization

Li¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

Learning Algebraic Recombination for Compositional Generalization

Liu¹,

An²,

Lin³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Neural sequence models exhibit limited compositional generalization ability in semantic parsing tasks. Compositional generalization requires algebraic recombination, i.e., dynamically recombining structured expressions in a recursive manner. However, most previous studies mainly concentrate on recombining lexical units, which is an important but not sufficient part of algebraic recombination. In this paper, we propose LEAR, an end-toend neural model to learn algebraic recombination for compositional generalization. The key insight is to model the semantic parsing task as a homomorphism between a latent syntactic algebra and a semantic algebra, thus encouraging algebraic recombination. Specifically, we learn two modules jointly: a Composer for producing latent syntax, and an Interpreter for assigning semantic operations. Experiments on two realistic and comprehensive compositional generalization benchmarks demonstrate the effectiveness of our model. The source code is publicly available at https://github.com/microsoft/ContextualSP.

show abstract

Distributionally Robust Recurrent Decoders with Random Network Distillation

Miceli-Barone¹,

Birch²,

Sennrich³

2022

Proceedings of the 7th Workshop on Representation Learning for NLP

View full text Add to dashboard Cite

Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to "shortcut learning": relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD, while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.

show abstract

Compositional Generalization by Learning Analytical Expressions

Cited by 4 publications

References 18 publications

Concepts, Properties and an Approach for Compositional Generalization

Concepts, Properties and an Approach for Compositional Generalization

Learning Algebraic Recombination for Compositional Generalization

Distributionally Robust Recurrent Decoders with Random Network Distillation

Contact Info

Product

Resources

About