A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

Abstract

Inspired by humans' remarkable ability to master arithmetic and generalize to unseen problems, we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three different levels: perception, syntax, and semantics. In particular, concepts in HINT, including both digits and operators, are required to learn in a weakly-supervised fashion: Only the final results of handwriting expressions are provided as supervision. Learning agents need to reckon how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics). With a focus on systematic generalization, we carefully design a fivefold test set to evaluate both the interpolation and the extrapolation of learned concepts. To tackle this challenging problem, we propose a neuralsymbolic system by integrating neural networks with grammar parsing and program synthesis, learned by a novel deduction-abduction strategy. In experiments, the proposed neural-symbolic system demonstrates strong generalization capability and significantly outperforms end-to-end neural methods like RNN and Transformer. The results also indicate the significance of recursive priors for extrapolation on syntax and semantics.

GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning

“…In other words, they can only memorize token patterns from existing corpus rather than understand the rationale behind the language context, thus would fail to perform deductive or abductive reasoning tasks. Similar observation has been investigated by other reasoning tasks, including IQ test (Zhang et al, 2019a(Zhang et al, ,b, 2021b, number sense , causal reasoning (Edmonds et al, 2018(Edmonds et al, , 2019bZhang et al, 2021a), and more generic generalization tasks (Lake et al, 2015;Xie et al, 2021;Li et al, 2021;.…”

Artificial Intelligence techniques based on the integration of symbolic logic and deep neural networks: A systematic review of the literature

“…La solución propuesta por Q. Li et al, (2021), exhibe una fuerte generalización sistemática con una precisión general del 72%, superando a los métodos neuronales de end-to.end en casi un 33%. Alzaeemi et al, (2019) basan su trabajo en la performance de los modelos híbridos, haciendo foco en la introducción de lógica para el cálculo de parámetros y para decrementar la cantidad de neuronas en las capas ocultas.…”

“…En Q. Li et al, (2021), los autores adoptan programas funcionales para entender el significado semántico de los conceptos, por lo que ven a la semántica del aprendizaje como una inducción del programa. La semántica de un concepto es tratada como una función.…”

“…Para realizar predicción de expresiones matemáticas, Q. Li et al, (2021) utilizan programación lógica inductiva, aumentada con redes Bayesianas.…”

Neural-Symbolic Recursive Machine for Systematic Generalization

“…This type of compositionality is central to the human ability to generalize from limited data to novel combinations (Lake et al, 2017). Recently, several datasets have been proposed to test systematic generalization of machine learning models-SCAN (Lake & Baroni, 2018), PCFG (Hupkes et al, 2020), CFQ (Keysers et al, 2020), and HINT (Li et al, 2021), to name a few. While conventional neural networks fail dramatically on these datasets, certain inductive biases have been explored to improve systematic generalization.…”