Generalising Recursive Neural Models by Tensor Decomposition

Abstract

Most machine learning models for structured data encode the structural knowledge of a node by leveraging simple aggregation functions (in neural models, typically a weighted sum) of the information in the node's neighbourhood. Nevertheless, the choice of simple context aggregation functions, such as the sum, can be widely sub-optimal. In this work we introduce a general approach to model aggregation of structural context leveraging a tensor-based formulation. We show how the exponential growth in the size of the parameter space can be controlled through an approximation based on the Tucker tensor decomposition. This approximation allows limiting the parameters space size, decoupling it from its strict relation with the size of the hidden encoding space. By this means, we can effectively regulate the trade-off between expressivity of the encoding, controlled by the hidden size, computational complexity and model generalisation, influenced by parameterisation. Finally, we introduce a new Tensorial Tree-LSTM derived as an instance of our framework and we use it to experimentally assess our working hypotheses on tree classification scenarios.

Tensor Decompositions in Deep Learning

“…Two input accumulation functions are discussed in the paper: one leveraging CP decomposition and the other relying on the Tensor-Train factorization. Similarly, [30] shows how a tensor recursive neuron can exploit the Tucker decomposition to control the trade-off between the size of the neural representation and the size of the neural parameter space.…”

Learning from Non-Binary Constituency Trees via Tensor Decomposition

“…This section aims to introduce new Tree-LSTM models that extend the recurrent approach to tensor-based processing tailored for constituency trees. These two models rely on Tensor Tree-LSTM (Castellana and Bacciu, 2020a) and canonical tensor decomposition. We focus on such decomposition since it can be combined with a weight sharing constraint, developing a composition function which can exploit nonbinary constituency trees without increasing the model parameters number.…”

“…The Binary Tensor Tree-LSTM (Castellana and Bacciu, 2020a) computes the hidden state of an intern node v by:…”

“…the maximum number of children for each node in the tree) required by the full-tensorial approach can be controlled by applying tensor decomposition. The tensorial models outperform sum-based models, especially when the tree out-degree increases (Castellana and Bacciu, 2020a;Castellana and Bacciu, 2020b).…”

“…Standard RecNNs define learnable composition functions which are based on the summation of the contribution of each constituent. Castellana and Bacciu (2020a) proposed a generalisation of such sum-based composition functions leveraging more expressive multi-affine maps represented as tensors. The exponential number of parameters with respect to the tree out-degree (i.e.…”