Continuous Self-Attention Models with Neural ODE Networks

Abstract

Stacked self-attention models receive widespread attention, due to its ability of capturing global dependency among words. However, the stacking of many layers and components generates huge parameters, leading to low parameter efficiency. In response to this issue, we propose a lightweight architecture named Continuous Self-Attention models with neural ODE networks (CSAODE). In CSAODE, continuous dynamical models (i.e., neural ODEs) are coupled with our proposed self-attention block to form a self-attention ODE solver. This solver continuously calculates and optimizes the hidden states via only one layer of parameters to improve the parameter efficiency. In addition, we design a novel accelerated continuous dynamical model to reduce computing costs, and integrate it in CSAODE. Moreover, since the original self-attention ignores local information, CSAODE makes use of N-gram convolution to encode local representations, and a fusion layer with only two trainable scalars are designed for generating sentence vectors. We perform a series of experiments on text classification, neural language inference (NLI) and text matching tasks. With fewer parameters, CSAODE outperforms state-of-the-art models on text classification tasks (e.g., 1.3% accuracy improved on SUBJ task), and has competitive performances for NLI and text matching tasks as well.

Social ODE: Multi-agent Trajectory Forecasting with Neural Ordinary Differential Equations

“…For example, due to the nature of adaptive step size ODE solvers, the situation that many consecutive layers are dynamically equivalent is very common, in [41], the problem is solved by applying optimal transportation theory to encourage simpler trajectory dynamics. Recent developments extend these ideas to continuous-time video forecasting [13] and continuous attention architectures [42], [43]. However, the application to anytime human 3D pose forecasting has not been explored.…”

Continuous-Time Attention for Sequential Learning

“…Neural differential equation (Chen et al 2018;Zhang et al 2021) was proposed to build a continuous-time state machine for learning representation of sequence data {x tn } where the time points {t n } N n=1 are irregularly sampled. NDE was implemented to learn the dynamics of transformation so as to characterize the state transition z(t) at continuous-time t between input samples and output targets based on an ordinary differential equation (ODE).…”

Harnessing Machine Learning for Unmasking Deception: An In-Depth Analysis using ML Approaches for Fake News Identification in News Media

“…An attention-based neural network solution to fake news detection is put forth by Zhang et al [16]. When producing predictions, their method focuses on the most crucial information in news items using an attention-based neural network.…”