Tensorial Recurrent Neural Networks for Longitudinal Data Analysis

Bai, Mingyuan; Zhang, Boyan; Gao, Junbin

doi:10.48550/arxiv.1708.00185

Cited by 5 publications

(6 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, we can also use other constraints instead of (9) and solve the optimization problem in (7), ( 8) and (9) in the same manner. As an example, a common choice of constraint for neural networks is the Frobenius norm [24], i.e., defined as…”

Section: A Anomaly Detection With the Oc-svm Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Anomaly Detection With LSTM Neural Networks

Ergen

Kozat

2020

IEEE Trans. Neural Netw. Learning Syst.

253

View full text Add to dashboard Cite

We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms. In particular, given variable length data sequences, we first pass these sequences through our LSTM based structure and obtain fixed length sequences. We then find a decision function for our anomaly detectors based on the One Class Support Vector Machines (OC-SVM) and Support Vector Data Description (SVDD) algorithms. As the first time in the literature, we jointly train and optimize the parameters of the LSTM architecture and the OC-SVM (or SVDD) algorithm using highly effective gradient and quadratic programming based training methods. To apply the gradient based training method, we modify the original objective criteria of the OC-SVM and SVDD algorithms, where we prove the convergence of the modified objective criteria to the original criteria. We also provide extensions of our unsupervised formulation to the semisupervised and fully supervised frameworks. Thus, we obtain anomaly detection algorithms that can process variable length data sequences while providing high performance, especially for time series data. Our approach is generic so that we also apply this approach to the Gated Recurrent Unit (GRU) architecture by directly replacing our LSTM based structure with the GRU based structure. In our experiments, we illustrate significant performance gains achieved by our algorithms with respect to the conventional methods.

show abstract

Section: A Anomaly Detection With the Oc-svm Algorithmmentioning

confidence: 99%

“…Remark 7. For the SVDD case, we update W (•) at the k th iteration as in (24). However, instead of (25), we have the following definition for G…”

Section: B Anomaly Detection With the Svdd Algorithmmentioning

confidence: 99%

Unsupervised Anomaly Detection With LSTM Neural Networks

Ergen

Kozat

2020

IEEE Trans. Neural Netw. Learning Syst.

253

View full text Add to dashboard Cite

show abstract

“…Compared with aforementioned explicit structure changes, the low-rank method is one orthogonal approach to implicitly prune the dense connections. Low-rank tensor methods have been successfully applied to address the redundant dense connection problem in CNNs [28,47,1,38,18]. Since the key operation in one perception is W • x, Sainath et al [31] decompose W with Singular Value Decomposition (SVD), reducing up to 30% parameters in W, but also demonstrates up to 10% accuracy loss [46].…”

Section: Related Workmentioning

confidence: 99%

“…In this work, we propose to design a sparsely connected tensor representation, i.e., the Block-Term decomposition (BTD) [7], to replace the redundant and densely connected operation in LSTM 1 . The Block-Term decomposition is a low-rank approximation method that decomposes a highorder tensor into a sum of multiple Tucker decomposition models [39,44,45,21].…”

Section: Introductionmentioning

confidence: 99%

“…• Each dimension in the input data can share weights with all the other dimensions as the existence of core tensors, thus BT representation has the strong connection between different dimensions, enhancing the 1 we focus on LSTM in this paper, but the proposed approach also applies for other variants such as GRU. ability to capture sufficient local correlations.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

Wang

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

119

View full text Add to dashboard Cite

Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in Videos and Image Captioning. To overcome this problem, we propose a compact and flexible structure, namely Block-Term tensor decomposition, which greatly reduces the parameters of RNNs and improves their training efficiency. Compared with alternative low-rank approximations, such as tensortrain RNN (TT-RNN), our method, Block-Term RNN (BT-RNN), is not only more concise (when using the same rank), but also able to attain a better approximation to the original RNNs with much fewer parameters. On three challenging tasks, including Action Recognition in Videos, Image Captioning and Image Generation, BT-RNN outperforms TT-RNN and the standard RNN in terms of both prediction accuracy and convergence rate. Specifically, BT-LSTM utilizes 17,388 times fewer parameters than the standard LSTM to achieve an accuracy improvement over 15.6% in the Action Recognition task on the UCF11 dataset.

show abstract

Block-term tensor neural networks

Chen

et al. 2020

Neural Networks

View full text Add to dashboard Cite

Deep neural networks (DNNs) have achieved outstanding performance in a wide range of applications, e.g., image classification, natural language processing, etc. Despite the good performance, the huge number of parameters in DNNs brings challenges to efficient training of DNNs and also their deployment in low-end devices with limited computing resources. In this paper, we explore the correlations in the weight matrices, and approximate the weight matrices with the low-rank block-term tensors. We name the new corresponding structure as block-term tensor layers (BT-layers), which can be easily adapted to neural network models, such as CNNs and RNNs. In particular, the inputs and the outputs in BT-layers are reshaped into low-dimensional high-order tensors with a similar or improved representation power. Sufficient experiments have demonstrated that BT-layers in CNNs and RNNs can achieve a very large compression ratio on the number of parameters while preserving or improving the representation power of the original DNNs.

show abstract

Tensorial Recurrent Neural Networks for Longitudinal Data Analysis

Cited by 5 publications

References 4 publications

Unsupervised Anomaly Detection With LSTM Neural Networks

Unsupervised Anomaly Detection With LSTM Neural Networks

Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

Block-term tensor neural networks

Contact Info

Product

Resources

About