Sparse Nonnegative Tensor Factorization and Completion With Noisy Observations

Zhang, Xiongjun; Ng, Michael K.

doi:10.1109/tit.2022.3142846

Cited by 14 publications

(3 citation statements)

References 79 publications

(135 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Convolution implementations ( Bagherinezhad, Rastegari & Farhadi, 2017 ; Kim, Bae & Sunwoo, 2019 ) and quantization ( Gong et al, 2014 ) can also accelerate the deep neural networks. Tensor factorization has also been utilized to decompose weights into lightweight pieces ( Masana et al, 2017 ; Yuan & Dong, 2021 ; Zhang & Ng, 2022 ). As the field of artificial intelligence continues to grow and develop, the neural network models used for various tasks have become increasingly larger and more complex.…”

Section: Related Workmentioning

confidence: 99%

GAT TransPruning: progressive channel pruning strategy combining graph attention network and transformer

Lin,

Wang,

Lin

2024

PeerJ Computer Science

View full text Add to dashboard Cite

Recently, large-scale artificial intelligence models with billions of parameters have achieved good results in experiments, but their practical deployment on edge computing platforms is often subject to many constraints because of their resource requirements. These models require powerful computing platforms with a high memory capacity to store and process the numerous parameters and activations, which makes it challenging to deploy these large-scale models directly. Therefore, model compression techniques are crucial role in making these models more practical and accessible. In this article, a progressive channel pruning strategy combining graph attention network and transformer, namely GAT TransPruning, is proposed, which uses the graph attention networks (GAT) and the attention of transformer mechanism to determine the channel-to-channel relationship in large networks. This approach ensures that the network maintains its critical functional connections and optimizes the trade-off between model size and performance. In this study, VGG-16, VGG-19, ResNet-18, ResNet-34, and ResNet-50 are used as large-scale network models with the CIFAR-10 and CIFAR-100 datasets for verification and quantitative analysis of the proposed progressive channel pruning strategy. The experimental results reveal that the accuracy rate only drops by 6.58% when the channel pruning rate is 89% for VGG-19/CIFAR-100. In addition, the lightweight model inference speed is 9.10 times faster than that of the original large model. In comparison with the traditional channel pruning schemes, the proposed progressive channel pruning strategy based on the GAT and Transformer cannot only cut out the insignificant weight channels and effectively reduce the model size, but also ensure that the performance drop rate of its lightweight model is still the smallest even under high pruning ratio.

show abstract

Section: Related Workmentioning

confidence: 99%

GAT TransPruning: progressive channel pruning strategy combining graph attention network and transformer

Lin,

Wang,

Lin

2024

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…The objective functional f (x i ) = 1 2 u 2 i . Then we can easily derive the solution of problem (7), and denoted as Hard-Thresholding operator [29], [39], [40]:…”

Section: Optimization Proceduresmentioning

confidence: 99%

Hyperspectral Image Denoising via $L_{0}$ Regularized Low-Rank Tucker Decomposition

Tian,

Xie,

Zhang

2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

This paper studies the mixed noise removal problem for hyperspectral images (HSIs), which often suffer from Gaussian noise and sparse noise. Conventional denoising models mainly employ the L1-norm-based regularizers to remove sparse noise and ensure piecewise smoothness. However, the denoising performance is poor for highly structured images with severe noise since the L1-norm over-penalizes large entries. To tackle this limitation, we propose a denoising model that combines tensor decomposition with two kinds of L0-norm-based regularizers. Firstly, we use low-rank Tucker decomposition with the Stiefel manifold to characterize the global correlation of HSIs. Then, we utilize the L0-norm to leverage the intrinsic sparsity information of the corruption domain, thereby enhancing the effectiveness of sparse noise removal. Simultaneously, we introduce a weighted L0-norm regularizer on the gradient of each pixel to promote the local spectral-spatial smoothness. To solve the proposed model, we design a hard-thresholding-based alternating direction method of multipliers (ADMM) algorithm. Instead of spending time to find a proper rank in advance, we adopt a rank-increasing scheme to dynamically adjust the tensor rank during the optimization procedure. In this way, our algorithm avoids the rank selection burden and improves computational efficiency. Finally, we test the proposed method on both synthetic and real datasets. Numerical results demonstrate its superiority, especially, the improvements of our method over the best-compared results up to 2.07dB for mixed noise removal.

show abstract

“…Recent efforts have extended NTD to boost calculation efficiency and meet different demands in actual applications by incorporating suitable constrain conditions with NTD, including smoothness [16,17], graph Laplacian [18][19][20][21][22][23][24], sparsity [25], supervision [26][27][28], just to name a few. For examples, Liu et al stated a graph regularized L p smooth NTD method by adding the graph regularization and L p smooth constraint into NTD to retain smooth and more accurate solution of the objective function [17].…”

Section: Introductionmentioning

confidence: 99%

Approximately orthogonal non-negative Tucker decomposition with graph regularized for multiway dimensionality reduction

Gao,

Lu,

Liu

2024

Preprint

View full text Add to dashboard Cite

Non-negative Tucker decomposition (NTD) is one of the renowned technique in feature extraction and representation for non-negative high-dimensional tensor data. The main focus behind NTD is how to factorize the data to get hold of a high quality data representation from multidimensional directions. However, NTD does not conserve the geometrical structure of the data space and does not consider relationship and property among columns of the factor matrices. In this paper, by using approximately orthogonal constraint and graph regularized constraint , we manage to capture nonlinear local features of data space and further enhance expressiveness of the NTD clustering method. First, based on the uni-side and bi-side approximate orthogonality, we propose two novel approximately orthogonal non-negative Tucker decomposition with graph regularized models, which not only in part make factor matrix tend to be orthogonality, but also preserve the geometrical information from high-dimensional tensor data. Then we develop the iterative updating algorithm depended on multiplicative update rule to solve the proposed models, and provide its convergence and computational complexity. Finally, we use numerical experimental results to demonstrate the effectiveness, robustness and efficiency of the proposed new methods on the real-world image datasets.

show abstract

Sparse Nonnegative Tensor Factorization and Completion With Noisy Observations

Cited by 14 publications

References 79 publications

GAT TransPruning: progressive channel pruning strategy combining graph attention network and transformer

GAT TransPruning: progressive channel pruning strategy combining graph attention network and transformer

Hyperspectral Image Denoising via $L_{0}$ Regularized Low-Rank Tucker Decomposition

Approximately orthogonal non-negative Tucker decomposition with graph regularized for multiway dimensionality reduction

Contact Info

Product

Resources

About