Deep Component Analysis via Alternating Direction Neural Networks

Murdock, Calvin; Chang, Ming-Fang; Lucey, Simon

doi:10.1007/978-3-030-01267-0_50

Cited by 15 publications

(25 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…], showing that cascading basis pursuit problems can lead to competitive deep learning constructions. However, the Layered Basis Pursuit formulation, or other similar variations that attempt to unfold neural network architectures [33], [44], do not minimize (P ) and thus their solutions only represent sub-optimal and heuristic approximations to the minimizer of the multi-layer BP. More clearly, such a series of steps never provide estimatesγ i that can generate a signal according to the multi-layer sparse model.…”

Section: Algorithmsmentioning

confidence: 99%

“…While other works have indeed explored the unrolling of iterative algorithms in terms of CNNs (e.g. [33], [44]), we are not aware of any work that has attempted nor studied the unrolling of a global pursuit with convergence guarantees. Lastly, we demonstrate the performance of these networks in practice by training our models for image classification, consistently improving on the classical feed-forward architectures without introducing filters nor any other extra parameters in the model.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Sulam

Aberdam

Beck

et al. 2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representation layers in a symbiotic relation between synthesis and analysis sparse priors. We explore different iterative methods to solve this new problem in practice, and we propose a new Multi-Layer Iterative Soft Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show that these nested first order algorithms converge, in the sense that the function value of near-fixed points can get arbitrarily close to the solution of the original problem. We further show how these algorithms effectively implement particular recurrent convolutional neural networks (CNNs) that generalize feed-forward ones without introducing any parameters. We present and analyze different architectures resulting unfolding the iterations of the proposed pursuit algorithms, including a new Learned ML-ISTA, providing a principled way to construct deep recurrent CNNs. Unlike other similar constructions, these architectures unfold a global pursuit holistically for the entire network. We demonstrate the emerging constructions in a supervised learning setting, consistently improving the performance of classical CNNs while maintaining the number of parameters constant.

show abstract

Section: Algorithmsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Sulam

Aberdam

Beck

et al. 2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…[29] and [45] attempt to unfold neural networks with iterative thresholding and minimizing ML-BP. As a result, each representation estimate is required to explain the immediate layer only and a signal based on generation in global multilayer sparse model settings is not possible.…”

Section: E Layered Basis Pursuitmentioning

confidence: 99%

Multi-Layer Basis Pursuit for Compressed Sensing MR Image Reconstruction

et al. 2020

View full text Add to dashboard Cite

Compressive Sensing (CS) is a widely used technique in biomedical signal acquisition and reconstruction. The technique is especially useful for reducing acquisition time for magnetic resonance imaging (MRI) signal acquisitions and reconstruction, where effects of patient fatigue and Claustrophobia need mitigation. In addition to improving patient experience, faster MRI scans are important for time sensitive imaging, such as functional or cardiac MRI, where target movement is unavoidable. Inspired from recent research works on multi-layer convolutional sparse coding (ML-CSC) theory to model deep neural networks, this work proposes a multi-layer basis pursuit framework which combines the benefit from objective-based CS reconstructions and deep learning-based reconstruction by employing iterative thresholding algorithms for successfully training a CS-MRI restoration framework on GPU and reconstruct test images using parameters of the trained model. Extensive experiments show the effectiveness of the proposed framework on four MRI datasets in terms of faster convergence, improved PSNR/SSIM, and better restoration efficiency as compared to the state of the art frameworks with different CS ratios. INDEX TERMS Compressive sensing, inverse problems in imaging, iterative thresholding algorithms, multi-layered convolutional sparse coding

show abstract

“…Building upon recent connections between deep learning and sparse approximation [7], [8], [9], we introduce…”

Section: Introductionmentioning

confidence: 99%

Reframing Neural Networks: Deep Structure in Overcomplete Representations

Murdock¹,

Cazenavette²,

Lucey³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In comparison to classical shallow representation learning techniques, deep neural networks have achieved superior performance in nearly every application benchmark. But despite their clear empirical advantages, it is still not well understood what makes them so effective. To approach this question, we introduce deep frame approximation, a unifying framework for representation learning with structured overcomplete frames. While exact inference requires iterative optimization, it may be approximated by the operations of a feed-forward deep neural network. We then indirectly analyze how model capacity relates to the frame structure induced by architectural hyperparameters such as depth, width, and skip connections. We quantify these structural differences with the deep frame potential, a data-independent measure of coherence linked to representation uniqueness and stability. As a criterion for model selection, we show correlation with generalization error on a variety of common deep network architectures such as ResNets and DenseNets. We also demonstrate how recurrent networks implementing iterative optimization algorithms achieve performance comparable to their feed-forward approximations. This connection to the established theory of overcomplete representations suggests promising new directions for principled deep network architecture design with less reliance on ad-hoc engineering.

show abstract

Deep Component Analysis via Alternating Direction Neural Networks

Cited by 15 publications

References 38 publications

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Multi-Layer Basis Pursuit for Compressed Sensing MR Image Reconstruction

Reframing Neural Networks: Deep Structure in Overcomplete Representations

Contact Info

Product

Resources

About