Geometric Understanding of Deep Learning

Lei, Na; Luo, Zhongxuan; Yau, Shing‐Tung; Gu, Daqing

doi:10.48550/arxiv.1805.10451

Cited by 15 publications

(23 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DeconvNet (Zeiler and Fergus 2014), LIME (Ribeiro, Singh, and Guestrin 2016) and SincNet (Ravanelli and Bengio 2018) trains a new model to explain the trained model. Geometric analyis could also reveal the internal structure indirectly (Montufar et al 2014;Lei et al 2018;Fawzi et al 2018). The activation maximization (Erhan, Courville, and Bengio 2010), or GANs (Nguyen et al 2016) have been used to explain the neural network by using examples.…”

Section: Related Workmentioning

confidence: 99%

“…Although such regions are complicated, each region for a single classification in DNN classifiers is shown to be topologically connected (Fawzi et al 2018). It has also been shown that the manifolds learned by DNNs and distributions over them are highly related to the representation capability of a network (Lei et al 2018).…”

Section: Geometric Analysis On the Inside Of Deep Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networks

Jeon

Jeong

Choi

2020

AAAI

View full text Add to dashboard Cite

Deep generative neural networks (DGNNs) have achieved realistic and high-quality data generation. In particular, the adversarial training scheme has been applied to many DGNNs and has exhibited powerful performance. Despite of recent advances in generative networks, identifying the image generation mechanism still remains challenging. In this paper, we present an explorative sampling algorithm to analyze generation mechanism of DGNNs. Our method efficiently obtains samples with identical attributes from a query image in a perspective of the trained model. We define generative boundaries which determine the activation of nodes in the internal layer and probe inside the model with this information. To handle a large number of boundaries, we obtain the essential set of boundaries using optimization. By gathering samples within the region surrounded by generative boundaries, we can empirically reveal the characteristics of the internal layers of DGNNs. We also demonstrate that our algorithm can find more homogeneous, the model specific samples compared to the variations of ϵ-based sampling method.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Geometric Analysis On the Inside Of Deep Neural Networkmentioning

confidence: 99%

An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networks

Jeon

Jeong

Choi

2020

AAAI

View full text Add to dashboard Cite

show abstract

“…Measuring certain complexities of a fixed neural network via counting linear pieces arises in several recent works (e.g. (Lei et al, 2018)), and the question whether or not the dependence on n is redundant (comparing to that of sparse recovery guarantees) warrants further studies.…”

Section: Statistical Guaranteementioning

confidence: 99%

Robust One-Bit Recovery via ReLU Generative Networks: Near-Optimal Statistical Rate and Global Landscape Analysis

Qiu,

Wei,

Yang

2019

Preprint

View full text Add to dashboard Cite

We study the robust one-bit compressed sensing problem whose goal is to design an algorithm that faithfully recovers any sparse target vector θ 0 ∈ R d uniformly m quantized noisy measurements. Under the assumption that the measurements are sub-Gaussian random vectors, to recover any ksparse θ 0 (k ≪ d) uniformly up to an error ε with high probability, the best known computationally tractable algorithm requires 1 m ≥ O(k log d/ε 4 ) measurements. In this paper, we consider a new framework for the one-bit sensing problem where the sparsity is implicitly enforced via mapping a low dimensional representation x 0 ∈ R k through a known n-layer ReLU generative network G : R k → R d . Such a framework poses low-dimensional priors on θ 0 without a known basis. We propose to recover the target G(x 0 ) via an unconstrained empirical risk minimization (ERM) problem under a much weaker sub-exponential measurement assumption. For such a problem, we establish a joint statistical and computational analysis. In particular, we prove that the ERM estimator in this new framework achieves an improved statistical rate of m = O(kn log d/ε 2 ) recovering any G(x 0 ) uniformly up to an error ε. Moreover, from the lens of computation, despite non-convexity, we prove that the objective of our ERM problem has no spurious stationary point, that is, any stationary point are equally good for recovering the true target up to scaling with a certain accuracy. Furthermore, our analysis also shed lights on the possibility of inverting a deep generative model under partial and quantized measurements, complementing the recent success of using deep generative models for inverse problems.

show abstract

“…Manifold learning has been widely applied in many computer vision tasks, such as the face recognition [43], [44], image classification [28], as well as in the literature of hyperspectral image [42], [29]. Generally, a data manifold follows the law of manifold distribution: in real-world applications, high-dimensional data of the same class usually draws close to a low dimensional manifold [21]. Therefore, hyperspectral images, which provide a dense spectral sampling at each pixel, possess good intrinsic manifold structure.…”

Section: Introductionmentioning

confidence: 99%

Deep Manifold Embedding for Hyperspectral Image Classification

Gong¹,

et al. 2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Deep learning methods have played a more and more important role in hyperspectral image classification. However, the general deep learning methods mainly take advantage of the information of sample itself or the pairwise information between samples while ignore the intrinsic data structure within the whole data. To tackle this problem, this work develops a novel deep manifold embedding method(DMEM) for hyperspectral image classification. First, each class in the image is modelled as a specific nonlinear manifold and the geodesic distance is used to measure the correlation between the samples. Then, based on the hierarchical clustering, the manifold structure of the data can be captured and each nonlinear data manifold can be divided into several sub-classes. Finally, considering the distribution of each sub-class and the correlation between different subclasses, the DMEM is constructed to preserve the estimated geodesic distances on the data manifold between the learned low dimensional features of different samples. Experiments over three real-world hyperspectral image datasets have demonstrated the effectiveness of the proposed method.

show abstract

Geometric Understanding of Deep Learning

Cited by 15 publications

References 15 publications

An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networks

An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networks

Robust One-Bit Recovery via ReLU Generative Networks: Near-Optimal Statistical Rate and Global Landscape Analysis

Deep Manifold Embedding for Hyperspectral Image Classification

Contact Info

Product

Resources

About