No abstract

The training of two-layer neural networks with nonlinear activation functions is an important non-convex optimization problem with numerous applications and promising performance in layerwise deep learning. In this paper, we develop exact convex optimization formulations for two-layer neural networks with second degree polynomial activations based on semidefinite programming. Remarkably, we show that semidefinite lifting is always exact and therefore computational complexity for global optimization is polynomial in the input dimension and sample size for all input data. The developed convex formulations are proven to achieve the same global optimal solution set as their non-convex counterparts. More specifically, the globally optimal two-layer neural network with polynomial activations can be found by solving a semidefinite program (SDP) and decomposing the solution using a procedure we call Neural Decomposition. Moreover, the choice of regularizers plays a crucial role in the computational tractability of neural network training. We show that the standard weight decay regularization formulation is NP-hard, whereas other simple convex penalties render the problem tractable in polynomial time via convex programming. We extend the results beyond the fully connected architecture to different neural network architectures including networks with vector outputs and convolutional architectures with pooling. We provide extensive numerical simulations showing that the standard backpropagation approach often fails to achieve the global optimum of the training loss. The proposed approach is significantly faster to obtain better test accuracy compared to the standard backpropagation procedure.

No abstract

Sparse recovery aims to reconstruct signals that are sparse in a linear transform domain from a heavily underdetermined set of measurements. The success of sparse recovery relies critically on the knowledge of transform domains that give compressible representations of the signal of interest. Here we consider two-and three-dimensional images, and investigate various multi-dimensional transforms in terms of the compressibility of the resultant coefficients. Specifically, we compare the fractional Fourier (FRT) and linear canonical transforms (LCT), which are generalized versions of the Fourier transform (FT), as well as Hartley and simplified fractional Hartley transforms, which differ from corresponding Fourier transforms in that they produce real outputs for real inputs. We also examine a cascade approach to improve transform-domain sparsity, where the Haar wavelet transform is applied following an initial Hartley transform. To compare the various methods, images are recovered from a subset of coefficients in the respective transform domains. The number of coefficients that are retained in the subset are varied systematically to examine the level of signal sparsity in each transform domain. Recovery performance is assessed via the structural similarity index (SSIM) and mean squared error (MSE) in reference to original images. Our analyses show that FRT and LCT transform yield the most sparse representations among the tested transforms as dictated by the improved quality of the recovered images. Furthermore, the cascade approach improves transform-domain sparsity among techniques applied on small image patches.

We propose a serverless computing mechanism for distributed computation based on polar codes. Serverless computing is an emerging cloud based computation model that lets users run their functions on the cloud without provisioning or managing servers. Our proposed approach is a hybrid computing framework that carries out computationally expensive tasks such as linear algebraic operations involving large-scale data using serverless computing and does the rest of the processing locally. We address the limitations and reliability issues of serverless platforms such as straggling workers using coding theory, drawing ideas from recent literature on coded computation. The proposed mechanism uses polar codes to ensure straggler-resilience in a computationally effective manner. We provide extensive evidence showing polar codes outperform other coding methods. We have designed a sequential decoder specifically for polar codes in erasure channels with full-precision input and outputs. In addition, we have extended the proposed method to the matrix multiplication case where both matrices being multiplied are coded. The proposed coded computation scheme is implemented for AWS Lambda. Experiment results are presented where the performance of the proposed coded computation technique is tested in optimization via gradient descent. Finally, we introduce the idea of partial polarization which reduces the computational burden of encoding and decoding at the expense of straggler-resilience.