A Deflation Method for Structured Probabilistic PCA

Abstract

Modern treatments of structured Principal Component Analysis often focus on the estimation of a single component under various assumptions or priors, such as sparsity and smoothness, and then the procedure is extended to multiple components by sequential estimation interleaved with deflation. While prior work has highlighted the importance of proper deflation for ensuring the quality of the estimated components, to our knowledge, proposed techniques have only been developed and applied to non-probabilistic principal component analyses, and are not trivially extended to probabilistic analyses. This work introduces a novel, robust and efficient deflation method for Probabilistic Principal Component Analysis using tools recently developed for constrained probabilistic estimation via information projection. The components estimated using the proposed deflation regain some of the interpretability of classic PCA such as straightforward estimates of variance explained, while retaining the ability to incorporate rich prior structure. Moreover, sequential estimation allows for scaling probabilistic techniques to be at par with their deterministic counterparts. Experimental results on simulated data demonstrate the utility of the proposed deflation in terms of component recovery, and evaluation on neuroimaging data show both qualitative and quantitative improvements in the quality of the estimated components. We also present timing experiments on real data to illustrate the importance of sequential estimation with proper deflation for scalability.

Understanding Distributed Representations of Concepts in Deep Neural Networks without Supervision

“…To avoid the requirement of supervision, example-based explanation methods select representative exemplars that summarize data distribution (Kim, Khanna, and Koyejo 2016;Khanna et al 2019;Cho et al 2021). Despite the advantages of their unsupervised algorithms, there is no guarantee that the exemplars precisely reflect the decision logic of the model.…”

Layer-Wise De-Training and Re-Training for ConvS2S Machine Translation

“…When analyzing the model parameters, we employ the principal component analysis (PCA) [10,12] method, which is a commonly used data-mapping method in machine learning. The PCA method mainly finds the eigenvectors corresponding to the largest eigenvalues of the covariance matrix of the dataset, thereby finding the directions with the largest data variance, reducing the dimensionality of the data, and reducing an n-dimensional vector to a d-dimension, wherein d < n. Fig.…”

Information Projection and Approximate Inference for Structured Sparse Variables

“…where x, w are vectors. The general matrix case follows using standard deflation techniques for multiple factors [8,21,9]. The generative model for the observed data matrix is T = xw + , where ∼ N (0, σ 2 ).…”