On the Achievability of Blind Source Separation for High-Dimensional Nonlinear Source Mixtures

Abstract

For many years, a combination of principal component analysis (PCA) and independent component analysis (ICA) has been used for blind source separation (BSS). However, it remains unclear why these linear methods work well with real-world data that involve nonlinear source mixtures. This work theoretically validates that a cascade of linear PCA and ICA can solve a nonlinear BSS problem accurately—when the sensory inputs are generated from hidden sources via nonlinear mappings with sufficient dimensionality. Our proposed theorem, termed the asymptotic linearization theorem, theoretically guarantees that applying linear PCA to the inputs can reliably extract a subspace spanned by the linear projections from every hidden source as the major components—and thus projecting the inputs onto their major eigenspace can effectively recover a linear transformation of the hidden sources. Then subsequent application of linear ICA can separate all the true independent hidden sources accurately. Zero-element-wise-error nonlinear BSS is asymptotically attained when the source dimensionality is large and the input dimensionality is sufficiently larger than the source dimensionality. Our proposed “Data Availability” section just before the Acknowledgments is validated analytically and numerically. Moreover, the same computation can be performed by using Hebbian-like plasticity rules, implying the biological plausibility of this nonlinear BSS strategy. Our results highlight the utility of linear PCA and ICA for accurately and reliably recovering nonlinearly mixed sources and suggest the importance of employing sensors with sufficient dimensionality to identify true hidden sources of real-world data.

Dimensionality reduction to maximize prediction generalization capability

“…Methods C and D prove that PredPCA inherits preferable properties of both the standard PCA and AR models, and outperforms naïve PCA and AR models in the robustness to noise and generalization of prediction. Methods E and F demonstrate that PredPCA identifies the optimal hidden state estimator and the true system parameters with a global convergence guarantee, owing to asymptotic property of linear neural networks with high-dimensional inputs [33]. Methods G provides the simulation protocols.…”

“…In particular, we demonstrated the following two key properties: (1) It is mathematically guaranteed that PredPCA can identify the optimal (explained below) hidden state representation and parameter estimators̶up to a linear transformation that does not affect prediction accuracy̶for general linear systems and, asymptotically, even for nonlinear systems (Methods E, F). While using a linear neural network for the encoding, the asymptotic linearization theorem [33] ensures that PredPCA will extract true hidden states when the mappings from hidden states to sensory inputs are sufficiently high-dimensional. Briefly, this is because projecting the high-dimensional input onto the directions of its major eigenvectors effectively magnifies the linearly transformed components of the hidden states included in the input, while filtering out the nonlinear components (see Methods E for its mathematical statement and the conditions for application; see [33] for the mathematical proof).…”

“…While using a linear neural network for the encoding, the asymptotic linearization theorem [33] ensures that PredPCA will extract true hidden states when the mappings from hidden states to sensory inputs are sufficiently high-dimensional. Briefly, this is because projecting the high-dimensional input onto the directions of its major eigenvectors effectively magnifies the linearly transformed components of the hidden states included in the input, while filtering out the nonlinear components (see Methods E for its mathematical statement and the conditions for application; see [33] for the mathematical proof).…”

“…The asymptotic linearization theorem [33] was originally introduced to guarantee accurate extraction of independently and identically distributed hidden sources from its high-dimensional nonlinear transformations. In this paper, we use this theorem to prove that the true hidden state 𝑥 " ∈ ℝ ¢ ¥ is accurately estimated from its unknown nonlinear transformation 𝜓(𝑥 " ) = 𝐶𝜌(𝑅𝑥 " + 𝑟) ∈ ℝ ¢ £ with asymptotically zero element-wise error as 𝑁 h and 𝑁 ¤ /𝑁 h diverge.…”

“…This estimator converges to the true Ψ up to the ambiguity of Ω ¤ . The variance of the linearization error term 𝒪Z𝜎 ¤ [ is in the same order as the variance of nonlinearly transformed components of 𝑥 " that are involved in 𝜓 " ; thus, using the asymptotic linearization theorem [33],…”