Bayesian Ying Yang System, Best Harmony Learning, and Gaussian Manifold Based Family

Abstract

Firstly proposed in 1995 and systematically developed in the past decade, Bayesian YingYang learning 1) is a statistical approach for a two pathway featured intelligent system via two complementary Bayesian representations of a joint distribution on the external observation X and its inner representation R, which can be understood from a perspective of the ancient Ying-Yang philosophy. We have q(X, R) = q(X|R)q(R) as Ying that is primary, with its structure designed according to tasks of the system, and p(X, R) = p(R|X)p(X) as Yang that is secondary, with p(X) given by samples of X while the structure of p(R|X) designed from Ying according to a Ying-Yang variety preservation principle, i.e., p(R|X) is designed as a functional with q(X|R), q(R) as its arguments. We call this pair Bayesian Ying-Yang (BYY) system. A YingYang best harmony principle is proposed for learning all the unknowns in the system, in help of an implementation featured by a five action circling under the name of A5 paradigm. Interestingly, it coincides with the famous ancient WuXing theory that provides a general guide to keep the A5 circling well balanced towards a Ying-Yang best harmony. This BYY learning provides not only a general framework that accommodates typical learning approaches from a unified perspective but also a new road that leads to improved model selection criteria, Ying-Yang alternative learning with automatic model selection, as well as coordinated implementation of Ying based model selection and Yang based learning regularization.This paper aims at an introduction of BYY learning in a twofold purpose. On one hand, we introduce fundamentals of BYY learning, including system design

Machine learning problems from optimization perspective

“…The simplest and widely studied example is Gaussian based linear regression, see Table 2c for some examples. Third, in addition to these structures, the knowledge is also represented by jointly, which may be further confined by a background knowledge via a priori structure q( | ) with a unknown parameter set q , for which readers are referred to several choices discussed in [52].…”

“…5b can be determined as an upper bound estimate of k * during parameter learning on * and * by only implementing Stage I in Fig. 5a, which can significantly reduce the computational cost needed for a two stage implementation [52]. However, the performance of this automatic model selection will deteriorate as the sample size N reduces.…”

Theoretical Analysis and Comparison of Several Criteria on Linear Model Dimension Reduction

“…The two formulations, i.e., FA(a) and FA(b), are equivalent under the Maximum Likelihood principle for parameter learning, but they are different under the BYY harmony learning [14] for selecting m which will be introduced in Section 3.3&4.1. In the sequel, we assume µ = 0, Σ e = σ …”

“…However, the conditions will be probably violated when N and SNR is small. Another approach to tackling model selection problems is the Bayesian YingYang (BYY) harmony learning theory [14]. We defer its detailed introduction to the next section.…”

“…One conventional approach is hypothesis tests based on the likelihood ratio statistic [9] and a subjective threshold. Another approach is the two-phase implementation that requires no subjective threshold with the help of an information criterion such as Akaike's Information Criterion (AIC) [10], Bozdogan's Consistent Akaike's Information Criterion (CAIC) [11], Schwarz's Bayesian Information Criterion (BIC) [12] (which coincides with Rissanen's Minimum Description Length (MDL) [13]), and Bayesian Ying-Yang (BYY) harmony learning criterion [14].…”