A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

Abstract

Abstract-Typical dimensionality reduction (DR) methods are data-oriented, focusing on directly reducing the number of random variables (or features) while retaining the maximal variations in the high-dimensional data. Targeting unsupervised situations, this paper aims to address the problem from a novel perspective and considers model-oriented dimensionality reduction in parameter spaces of binary multivariate distributions. Specifically, we propose a general parameter reduction criterion, called Confident-Information-First (CIF) principle, to maximally preserve confident parameters and rule out less confident ones. Formally, the confidence of each parameter can be assessed by its contribution to the expected Fisher information distance within a geometric manifold over the neighbourhood of the underlying real distribution. Then we demonstrate two implementations of CIF in different scenarios. First, when there are no observed samples, we revisit the Boltzmann Machines (BM) from a model selection perspective and theoretically show that both the fully visible BM (VBM) and the BM with hidden units can be derived from the general binary multivariate distribution using the CIF principle. This finding would help us uncover and formalize the essential parts of the target density that BM aims to capture and the non-essential parts that BM should discard. Second, when there exist observed samples, we apply CIF to the model selection for BM, which is in turn made adaptive to the observed samples. The sample-specific CIF is a heuristic method to decide the priority order of parameters, which can improve the search efficiency without degrading the quality of model selection results as shown in a series of density estimation experiments.

Regularizing Neural Networks via Retaining Confident Connections

“…Two coordinate parameters ξ i and ξ j are called orthogonal if and only if their Fisher information vanishes, i.e., g ij = 0, meaning that their influences on the log likelihood function are uncorrelated. Based on G η and G θ , we can calculate the Fisher information matrix G ζ for the mixed coordinates [ζ] l [13]:…”

“…Specifically, we would need an efficient approach to recognize how much a unit or connection is useful for revealing the underlying structure of the current data. Recently, a general parametric reduction criterion, named the Confident-Information-First principle (CIF) [13], has been proposed, in the theoretical framework of Information Geometry (IG) [14]. From a model selection perspective, they proved that both the fully Visible Boltzmann Machine (VBM) and the Boltzmann Machine (BM) with hidden units can be derived from the general multivariate binary distribution using the CIF principle.…”

Exploiting Document Boltzmann Machine in Query Extension

“…Thus, we can give strong term associations more influence. Formally, we define a weight when calculating the query likelihood for node andnode : (6) With this query likelihood, we combine the original query terms and extended query terms with proper and automatically learned weight. This extended query not only considers the single words, but also the associations between query terms.…”

“…Recently, A confident information First (CIF) Principle [6] is proposed for parameter reduction criterion. The proposed CIF is fundamentally different from the traditional feature reduction method.…”

“…It is worth emphasizing that the CIF principle can reduce the time complexity of the model selection procedure for BM. Moreover, there is an elegant algorithm proposed in [6] (see Algorithm 1 in [6]). This algorithm give a model selection procedure for BM without manually assigned parameter, which appeals since we hope to avoid introducing extra parameters into our rDBM framework.…”