Learning Coefficient of Generalization Error in Bayesian Estimation and Vandermonde Matrix-Type Singularity

Abstract

The term algebraic statistics arises from the study of probabilistic models and techniques for statistical inference using methods from algebra and geometry (Sturmfels, 2009 ). The purpose of our study is to consider the generalization error and stochastic complexity in learning theory by using the log-canonical threshold in algebraic geometry. Such thresholds correspond to the main term of the generalization error in Bayesian estimation, which is called a learning coefficient (Watanabe, 2001a , 2001b ). The learning coefficient serves to measure the learning efficiencies in hierarchical learning models. In this letter, we consider learning coefficients for Vandermonde matrix-type singularities, by using a new approach: focusing on the generators of the ideal, which defines singularities. We give tight new bound values of learning coefficients for the Vandermonde matrix-type singularities and the explicit values with certain conditions. By applying our results, we can show the learning coefficients of three-layered neural networks and normal mixture models.

Consideration on Singularities in Learning Theory and the Learning Coefficient

“…These two theorems enable us to obtain a new bound for the log canonical thresholds of Vandermonde matrix-type singularities in Theorem 5. These bounds are much tighter than those in [14].…”

“…By applying these methods to Vandermonde matrix-type singularities and using the inclusion of ideals and recursive blowup from algebraic geometry, we found new bounds on learning coefficients for Vandermonde matrix-type singularities. These bounds are much tighter than those in [14]. Our future research aim is to improve our methods and to obtain exact values for the general machine model.…”

“…We compare these values with those obtained by the past work in [14]. In the figures, the horizontal axis is the number, M , and the vertical one, the value of such bounds.…”

“…We had other exact values when H is small on paper [14]. Both sets of exact values are the bounded values in Theorem 5.…”

“…The paper [14] derived bounds on the learning coefficients for the Vandermonde matrix-type singularities and explicit values under some conditions. The learning coefficients for the restricted Boltzmann machine [19] have also been considered recently.…”