Simplifying, Regularizing and Strengthening Sum-Product Network Structure Learning

Abstract

The need for feasible inference in Probabilistic Graphical Models (PGMs) has lead to tractable models like Sum-Product Networks (SPNs). Their highly expressive power and their ability to provide exact and tractable inference make them very attractive for several real world applications, from computer vision to NLP. Recently, great attention around SPNs has focused on structure learning, leading to different algorithms being able to learn both the network and its parameters from data. Here, we enhance one of the best structure learner, LearnSPN, aiming to improve both the structural quality of the learned networks and their achieved likelihoods. Our algorithmic variations are able to learn simpler, deeper and more robust networks. These results have been obtained by exploiting some insights in the building process done by LearnSPN, by hybridizing the network adopting tree-structured models as leaves, and by blending bagging estimations into mixture creation. We prove our claims by empirically evaluating the learned SPNs on several benchmark datasets against other competitive SPN and PGM structure learners

Automatic Bayesian Density Analysis

“…To randomly create a ground truth generative model, i.e., an SPN-denoted as S -we simulate a stochastic guillotine partitioning of a fictitious N × D data matrix. Specifically, we follow a LearnSPN-like [14,34] structure learning scheme in which columns and rows of this matrix are clustered together, in this case in a random fashion.…”

Deep Model Compression and Inference Speedup of Sum-Product Networks on Tensor Trains

“…To articulate with tensor-based reformulation of the SPN, we will need to slightly modify the induced tree (Definition 3) by terminating at the leaf nodes, i.e., the bottom nodes of the In fact, prevailing SPN learning algorithms or alike, e.g, LearnSPN, SPN-B and SPN-BT [11], [2], all produce SPN trees terminating at leaf nodes. Although SPN illustrations often utilize networks with shared weights (e.g., the two top branches in Fig.…”

“…denote scalars. A set of d tensors, like that of a tensor train (TT), is denoted as A (1) , A (2) , . .…”

“…S INCE the inception of sum-product networks (SPNs) [1], a multitude of works have emerged with respect to their structure and weight learning, e.g., [2], [3], [4], as well as their application in image completion, speech modeling, semantic mapping and robotics, e.g., [5], just to name a few. An SPN exhibits a clear semantics of mixtures (sum nodes) and features (product nodes).…”

“…We focus on structure compression rather than the learning of SPNs. Indeed, our work leverages on the wealth of existing SPN learning algorithms such as [11], [2], [3], and attempts to turn their inherently wide SPN tree outputs (owing to the intrinsic way of learning through partitioning the data matrix) into a "deep" tree via tensor network techniques subject to a non-negativity constraint. This is in-line with the recent * C.Y.…”