Exploring Compositional High Order Pattern Potentials for Structured Output Learning

Abstract

When modeling structured outputs such as image segmentations, prediction can be improved by accurately modeling structure present in the labels. A key challenge is developing tractable models that are able to capture complex high level structure like shape. In this work, we study the learning of a general class of pattern-like high order potential, which we call Compositional High Order Pattern Potentials (CHOPPs). We show that CHOPPs include the linear deviation pattern potentials of Rother et al. [26] and also Restricted Boltzmann Machines (RBMs); we also establish the near equivalence of these two models. Experimentally, we show that performance is affected significantly by the degree of variability present in the datasets, and we define a quantitative variability measure to aid in studying this. We then improve CHOPPs performance in high variability datasets with two primary contributions: (a) developing a loss-sensitive joint learning procedure, so that internal pattern parameters can be learned in conjunction with other model potentials to minimize expected loss;and (b) learning an image-dependent mapping that encourages or inhibits patterns depending on image features. We also explore varying how multiple patterns are composed, and learning convolutional patterns. Quantitative results on challenging highly variable datasets show that the joint learning and image-dependent high order potentials can improve performance.

HiRF: Hierarchical Random Field for Collective Activity Recognition in Videos

“…We also locally constrain the connectivity between our latent variables over certain temporal windows along the video, in order to identify latent subactivities relevant for recognition of the collective activity. HiRF is also related to Conditional Random Fields of [18,12] which encode higher-order potentials of image features using Restricted Boltzmann Machines. Similarly, HiRF uses a hierarchy of latent variables to identify latent groupings of video features, which amounts to encoding their higher-order dependencies in time and space.…”

Learning undirected graphical models using persistent sequential Monte Carlo

“…Even though some progresses were made to handle densely-connected graphs, they are still restricted to the higher-order potentials of certain specific form, e.g. a function of the cardinality (Li et al 2013) or piece-wise linear potentials (Kohli et al 2009). By contrast, sampling-based methods are more general since they can work on arbitrary UGMs given enough computation resources.…”

Max-Margin Boltzmann Machines for Object Segmentation

“…Li et al [16] combine pairwise, data-dependent potentials with a onelayer RBM prior in CRFs (referred as Compositional High Order Pattern Potentials (CHOPPs) in Figure 1(b)), and show the relationship between the marginalized RBM free energy and high-order potentials [19]. Kae et al [12] augment CRFs with an RBM shape prior in a two-layer model for image labeling.…”

“…Recent work [12,16] on object segmentation realizes the power of Boltzmann Machines to represent high-order interactions in combining RBMs with CRFs. Li et al [16] combine pairwise, data-dependent potentials with a onelayer RBM prior in CRFs (referred as Compositional High Order Pattern Potentials (CHOPPs) in Figure 1(b)), and show the relationship between the marginalized RBM free energy and high-order potentials [19].…”

“…We investigate the effects of four kinds of margin functions on discriminative training, and demonstrate the importance of combined hidden and visible margin functions. We study two variants of MMBMs with a single hidden layer p(y, h|x) as well as two hidden layers p(y, h 1 , h 2 |x), and compare them with two state-of-the-art models: superpixel based CRFs [1] and Compositional High Order Pattern Potentials [16]. We carry out experiments on the Weizmann horse [5], Penn-Fudan pedestrian [4] and Caltech-UCSD birds 200 [27] datasets.…”