Twin Gaussian Processes for Structured Prediction

Abstract

We describe twin Gaussian processes (TGP), a generic structured prediction method that uses Gaussian process (GP) priors on both covariates and responses, both multivariate, and estimates outputs by minimizing the Kullback-Leibler divergence between two GP modeled as normal distributions over finite index sets of training and testing examples, emphasizing the goal that similar inputs should produce similar percepts and this should hold, on average, between their marginal distributions. TGP captures not only the interdependencies between covariates, as in a typical GP, but also those between responses, so correlations among both inputs and outputs are accounted for. TGP is exemplified, with promising results, for the reconstruction of 3d human poses from monocular and multicamera video sequences in the recently introduced HumanEva benchmark, where we achieve 5 cm error on average per 3d marker for models trained jointly, using data from multiple people and multiple activities. The method is fast and automatic: it requires no hand-crafting of the initial pose, camera calibration parameters, or the availability of a 3d body model associated with human subjects used for training or testing.

Domain Adaptation for Structured Regression

“…Discriminative regression: Over the past 10 years many methods have been introduced that include both parametric (e.g., conditional mixture of experts (Kanaujia et al, 2007;Sminchisescu et al, 2006)) and nonparametric (e.g., nearest neighbor regression (Shakhnarovich et al, 2003), linear locally-weighted regression (Shakhnarovich et al, 2003), regression forests (Sun et al, 2012), local Gaussian process regression (Urtasun and Darrell, 2008), twin Gaussian processes regression (Bo and Sminchisescu, 2010), etc.) models.…”

“…In addition, we analyze the relationship between domain similarity and effectiveness of proposed USDA vs. SSDA methods. Moreover, we propose a computationally efficient alternative to TGP (Bo and Sminchisescu, 2010), and it's variants, called the direct TGP (dTGP). We show that our model outperforms a number of baselines, on two public datasets: HumanEva and ETH Face Pose Range Image Dataset.…”

“…A success of such methods in 3D full-body pose estimation is evident from recent results that use Microsoft Kinect sensor (Girshick et al, 2011;Sun et al, 2012); such discriminative methods have also proved effective for other problems, including image-based 3D pose (Bo and Sminchisescu, 2010;Kanaujia et al, 2007;Shakhnarovich et al, 2003;Sminchisescu et al, 2006;Urtasun and Darrell, 2008), head pose (Fanelli et al, 2011) and body shape (Chen et al, 2011;Sigal et al, 2007) estimation. The typical goal of discriminative regression methods is to learn a direct (and sometimes multi-modal) mapping, f : R dx → R dy , from features (e.g., computed from image or depth data) to pose (e.g., 3D position and orientation of the head, or full 3D pose of the body encoded by joint positions or joint angles).…”

“…Finally, we use the resulting weighted augmented samples to learn a non-linear multivariate regression model in the form of Importance Weighted Twin Gaussian Processes (Yamada et al, 2012). Moreover, we propose a computationally efficient alternative to TGP (Bo and Sminchisescu, 2010), that we call direct TGP (dTGP). The benefit of dTGP is that the learning and inference can be carried out by using only simple linear algebra.…”

Articulated tracking with manifold regularized particle filter

“…This approach is usually composed of two parts where feature extraction is followed by prediction using multivariate regression model. To obtain informative features simple techniques like binary silhouettes [1,14] as well as more sophisticated descriptors like histogram of oriented gradients or a HMAX model [3] were adapted. As a regression model a whole spectrum of different techniques were used, e.g., ridge regression and support vector machines [1], mixture of experts [10], gaussian processes [3], kernel information embedding [14].…”