From Stochastic Grammar to Bayes Network: Probabilistic Parsing of Complex Activity

Abstract

We propose a probabilistic method for parsing a temporal sequence such as a complex activity defined as composition of sub-activities/actions. The temporal structure of the high-level activity is represented by a string-length limited stochastic context-free grammar. Given the grammar, a Bayes network, which we term Sequential Interval Network (SIN), is generated where the variable nodes correspond to the start and end times of component actions. The network integrates information about the duration of each primitive action, visual detection results for each primitive action, and the activity's temporal structure. At any moment in time during the activity, message passing is used to perform exact inference yielding the posterior probabilities of the start and end times for each different activity/action. We provide demonstrations of this framework being applied to vision tasks such as action prediction, classification of the high-level activities or temporal segmentation of a test sequence; the method is also applicable in Human Robot Interaction domain where continual prediction of human action is needed.

Action recognition using saliency learned from recorded human gaze

“…First, a classical SVM misclassification penalty term 35 and second, a term that is related to the saliency within the bounding box. The proposed SVM tries to find the class label and the location of bounding box that optimally balance the minimization of the missclassification cost once a linear classifier is applied on features extracted within the bounding box in question, and the maximization of the sum of the predicted saliencies within the bounding 40 box.…”

“…Usually features based on shape, such as HOG and Scale- and Latent Dirichlet Allocation (LDA) [31] can be used. Action grammars [32, A C C E P T E D M A N U S C R I P T ACCEPTED MANUSCRIPT 33,34,35], models that use graph relations [18,36] and latent SVMs [37,38,39] are also used to model the higher levels of action recognition frameworks.…”

Sequential Interval Network for parsing complex structured activity

“…Qualitative Result: in Figure 7, some example posterior distribution outputs when running our method on a sequence in streaming mode are shown (we encourage readers to watch the supplementary video [3]). At first no observation is available; the distributions are determined by the prior information about the start of the task (which we set to be a uniform in first 30s) and duration models of primitives.…”

“…This is an enhanced version of our previous conference paper [3]. We have added more detail and discussion, as well as additional experiments to validate the proposed method.…”

Gaussian Process for Activity Modeling and Anomaly Detection

“…An abnormal activity is detected if it has low likelihood under a criterion. Typical approaches include Dynamic Bayesian Networks (DBNs) (Swears et al, 2014;Vo and Bobick, 2014) such as Hidden Markov Models (HMM) (Banerjee and Nevatia, 2014). The probabilistic topic models (PTMs) (Kinoshita et al, 2014) such as Latent Dirichlet Allocation (LDA) (Hospedales et al, 2011) or Hierarchical Dirichlet Process (HDP) (Kuettel et al, 2010) are powerful methods to learn activities in surveillance videos.…”