Explaining Activities as Consistent Groups of Events

Abstract

We propose a method for disambiguating uncertain detections of events by seeking global explanations for activities. Given a noisy visual input, and exploiting our knowledge of the activity and its constraints, one can provide a consistent set of events explaining all the detections. The paper presents a complete framework that starts with a general way to formalise the set of global explanations for a given activity using attribute multiset grammars (AMG). AMG combines the event hierarchy with the necessary features for recognition and all natural constraints. Parsing a set of detections by such a grammar finds a consistent set of events that satisfies the activity's natural constraints. Each parse tree has a posterior probability in a Bayesian sense. To find the best parse tree, the grammar and a finite set of detections are mapped into a Bayesian Network (BN). The set of possible labellings of the Bayesian network corresponds to the set of all parse trees for a given set of detections. We compare greedy, multiplehypotheses trees, reversible jump MCMC, and integer programming for finding the Maximum a Posteriori (MAP) solution over the space of explanations. The framework is tested for two applications; the activity in a bicycle rack and around a building entrance.

Attribute And-Or Grammar for Joint Parsing of Human Pose, Parts and Attributes

“…In contrast to the hard deterministic constraints in compiler writing, the attribute grammar in scene parsing uses soft constraints in the form of energy terms. Later, the attribute grammar is adopted for action representations [18], [19], scene attribute tagging [20], and 3D scene construction from a single view [21]. The grammar rules are manually designed in the scene parsing work [17] and 3D reconstruction work [21].…”

Sequential Interval Network for parsing complex structured activity

“…In [14,15], traditional stochastic context free grammar parsing algorithm was adapted for Computer Vision problems; adjustments were made to handle different types of errors. To the same end, Damen et al [16] proposed Attribute Multiset Grammars which can encode richer constraints on activity structure. For parsing, an automatically generated Bayesian Network is used to find the detections that correspond to the best explanation.…”

From Stochastic Grammar to Bayes Network: Probabilistic Parsing of Complex Activity

“…In [11,16], traditional stochastic context free grammar parsing algorithm was adapted for Computer Vision problems; adjustments were made to handle different types of errors. To the same end, Damen et al [4] proposed Attribute Multiset Grammars which can encode richer constraints on activity structure. For parsing, an automatically generated Bayesian Network is used to find the detections that correspond to the best explanation.…”