Learning latent temporal structure for complex event detection

Tang, Kevin; Li, Feifei; Koller, Daphne

doi:10.1109/cvpr.2012.6247808

Cited by 309 publications

(281 citation statements)

References 28 publications

Supporting

Mentioning

280

Contrasting

Order By: Relevance

“…Most prior works in robotics and computer vision formulate the activity recognition problem as Conditional Random Fields (CRFs) [4,5,6,9,10,14,15,20,21]. The CRFs model the environment with nodes and edges.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Recognize Human Activities from Soft Labeled Data

Lou²,

Englebienne³

et al. 2014

Robotics: Science and Systems X

View full text Add to dashboard Cite

Abstract-An activity recognition system is a very important component for assistant robots, but training such a system usually requires a large and correctly labeled dataset. Most of the previous works only allow training data to have a single activity label per segment, which is overly restrictive because the labels are not always certain. It is, therefore, desirable to allow multiple labels for ambiguous segments. In this paper, we introduce the method of soft labeling, which allows annotators to assign multiple, weighted, labels to data segments. This is useful in many situations, e.g. when the labels are uncertain, when part of the labels are missing, or when multiple annotators assign inconsistent labels. We treat the activity recognition task as a sequential labeling problem. Latent variables are embedded to exploit sub-level semantics for better estimation. We propose a novel method for learning model parameters from soft-labeled data in a max-margin framework. The model is evaluated on a challenging dataset (CAD-120), which is captured by a RGB-D sensor mounted on the robot. To simulate the uncertainty in data annotation, we randomly change the labels for transition segments. The results show significant improvement over the state-of-the-art approach.

show abstract

Section: Related Workmentioning

confidence: 99%

“…1). This topic has been widely studied in both robotics communities [3,5,10,14] and other fields [15,19,20]. Most of the work uses a dataset where labels are hard assigned regardless of uncertainty.…”

Section: Introductionmentioning

confidence: 99%

Learning to Recognize Human Activities from Soft Labeled Data

Lou²,

Englebienne³

et al. 2014

Robotics: Science and Systems X

View full text Add to dashboard Cite

show abstract

“…However, EM requires an arbitrary initialization at its turn. In [12], the initial states are first assigned with a unique label, and then the number of labels is reduced by agglomerative clustering. In this work, we propose initialization strategies inspired by the assumed semantics for the states:…”

Section: Latent State Initializationmentioning

confidence: 99%

“…Recently, works such as [12], [13] have demonstrated the importance of temporal structure in complex event recognition. In this paper, we propose to combine the use of trained concept detectors with a latent temporal model.…”

Section: Introduction and Related Workmentioning

confidence: 99%

Complex event recognition by latent temporal models of concepts

Borzeshi

Dehghan

Piccardi

et al. 2014

2014 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Complex event recognition is an expanding research area aiming to recognize entities of high-level semantics in videos. Typical approaches exploit the so-called "bags" of spatiotemporal features such as STIP, ISA and DTF-HOG; yet, more recently, the notion of concept has emerged as an alternative, intermediate representation with greater descriptive power, and "bags of concepts" have been used for recognition. In this paper we argue that concepts in an event tend to articulate over a discernible temporal structure and we exploit a temporal model using the scores of concept detectors as measurements. In addition, we propose several heuristics to improve the initialization of the model's latent states and take advantage of the time-sparsity of the concepts. Experimental results on videos from the challenging TRECVID MED 2012 dataset show that the proposed approach achieves an improvement in average precision of 8.92% over comparable bags of concepts, thus validating the use of temporal structure over concepts for complex event recognition.

show abstract

“…(e.g. HMMs [3], Dynamic Bayesian Networks [24], prototype trees [11], AND-OR graphs [23], latent SVM [22], Sum-Product Network [4], and Markov Logic Networks [16]). The key advantage of graphical structures is that they model the dependence of actions by local relationships while allowing for the joint optimization of a global task-dependent objective function.…”

Section: Introductionmentioning

confidence: 99%

Clauselets: Leveraging Temporally Related Actions for Video Event Analysis

Lee

Morariu

Davis

2015

2015 IEEE Winter Conference on Applications of Computer Vision

View full text Add to dashboard Cite

We propose clauselets, sets of concurrent actions and their temporal relationships, and explore their application to video event analysis. We train clauselets in two stages. We initially train first level clauselet detectors that find a limited set of actions in particular qualitative temporal configurations based on Allen's interval relations. In the second stage, we apply the first level detectors to training videos, and discriminatively learn temporal patterns between activations that involve more actions over longer durations and lead to improved second level clauselet models. We demonstrate the utility of clauselets by applying them to the task of "in-the-wild" video event recognition on the TRECVID MED 11 dataset. Not only do clauselets achieve state-of-the-art results on this task, but qualitative results suggest that they may also lead to semantically meaningful descriptions of videos in terms of detected actions and their temporal relationships.

show abstract

Learning latent temporal structure for complex event detection

Cited by 309 publications

References 28 publications

Learning to Recognize Human Activities from Soft Labeled Data

Learning to Recognize Human Activities from Soft Labeled Data

Complex event recognition by latent temporal models of concepts

Clauselets: Leveraging Temporally Related Actions for Video Event Analysis

Contact Info

Product

Resources

About