A New Bayesian Modeling for 3D Human-Object Action Recognition

Maurice, Camille; Madrigal, Francisco; Monin, A.; Lerasle, Frédéric

doi:10.1109/avss.2019.8909873

Cited by 3 publications

(10 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section we describe the proposed architecture for the fusion of probabilities predicted by the ANBM [9] Bayesian approach with those of the modified C3D [3] network. We recall our previous ANBM approach in Section III-A, and then briefly describe the C3D network in section III-B and its modification (C3D-GRU) in Section III-C.…”

Section: Proposed Approachmentioning

confidence: 99%

“…This approach [9] is based on the following insights: human pose, human-object and object-to-object interactions, performed during the execution of an action, provide spatiotemporal information that allows the recognition of the ongoing action. Moreover, it considers temporal information such as transition between actions during a sequence.…”

Section: A Bayesian Approach With Human-object Observationsmentioning

confidence: 99%

“…It allows ANBM 235 to be more robust to changes of point of view than an approach 236 based solely on 2D spatial characteristics. We invite the reader 237 to consult [9] the paper for more in-depth details.…”

Section: A Bayesian Approach With Human-object Observationsmentioning

confidence: 99%

“…Koppula and Saxena [8] propose the use of conditional random fields (CRFs) to model the scene and the spatio-temporal relationships that appear in CAD-120 [8]. More recently, we have proposed a new Bayesian ANBM [9] approach based on explicit 3D modeling of contextual features, both spatially and temporally. These approaches rely on a smaller number of parameters than those of C3D networks.…”

mentioning

confidence: 99%

“…We propose to merge two spatio-temporal approaches, one based on context modeling via learning such as C3D [3] and our ANBM [9] approach based on Bayesian models and 3D human and objects observations of the scene. This fusion is not done at the feature level but later at their predictions level towards the same layer.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Late Fusion of Bayesian and Convolutional Models for Action Recognition

Maurice

Madrigal

Lerasle

2021

2020 25th International Conference on Pattern Recognition (ICPR)

Self Cite

View full text Add to dashboard Cite

The activities we do in our daily-life are generally carried out as a succession of atomic actions, following a logical order. During a video sequence, actions usually follow a logical order. In this paper, we propose a hybrid approach resulting from the fusion of a deep learning neural network with a Bayesianbased approach. The latter models human-object interactions and transition between actions. The key idea is to combine both approaches in the final prediction. We validate our strategy in two public datasets: CAD-120 and Watch-n-Patch. We show that our fusion approach yields performance gains in accuracy of respectively +4 percentage points (pp) and +6 pp over a baseline approach. Temporal action recognition performances are clearly improved by the fusion, especially when classes are imbalanced.65 less, they generally require less data because they also have 66 fewer underlying free parameters to tune. Therefore their 67 interpretability is less dependent on the available learning 68 data (e.g. less subject to over-fitting). These approaches are 69 relevant in the case of a small number of samples available 70 for training. For example, our previous Bayesian approach for 71 action recognition ANBM (for A New Bayesian Model [9]), 72 models both the interactions between objects and human-73 objects through about 50 parameters. Let us note that our 74 ANBM approach also takes into account the transitions be-75 tween different actions in order to ensure temporal consistency 76 throughout the sequence of actions. 77 Building on the observation of a possible synergy of the two 78 approaches, we propose a hybrid framework with a fusion at 79 the decision level, of a C3D [3] convolutional network and our 80 probabilistic ANBM [9] approach based on explicit human-81 object observations.These two approaches take into account 82 the spatio-temporal characteristics of the different classes of 83 actions. Due to the large number of parameters, the C3D 84 network needs a lot of annotated data to be relevant since 85 learning is difficult in the case of under-represented classes. 86 The ANBM approach depends on handcrafted models and 87 even with a little data the prediction of under-represented 88 classes is possible. 89 Thus, our contributions are: (1) one first minor contribution 90 is the addition of a Gated Recurrent Unit (GRU) recurrent 91 layer to the C3D architecture for action recognition which 92 also models the temporal correlations between actions, (2) 93 the comparison of both approaches (ANBM and C3D-GRU) 94 on two public datasets CAD-120 and Watch-n-Patch, (3) 95 implementation and evaluation of a late fusion mechanism of 96 the predictions of these two approaches and comparison with 97 the literature. We observe a performance gain from this hybrid 98 approach. 99The article is organized as follows. In section 2 we present 100 the state of the art and the context of our work. Then in 101 section 3 we present our hybrid approach for action detection. 102A comparative study of our results is presented in secti...

show abstract

Section: Proposed Approachmentioning

confidence: 99%

Section: A Bayesian Approach With Human-object Observationsmentioning

confidence: 99%

Section: A Bayesian Approach With Human-object Observationsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Late Fusion of Bayesian and Convolutional Models for Action Recognition

Maurice

Madrigal

Lerasle

2021

2020 25th International Conference on Pattern Recognition (ICPR)

Self Cite

View full text Add to dashboard Cite

show abstract

Action Recognition with Fusion of Multiple Graph Convolutional Networks

Maurice

Lerasle

2021

2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

Self Cite

View full text Add to dashboard Cite

We propose two light-weight and specialized Spatio-Temporal Graph Convolutional Networks (ST-GCNs): one for actions characterized by the motion of the human body and a novel one we especially design to recognize particular objects configurations during human actions execution. We propose a late-fusion strategy of the predictions of both graphs networks to get the most out of the two and to clear out ambiguities in the action classification. This modular approach enables us to reduce memory cost and training times. Moreover we also propose the same late fusion mechanism to further improve the performance using a Bayesian approach.We show results on 2 public datasets: CAD-120 and Watch-n-Patch. Our late-fusion mechanism yields performance gains in accuracy of respectively +21 percentage points (pp), +7 pp on Watch-n-Patch and CAD-120 compared to the individual graphs. Our approach outperforms most of the significant existing approaches.

show abstract

Multimodal spatial-temporal feature representation and its application in action recognition

Shi,

Hou,

Chao

et al. 2023

Journal of Image and Graphics

View full text Add to dashboard Cite

： Objective Human body motion-related recognition has been developing in the context of computer vision and pattern recognition like auxiliary human-computer interaction， motion analysis， intelligent monitoring， and virtual reality.To obtain two-dimensional information for its behavioral recognition， conventional motion behavior recognition is mainly used the RGB image sequence captured by RGB camera. To improve the ability to detect short-duration fragments， current feature descriptors for RGB image sequences are employed to characterize human behavior， such as histogram of oriented gradient (HOG)， histogram of optical flow (HOF)， and a three-dimensional feature pyramid. Some researchers are focused on the feature that image depth is insensitive to ambient light since RGB images are oriented to behavior image sequences of objects in terms of two-dimensional information. The depth information of the image is coordinated with the 中图法分类号： TP391文献标识码： A

show abstract

A New Bayesian Modeling for 3D Human-Object Action Recognition

Cited by 3 publications

References 25 publications

Late Fusion of Bayesian and Convolutional Models for Action Recognition

Late Fusion of Bayesian and Convolutional Models for Action Recognition

Action Recognition with Fusion of Multiple Graph Convolutional Networks

Multimodal spatial-temporal feature representation and its application in action recognition

Contact Info

Product

Resources

About