Semantic action recognition by learning a pose lexicon

Zhou, Lijuan; Li, Wanqing; Ogunbona, Philip; Zhang, Zhengyou

doi:10.1016/j.patcog.2017.06.035

Cited by 18 publications

(26 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, textual descriptions of human actions or visual features extracted from still images are studied as a semantic space [13]. In [24], [25], semantically meaningful poses extracted from associated textual instructions of actions are also defined as a semantic space. Chen and Huang [26] use the textural description, called Elaborative Description (ED), of actions and embed them to define a semantic space.…”

Section: A Zero-shot Action Recognitionmentioning

confidence: 99%

Learning Using Privileged Information for Zero-Shot Action Recognition

Gao¹,

Hou²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Section: A Zero-shot Action Recognitionmentioning

confidence: 99%

Learning Using Privileged Information for Zero-Shot Action Recognition

Gao¹,

Hou²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

“…In the paper in Ref. [3], visual poses are modelled by a Gaussian mixture model, which is learnt from training samples to characterise the likelihood of an observed visual frame being generated by a visual pose. A pose lexicon model is subsequently learnt through an extended hidden Markov alignment model to encode the relationship between semantic pose sequences and hidden visual pose sequences.…”

Section: Semantic Action Recognitionmentioning

confidence: 99%

“…As suggested in Ref. [3], an action can be described by multiple sequences of semantic poses extracted from textual instructions. Here, semantic poses refer to the configurations of body parts defined by linguistic description.…”

Section: Introductionmentioning

confidence: 99%

“…Third, this paper proposes to learn multiple pose lexicons jointly and one pose lexicon corresponds to the actions of one body part. The temporal relationship of semantic pose of body parts includes not only sequential but also simultaneous while only sequential relationship is included in the whole body-based pose lexicon [3,5]. Moreover, a transition model is proposed for capturing simultaneous actions of different body parts, which is not required in the paper [3,5].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning body part‐based pose lexicons for semantic action recognition

Zhou

Jiang

2022

IET Computer Vision

Self Cite

View full text Add to dashboard Cite

Semantic action recognition aims to classify actions based on the associated semantics, which can be applied in video captioning and human‐machine interaction. In this paper the problem is addressed by jointly learning multiple pose lexicons based on multiple body parts. Specifically, multiple visual pose models are learnt, and one visual pose model is associated with one body part, which characterises the likelihood of an observed video frame being generated from hidden visual poses. Moreover, multiple pose lexicon models are simultaneously learnt along with visual pose models. One pose lexicon model is associated with one body part that establishes a probabilistic mapping between the hidden visual poses and semantic poses parsed from textual instructions. To capture the temporal relations among body parts, a transition model is also learnt to measure the probability of the alignment transitioned from one position to another position. The body part‐based pose lexicon learning provides a novel method of cross‐modality semantic correlation, which can be applied in other spatial and temporal data. Action classification is finally formulated as the problem of finding the maximum posterior probability that a given multiple sequences of visual frames follow multiple sequences of semantic poses, subject to the most likely visual pose sequences and alignment sequences. Experiments were conducted on five action datasets to validate the effectiveness of the proposed method.

show abstract

“…Each body part, and the full ROI are encoded as streams in a multistream CNN. The main challenge is to estimate body poses [15] . It is suggested that the current robustness of algorithms is insufficient for action recognition [16] .…”

Section: Introductionmentioning

confidence: 99%