Challenges in Multi-modal Gesture Recognition

Escalera, Sérgio; Athitsos, Vassilis; Guyon, Isabelle

doi:10.1007/978-3-319-57021-1_1

Cited by 53 publications

(43 citation statements)

References 156 publications

(168 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, this line of research has not crossed over to fields such as gesture recognition. The state of the art in gesture recognition heavily relies on data mining and visual characteristics [7], [8], yet the cognitive processes related with gesture production and perception have not been considered as a prominent source of features for gesture recognition.…”

Section: Introductionmentioning

confidence: 99%

What Makes a Gesture a Gesture? Neural Signatures Involved in Gesture Recognition

Cabrera

Novak

Foti

et al. 2017

2017 12th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2017)

View full text Add to dashboard Cite

Previous work in the area of gesture production, has made the assumption that machines can replicate "humanlike" gestures by connecting a bounded set of salient points in the motion trajectory. Those inflection points were hypothesized to also display cognitive saliency. The purpose of this paper is to validate that claim using electroencephalography (EEG). That is, this paper attempts to find neural signatures of gestures (also referred as placeholders) in human cognition, which facilitate the understanding, learning and repetition of gestures. Further, it is discussed whether there is a direct mapping between the placeholders and kinematic salient points in the gesture trajectories. These are expressed as relationships between inflection points in the gestures' trajectories with oscillatory mu rhythms (8-12 Hz) in the EEG. This is achieved by correlating fluctuations in mu power during gesture observation with salient motion points found for each gesture. Peaks in the EEG signal at central electrodes (motor cortex; C3/Cz/C4) and occipital electrodes (visual cortex; O3/Oz/O4) were used to isolate the salient events within each gesture. We found that a linear model predicting mu peaks from motion inflections fits the data well. Increases in EEG power were detected 380 and 500ms after inflection points at occipital and central electrodes, respectively. These results suggest that coordinated activity in visual and motor cortices is sensitive to motion trajectories during gesture observation, and it is consistent with the proposal that inflection points operate as placeholders in gesture recognition.

show abstract

Section: Introductionmentioning

confidence: 99%

What Makes a Gesture a Gesture? Neural Signatures Involved in Gesture Recognition

Cabrera

Novak

Foti

et al. 2017

2017 12th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2017)

View full text Add to dashboard Cite

show abstract

“…Although the number of observations may be of the order of 10 (Yamato et al, 1992;Hertz et al, 2006;Wasikowski and Chen, 2010), it is more common for hundreds of observations to be made (Rigoll et al, 1997;Liang and Ouhyoung, 1998;Wei et al, 2011;Jost et al, 2015;Mapari and Kharat, 2015) and sometimes even thousands (Babu, 2016;Sun et al, 2015;Zheng et al, 2015;Zhou et al, 2015). The number depends strongly on the application, which may vary from object or face recognition in images or clips (Serre et al, 2005;Huang et al, 2007;Toshev et al, 2009) to gestures or patterns coming from complex multimodal inputs (Jaimes and Sebe, 2007;Escalera et al, 2016). Some of the major challenges regarding recognition lie in representation, learning, and detection (Lee et al, 2016).…”

Section: N-shot Learningmentioning

confidence: 99%

A Human-Centered Approach to One-Shot Gesture Learning

Cabrera

Wachs

2017

Front. Robot. AI

View full text Add to dashboard Cite

This article discusses the problem of one-shot gesture recognition using a humancentered approach and its potential application to fields such as human-robot interaction where the user's intentions are indicated through spontaneous gesturing (one shot). Casual users have limited time to learn the gestures interface, which makes one-shot recognition an attractive alternative to interface customization. In the aim of natural interaction with machines, a framework must be developed to include the ability of humans to understand gestures from a single observation. Previous approaches to oneshot gesture recognition have relied heavily on statistical and data-mining-based solutions and have ignored the mechanisms that are used by humans to perceive and execute gestures and that can provide valuable context information. This omission has led to suboptimal solutions. The focus of this study is on the process that leads to the realization of a gesture, rather than on the gesture itself. In this case, context involves the way in which humans produce gestures-the kinematic and anthropometric characteristics. In the method presented here, the strategy is to generate a data set of realistic samples based on features extracted from a single gesture sample. These features, called the "gist of a gesture," are considered to represent what humans remember when seeing a gesture and, later, the cognitive process involved when trying to replicate it. By adding meaningful variability to these features, a large training data set is created while preserving the fundamental structure of the original gesture. The availability of a large data set of realistic samples allows the use of training classifiers for future recognition. The performance of the method is evaluated using different lexicons, and its efficiency is compared with that of traditional N-shot learning approaches. The strength of the approach is further illustrated through human and machine recognition of gestures performed by a dual-arm robotic platform.

show abstract

“…In this work, we cover all the recent advancements in automatic emotion recognition from body gestures. The reader interested in emotion recognition from facial expressions or speech is encouraged to consult dedicated surveys [12], [13], [14]. In this work we refer to these only marginally and only as complements to emotional body gestures.…”

Section: Introductionmentioning

confidence: 99%

Survey on Emotional Body Gesture Recognition

Noroozi

Corneanu

Kamińska

et al. 2021

IEEE Trans. Affective Comput.

Self Cite

308

130

View full text Add to dashboard Cite

Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound recognizing affect from body gestures remains a less explored topic. We present a new comprehensive survey hoping to boost research in the field. We first introduce emotional body gestures as a component of what is commonly known as "body language" and comment general aspects as gender differences and culture dependence. We then define a complete framework for automatic emotional body gesture recognition. We introduce person detection and comment static and dynamic body pose estimation methods both in RGB and 3D. We then comment the recent literature related to representation learning and emotion recognition from images of emotionally expressive gestures. We also discuss multi-modal approaches that combine speech or face with body gestures for improved emotion recognition. While pre-processing methodologies (e.g. human detection and pose estimation) are nowadays mature technologies fully developed for robust large scale analysis, we show that for emotion recognition the quantity of labelled data is scarce, there is no agreement on clearly defined output spaces and the representations are shallow and largely based on naive geometrical representations.

show abstract

Challenges in Multi-modal Gesture Recognition

Cited by 53 publications

References 156 publications

What Makes a Gesture a Gesture? Neural Signatures Involved in Gesture Recognition

What Makes a Gesture a Gesture? Neural Signatures Involved in Gesture Recognition

A Human-Centered Approach to One-Shot Gesture Learning

Survey on Emotional Body Gesture Recognition

Contact Info

Product

Resources

About