Two-handed gesture recognition and fusion with speech to command a robot

Burger, Brice; Ferrané, Isabelle; Lerasle, Frédéric; Infantes, Guillaume

doi:10.1007/s10514-011-9263-y

Cited by 79 publications

(55 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, touchscreens could be smeared with residues, and remote controls could easily be damaged or misplaced. Examples of perceptual gesture-based interaction in a context of use resembling ours can be found in human-robot interaction [5][6][7]. However, to our knowledge, our work is one of the first user-centered studies of designing gesture-based control to operate factory automation systems.…”

Section: Gesture-based Interactionmentioning

confidence: 93%

Designing Gesture-Based Control for Factory Automation

Heimonen¹,

Hakulinen²,

Turunen³

et al. 2013

Human-Computer Interaction – INTERACT 2013

View full text Add to dashboard Cite

Abstract. We report the development and evaluation of a gesture-based interaction prototype for controlling the loading station of a factory automation system. In this context, gesture-based interaction has the potential to free users from the tedious physical controls but it must also account for safety considerations and users' perceptions. We evaluated the gesture interaction concept in the field to understand its applicability to industrial settings. Our findings suggest that gesture-based interaction is an emotional, physically charged experience that has the potential to enhance the work process. Participants' feedback also highlighted challenges related to the reliability of gesture recognition technology in the workplace, the perceived professionalism of gesture-based interaction, and the role of physical feedback in promoting feeling of control. Our results inform the development of gesture-based interaction for similar contexts.Keywords: Gesture-based interaction, emotions, user experience, field study. IntroductionGesture-based interaction could provide many benefits as an alternative to physical controls in industrial settings, where the manual operation of machinery can be ergonomically challenging and hazardous or is governed by safety regulations. Several characteristics of gesture-based systems affect their usability in such settings, including technical aspects such as latency of operation, stability, resolution, and precision [1], as well as human-centered aspects such as intuitiveness, ergonomics, and ease of gesture performance and recall [2]. Appropriately designed systems must balance these sometimes contradictory concerns. We report the findings from a year-long design and development process of a gesture-based interaction concept, focusing on the results of evaluations conducted in real context of use. The overall goal of the research was to create novel interaction concepts for metal workshops and to study their potential in real work tasks. Our research was aimed at answering the following questions:

show abstract

Section: Gesture-based Interactionmentioning

confidence: 93%

Designing Gesture-Based Control for Factory Automation

Heimonen¹,

Hakulinen²,

Turunen³

et al. 2013

Human-Computer Interaction – INTERACT 2013

View full text Add to dashboard Cite

show abstract

“…The same approach does not directly apply to reconstruct the absolute orientation of the rigid body, because ∆r t + ∆r t+1 = r t+1 . Having chosen H r,t [4,4] = 0 in (32), we can extract the relative rotation vector…”

Section: Pose Reconstructionmentioning

confidence: 99%

“…To ensure a smooth and effective cooperation, there is a need of representing human and robot actions in a compact and robust way that facilitates the understanding of performed motions as well as the generation of motions which flexibly adapt to different scenarios. Examples include recognizing human actions [44], gesture-based human-robot interaction [38,4,18], learning human activities [14], tasks learning by demonstration [1,17,33]. Aforementioned applications usually adopt Cartesian trajectories as task descriptors.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bidirectional invariant representation of rigid body motions and its application to gesture recognition and reproduction

2017

View full text Add to dashboard Cite

In this paper we propose a new bidirectional invariant motion descriptor of a rigid body. The proposed invariant representation is not affected by rotations, translations, time, linear and angular scaling. Invariant properties of the proposed representation enable to recognize gestures in realistic scenarios with unexpected variations (e.g., changes in user's initial pose, execution time or an observation point), while Cartesian trajectories are sensitive to these changes. The proposed invariant representation also allows reconstruction of the original motion trajectory, which is useful for human-robot interaction applications where a robot recognizes human actions and executes robot's proper behaviors using same descriptors. By removing the dependency on absolute pose and scaling factors of the Cartesian trajectories the proposed descriptor achieves flexibility to generate different motion instances from the same invariant representation. In order to illustrate the effectiveness of our proposed descriptor in motion recognition and generation, it is tested on three datasets and experiments on a NAO humanoid robot and a KUKA LWR IV+ manipulator and compared with other existing invariant representations.

show abstract

“…However, because it is still a difficult problem to extract these two small positions, there are many alternative methods for estimating the pointing direction; head-hand line [2,6,8], head-finger line [4,6,13], forearm direction [1,6,8], and head orientation [8,11] methods. Regarding the head orientation approach, this method does not use the hand position so we need to obtain more information to identify the target object, for example, speech recognition to obtain the object features [11].…”

Section: Related Workmentioning

confidence: 99%

An efficient method for human pointing estimation for robot interaction

Ueno

Naito

Chen

2014

2014 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

In this paper, we propose an efficient calibration method to estimate the pointing direction via a human pointing gesture to facilitate robot interaction. The ways in which pointing gestures are used by humans to indicate an object are individually diverse. In addition, people do not always point at the object carefully, which means there is a divergence between the line from the eye to the tip of the index finger and the line of sight. Hence, we focus on adapting to these individual ways of pointing to improve the accuracy of target object identification by means of an effective calibration process. We model these individual ways as two offsets, the horizontal offset and the vertical offset. After locating the head and fingertip positions, we learn these offsets for each individual through a training process with the person pointing at the camera. Experimental results show that our proposed method outperforms other conventional head-hand, head-fingertip, and eye-fingertipbased pointing recognition methods.

show abstract

Two-handed gesture recognition and fusion with speech to command a robot

Cited by 79 publications

References 48 publications

Designing Gesture-Based Control for Factory Automation

Designing Gesture-Based Control for Factory Automation

Bidirectional invariant representation of rigid body motions and its application to gesture recognition and reproduction

An efficient method for human pointing estimation for robot interaction

Contact Info

Product

Resources

About