Gesture Recognition by Leap Motion Controller and LSTM Networks for CAD-oriented Interfaces

Mazzini, Lisa; Franco, Annalisa; Maltoni, Davide

doi:10.1007/978-3-030-30642-7_17

Cited by 4 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, they investigate the use of a 3D-CNN on several data types, i.e., RGB, depth and infrared images. In [9] a Leap Motion controller is used to develop an interaction system based on a LSTM network; the system is oriented to the interaction for a CAD software. A Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU) [35] are exploited, and the final classification is performed with a fully-connected layer.…”

Section: Methodsmentioning

confidence: 99%

“…In the CADGestures dataset [9] gestures are encoded in an 18-dimensional vector exploiting the 3D joints of fingers, palm and arm, which enable the computation of significant angles and translation values. Using their dominant hand, 30 performed a gesture from a set of 8 classes twice, obtaining a total of 480 gestures.…”

Section: Datasetmentioning

confidence: 99%

“…Among different solutions for the acquisition of 3D data, the usage of low-cost active depth, RGB-D or stereo infrared sensors [1,[6][7][8][9], which are usually coupled with an infrared emitter and easily allow the acquisition of 2.5D data, has overcome other expensive and cumbersome 3D acquisition devices, such as Lidar and 3D scanners.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal Hand Gesture Classification for the Human–Car Interaction

et al. 2020

View full text Add to dashboard Cite

The recent spread of low-cost and high-quality RGB-D and infrared sensors has supported the development of Natural User Interfaces (NUIs) in which the interaction is carried without the use of physical devices such as keyboards and mouse. In this paper, we propose a NUI based on dynamic hand gestures, acquired with RGB, depth and infrared sensors. The system is developed for the challenging automotive context, aiming at reducing the driver’s distraction during the driving activity. Specifically, the proposed framework is based on a multimodal combination of Convolutional Neural Networks whose input is represented by depth and infrared images, achieving a good level of light invariance, a key element in vision-based in-car systems. We test our system on a recent multimodal dataset collected in a realistic automotive setting, placing the sensors in an innovative point of view, i.e., in the tunnel console looking upwards. The dataset consists of a great amount of labelled frames containing 12 dynamic gestures performed by multiple subjects, making it suitable for deep learning-based approaches. In addition, we test the system on a different well-known public dataset, created for the interaction between the driver and the car. Experimental results on both datasets reveal the efficacy and the real-time performance of the proposed method.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Hand Gesture Classification for the Human–Car Interaction

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Ganguly et al [34] have compared LMC to the standard motion capture techniques. Integration of CAD systems and LMC has been already successfully achieved [35].…”

Section: State Of the Artmentioning

confidence: 99%

Conceptualisation of Virtual Reality Experiments for Optimised Sinus Surgery Planning and Execution

Kunica¹,

Poje²,

Mlivić³

et al. 2023

Int J Ind Eng Manag

View full text Add to dashboard Cite

Contemporary research in the field of technology and production is focused on the intensive development of highly automated --so called intelligent and autonomous devices and systems, for smaller (discretized) production volumes (of products and business --production activities of their realization). This development implies a need to create a new knowledge and change --re-evaluate and upgrade of existing knowledge: project approaches and content related to product, process, and system design, as well as a need to include traditional non-engineering areas such as biology and medicine (neuroscience, anatomy, psychiatry). Regarding products, especially, it is interesting to consider the performance of processes in which deformable objects and sculpture surfaces appear in different industries. Consideration of advanced sensorics and virtual reality (VR) involves the analysis of manual processes [1], as yet the most widespread, and as a basis for better understanding and optimization (stabilization) as well as standardization ("normization") of manual processes [2], thus enabling development of mechatronic devices for automatic process execution.Technology development allows the design and standardization of manual work in human activities which are yet to be observed, especially those of low-volume discrete production, such as in ear, nose and throat (ENT) surgery, particularly endoscopic sinus surgery (ESS). The paper considers the possibilities of developing an approach that would be useful for specific ESS manual work analysis and design and its automation in a future. The consideration includes, beside other aspects, initial steps, such as the traditional work motion capture by camera and experiments by Leap Motion Controller, Perception Neuron 2.0 and Emotiv Epoc+.

show abstract

Architectural design game: A serious game approach to promote teaching and learning using multimodal interfaces

Goli¹,

Teymournia²,

Naemabadi³

et al. 2022

Educ Inf Technol

View full text Add to dashboard Cite

The present article introduces and develops an educational tool as an interactive digital game for architectural design, allowing the architectural students to challenge their knowledge and experiences. The framework of this educational tool supports a serious open-ended game, which helps students get involved with the game through self-assessment and a multi-modal natural user interface, including gesture recognition and speech recognition in a familiar CAD environment without any right or wrong solutions. The students can immediately compare their game results with the architecture of iconic buildings and get familiar with the complexity of the design process through five different games in the initial version of this tool without the fear of being judged. According to the results of the questionnaire, this tool can simulate the design process, enhance its quality, and thus, assist the learners with developing their required skills with a wide variety of motivations and opportunities for engagement while helping them connect their experiences and activities to their learning and development in a meaningful way to fill the gap between their knowledge acquisition and knowledge application.

show abstract

Gesture Recognition by Leap Motion Controller and LSTM Networks for CAD-oriented Interfaces

Cited by 4 publications

References 13 publications

Multimodal Hand Gesture Classification for the Human–Car Interaction

Multimodal Hand Gesture Classification for the Human–Car Interaction

Conceptualisation of Virtual Reality Experiments for Optimised Sinus Surgery Planning and Execution

Architectural design game: A serious game approach to promote teaching and learning using multimodal interfaces

Contact Info

Product

Resources

About