A real-time applicable 3D gesture recognition system for automobile HMI

Abstract

Abstract-We present a system for 3D hand gesture recognition based on low-cost time-of-flight(ToF) sensors intended for outdoor use in automotive human-machine interaction. As signal quality is impaired compared to Kinect-type sensors, we study several ways to improve performance when a large number of gesture classes is involved. Our system fuses data coming from two ToF sensors which is used to build up a large database and subsequently train a multilayer perceptron (MLP). We demonstrate that we are able to reliably classify a set of ten hand gestures in real-time and describe the setup of the system, the utilised methods as well as possible application scenarios. I. INTRODUCTIONAs "intelligent" devices enter more and more areas of everyday life, the issue of man-machine interaction becomes ever more important. As interaction should be easy and natural for the user and also not require a high cognitive load, non-verbal means of interaction such as hand gestures will play a decisive role in this field of research. With the advent of low-cost Kinect-type 3D sensors, and more recently of low-cost ToF sensors (400-500e) that can be applied in outdoor scenarios, the use of point clouds seems a very logical choice. This presents challenges to machine learning approaches as the data dimensionality and sensor noise are high, as well as the number of interesting gesture categories. In this article we build upon earlier results [1] and demonstrate how a system can be developed and integrated into a car in order to be able to classify a gesture alphabet of ten hand poses. Our approach is purely data-driven, i.e. by extending and applying a Pointcloud descriptor to our needs we are able to set up a real-time applicable system which is robust versus daylight interferences, invariant to rotation and translation problems and moreover works without the need to formalise a possibly complicated hand model. We will first discuss the related work relevant for our research (Sec. II) and then go on to describe the setup of our system within an automobile environment. Subsequently we describe the sensors and the used database in Sec. IV. In Sec. V we go on to give an account of the used different holistic point cloud descriptors and explain the meaning of the parameter variations we will test. Sec. VI summarises the implemented NN classes and the choice of parameters. The key questions we will investigate in Sec. VII concern the generalisation

A simple technique for improving multi-class classification with neural networks

“…In these cases however one can simply rely on the first net when designing a decision function and use the second net for the other instances. In terms of applicability we have implemented this fusion technique into our realtime hand gesture recognition system and it has shown to stabilize the results when designing the decision process as described above [8]. We are confident to be able to improve our performance even further by employing confidence measures for rejecting false positives and thus stabilizing the recognition performance.…”

Hand Gesture Recognition in Automotive Human–Machine Interaction Using Depth Cameras

“…In this review, we examined current state-of-the-art deep learning technologies for hand gesture recogniton and consolidated a line of research from the Computational Neuroscience laboratory at the Ruhr West University of Applied Sciences. Kopinski’s contributions [34,36,37,41,42,43,44,62,66,73,74,75,76,77,78,79] and PhD thesis [72] form the basis of our hand gesture recognition research. We investigated deep learning technologies for the purpose of hand gesture recognition in automotive context with three-dimensional data from time-of-flight infrared sensors in order to provide new means of controls for driver assistance systems.…”

“…Ref. [62] evaluated the impact of varying confidence measures and thresholds. The dataset in this experiment features a total of 400,000 samples.…”

A light-weight real-time applicable hand gesture recognition system for automotive applications

“…Building upon prior results [1], [2], we extend our database to have more variance in the data (by adding more persons), and present a novel temporal fusion technique which *This work was not supported by any organization 1 boosts recognition rates by taking into account preceding recognitions as well. Moreover, our temporal fusion of data lets us take an initial step towards defining dynamic gestures via static hand poses by taking into consideration several snapshots taken over time.…”