Multi-sensor system for driver's hand-gesture recognition

Abstract

We propose a novel multi-sensor system for accurate and power-efficient dynamic car-driver hand-gesture recognition, using a short-range radar, a color camera, and a depth camera, which together make the system robust against variable lighting conditions. We present a procedure to jointly calibrate the radar and depth sensors. We employ convolutional deep neural networks to fuse data from multiple sensors and to classify the gestures. Our algorithm accurately recognizes 10 different gestures acquired indoors and outdoors in a car during the day and at night. It consumes significantly less power than purely vision-based systems.

Short-range FMCW monopulse radar for hand-gesture sensing

“…Our system is similar to long-range radars currently employed in automobiles to sense external environments, but for our solution, which was designed to operate inside a car, we adapted these radar principles to the short-range (< 1m) scenario. The proposed radar system is part of a multi-sensor system for drivers' hand-gesture recognition [5]. In this paper we describe in detail the design and signal processing procedures of the radar system.…”

Sparsity-based dynamic hand gesture recognition using micro-Doppler signatures

“…According to the sparse signal processing theories [15], when K≪N<M, the sparse representation vector x in (1) (4) where |||| 0 and |||| 2 denote the L 0 and L 2 norms, respectively. According to previous literatures, the solution for (3) can be obtained by greedy algorithms such as the orthogonal matching pursuit algorithm (OMP) [15] or linear programming after replacing L 0 norm with L 1 norm in (3).…”

“…In [3], a portable radar sensor is employed to recognize dynamic hand gestures by using application-specific features and principal component analysis (PCA), and the results illustrate the potential of radar-based dynamic hand gesture recognition for smart home applications. The authors of [4,5] model human hand as a non-rigid object and use a frequency modulated continuous wave (FMCW) radar to obtain the range-Doppler images of dynamic hand gestures of drivers' gestures. As presented in [5], radar echoes of dynamic hand gestures contain multiple components with time-varying frequency modulations, which are referred to as micro-Doppler signatures in radar jargon [6][7].…”

“…In contrast, radar sensor is capable of detecting and classifying moving targets with high robustness to light conditions. Recently, radar-based approaches for dynamic hand gesture recognition have attracted much attention [2][3][4][5][6]. In [2], a Doppler radar system is developed for detecting three kinds of dynamic hand gestures.…”

Bonseyes

“…New technologies will provide for greater assistance in navigation, enhanced driver information about the vehicle, its environment [10] and vehicle connectivity. Consumers, with a plethora of electronic devices that inform them [5], entertain them and keep them safe [9], [6], will nd themselves enjoying the overall experience of their vehicles. Connectivity and lifestyle trends will change the way cars are used.…”