Human-Computer Interaction through Time-of-Flight and RGB Cameras

Technological advances and scalability are leading Human-Computer Interaction (HCI) to evolve towards intuitive forms, such as through gesture recognition. Among the various interaction strategies, radar-based recognition is emerging as a touchless, privacy-secure, and versatile solution in different environmental conditions. Classical radar-based gesture HCI solutions involve deep learning but require training on large and varied datasets to achieve robust prediction. Innovative self-learning algorithms can help tackling this problem by recognizing patterns and adapt from similar contexts. Yet, such approaches are often computationally expensive and hardly integrable into hardware-constrained solutions. In this paper, we present a gesture recognition algorithm which is easily adaptable to new users and contexts. We exploit an optimization-based meta-learning approach to enable gesture recognition in learning sequences. This method targets at learning the best possible initialization of the model parameters, simplifying training on new contexts when small amounts of data are available. The reduction in computational cost is achieved by processing the radar sensed data of gestures in the form of time maps, to minimize the input data size. This approach enables the adaptation of simple convolutional neural network (CNN) to new hand poses, thus easing the integration of the model into a hardware-constrained platform. Moreover, the use of a Variational Autoencoders (VAE) to reduce the gestures' dimensionality leads to a model size decrease of an order of magnitude and to half of the required adaptation time. The proposed framework, deployed on the Intel ® Neural Compute Stick 2 (NCS 2), leads to an average accuracy of around 84% for unseen gestures when only one example per class is utilized at training time. The accuracy increases up to 92.6% and 94.2% when three and five samples per class are used.

show abstract

“…To further increase the SNR for the RDI computation, the absolute value of the average of S IF (n) over the N RX , as shown in (6).…”

Section: ) Single Frame Preprocessingmentioning

confidence: 99%

“…Many advances in the field focus on vision-based inter-facing, i.e. the use of camera sensors such as Red Green Blue (RGB) and Time of Flight (ToF) [6]- [9]. In Fact, Camera sensors bring the advantage of touchless communication.…”

Section: Introductionmentioning

confidence: 99%

Few-Shot User-Definable Radar-Based Hand Gesture Recognition at the Edge

et al. 2022

View full text Add to dashboard Cite

show abstract

“…As can be seen, the system shows a good learning curve: no user made more than one mistake per test, even when writing an entire sentence. Tables 2-4 summarise, for each test, the performance comparison between the previous [4] and the current interaction mode. The obtained outcomes show that the new solution provides a remarkable reduction of errors without affecting time performances, which, indeed, slightly improve compared with the previous version.…”

Section: Fig 8 Possible Movements Of a 3d Object (Top Line) And Corrmentioning

confidence: 99%

“…The maximum number of mistakes is the same for the two cases and their occurrence rates are very close (8 and 5%). On the contrary, in [4] the keyboard was less precise than the numeric pad (53 against 26%).…”

Section: User Interaction Testsmentioning

confidence: 99%

“…Two different use cases have been developed and tested, in order to check the flexibility of the system: a virtual keyboard and a 3D object manipulator. A preliminary analysis of the method and these applications has been presented in two previous works [4, 5]. This paper introduces several refinements and new features, such as an increased segmentation robustness, thanks to the use of specific noise reduction techniques, a higher writing precision with the keyboard and the recognition of new gestures in the 3D object application ( Z ‐axis rotation).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Development of gesture‐based human–computer interaction applications by fusion of depth and colour video streams

Dondi

Lombardi

Porta

2014

IET Computer Vision

Self Cite

View full text Add to dashboard Cite

Hand detection and gesture recognition are two of the most studied topics in human-computer interaction (HCI). The increasing availability of sensors able to provide real-time depth measurements, such as time-of-flight cameras or the more recent Kinect, has helped researchers to find more and more efficient solutions for these issues. With the main aim to implement effective gesture-based interaction systems, this study presents an approach to hand detection and tracking that exploits two different video streams: the depth one and the colour one. Both hand and gesture recognition are based only on geometrical and colour constraints, and no learning phase is needed. The use of a Kalman filter to track hands guarantees system robustness also in presence of many persons in the scene. The entire procedure is designed to maintain a low computational cost and is optimised to efficiently execute HCI tasks. As use cases two common applications are described: a virtual keyboard and a three-dimensional object manipulation virtual environment. These applications have been tested with a representative sample of non-trained users to assess the usability and flexibility of the system.

show abstract

Human-Computer Interaction through Time-of-Flight and RGB Cameras

Cited by 2 publications

References 8 publications

Few-Shot User-Definable Radar-Based Hand Gesture Recognition at the Edge

Few-Shot User-Definable Radar-Based Hand Gesture Recognition at the Edge

Development of gesture‐based human–computer interaction applications by fusion of depth and colour video streams

Contact Info

Product

Resources

About