On-device Real-time Hand Gesture Recognition

Sung, George; Sokal, Kanstantsin; Uboweja, Esha; Bazarevsky, Valentin; Baccash, Jonathan; Bazavan, Eduard Gabriel; Chang, Chuo-Ling; Grundmann, Matthias

doi:10.48550/arxiv.2111.00038

Cited by 7 publications

(7 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although we could have implemented TherAImin without antennas using, e.g., Mediapipe Handpose [29], we have decided to be more faithful to the original instrument and thus use the [30], which provides a versatile Theremin implementation with Pitch and Volume outputs.…”

Section: Theraiminmentioning

confidence: 99%

FAIME: A Framework for AI-Assisted Musical Devices

Civit¹,

Muñoz-Saavedra²,

Cuadrado³

et al. 2023

Internet of Things - New Trends, Challenges and Hurdles

View full text Add to dashboard Cite

In this paper, we present a novel framework for the study and design of AI-assisted musical devices (AIMEs). Initially, we present taxonomy of these devices and illustrate it with a set of scenarios and personas. Later, we propose a generic architecture for the implementation of AIMEs and present some examples from the scenarios. We show that the proposed framework and architecture are a valid tool for the study of intelligent musical devices.

show abstract

Section: Theraiminmentioning

confidence: 99%

FAIME: A Framework for AI-Assisted Musical Devices

Civit¹,

Muñoz-Saavedra²,

Cuadrado³

et al. 2023

Internet of Things - New Trends, Challenges and Hurdles

View full text Add to dashboard Cite

show abstract

“…24 This convolutional neural network includes palm and landmark detection functions, with a reported mean average precision of 95.7% for palm and 83.8% for landmark detection and an observed mean localization error of 1.3 cm in 3D space. 25 Twenty-one landmarks were detected on both hands simultaneously and automatically, without requiring any special registration, allowing calculation of spatial movement during the procedure. Computational analysis was performed on a laptop computer with Apple M1 Pro processor with a 10-core CPU, 32-core GPU, and 32 GB RAM (Apple Inc.).…”

Section: Development Of a Hand-tracking Detector Using Machine Learningmentioning

confidence: 99%

“…38 The machine learning model used in our study has good generalization capability to allow tracking of individuals irrespective of skin color, race, and sex (our study participants were Asian, Hispanic, and White males). 25 Other research groups have used this technology successfully in the field of American Sign Language recognition, reporting an accuracy of 87.6% to 98.4% in multiple sign language data sets, 18 and in the assessment of tremor in patients with Parkinson disease, reporting an observed mean (SD) absolute error of tremor measurements of 0.229 (0.174) Hz compared with an accelerometer. 40…”

Section: Hand Motion Detection Using Convolutional Neural Networkmentioning

confidence: 99%

Quantification of motion during microvascular anastomosis simulation using machine learning hand detection

González-Romo

Hanalıoğlu

Mignucci-Jiménez

et al. 2023

Neurosurgical Focus

View full text Add to dashboard Cite

OBJECTIVE Microanastomosis is one of the most technically demanding and important microsurgical skills for a neurosurgeon. A hand motion detector based on machine learning tracking technology was developed and implemented for performance assessment during microvascular anastomosis simulation. METHODS A microanastomosis motion detector was developed using a machine learning model capable of tracking 21 hand landmarks without physical sensors attached to a surgeon’s hands. Anastomosis procedures were simulated using synthetic vessels, and hand motion was recorded with a microscope and external camera. Time series analysis was performed to quantify the economy, amplitude, and flow of motion using data science algorithms. Six operators with various levels of technical expertise (2 experts, 2 intermediates, and 2 novices) were compared. RESULTS The detector recorded a mean (SD) of 27.6 (1.8) measurements per landmark per second with a 10% mean loss of tracking for both hands. During 600 seconds of simulation, the 4 nonexperts performed 26 bites in total, with a combined excess of motion of 14.3 (15.5) seconds per bite, whereas the 2 experts performed 33 bites (18 and 15 bites) with a mean (SD) combined excess of motion of 2.8 (2.3) seconds per bite for the dominant hand. In 180 seconds, the experts performed 13 bites, with mean (SD) latencies of 22.2 (4.4) and 23.4 (10.1) seconds, whereas the 2 intermediate operators performed a total of 9 bites with mean (SD) latencies of 31.5 (7.1) and 34.4 (22.1) seconds per bite. CONCLUSIONS A hand motion detector based on machine learning technology allows the identification of gross and fine movements performed during microanastomosis. Economy, amplitude, and flow of motion were measured using time series data analysis. Technical expertise could be inferred from such quantitative performance analysis.

show abstract

“…El reconocimiento de gestos con la mano (HGR) es una forma natural e intuitiva de interacción persona-ordenador (HCI), que ha sido objeto de numerosos estudios; con una amplia gama de dispositivos de entrada y enfoques, el HGR basado en el esqueleto se convirtió en una opción popular debido a su resistencia a las fluctuaciones del fondo y la luz (Sung et al, 2021). Entre las herramientas utilizadas para el reconocimiento de gestos con la mano, MediaPipe dispone de una solución para ella; esta emplea el ML para inferir 21 puntos de referencia en 3D de una mano, a partir de un solo fotograma (MediaPipe, 2020), como se presenta en la Figura 1.…”

Section: Reconocimiento De Manosunclassified

Inteligencia artificial y visión por computadora aplicada a la educación

López¹

2022

View full text Add to dashboard Cite

El presente trabajo de investigación realiza una revisión bibliográfica sobre la inteligencia artificial, la visión por computadora, las herramientas de MediaPipe y aprendizaje automático, con el objetivo de generar una propuesta de tecnología que permita mejorar la calidad de las clases. A lo largo del trabajo se describe la situación actual que tienen los estudiantes respecto a las clases en modalidad virtual, se incluyen y explican tanto definiciones como el funcionamiento de los distintos temas, al igual que se incluyen aplicaciones de ellos en la vida cotidiana. De igual manera se incluye el aporte a la educación que brinda cada uno de los campos, y se realiza un análisis de trabajos similares. Finalmente, se realiza la propuesta de una herramienta tecnológica que puede mejorar la calidad de las clases en la modalidad virtual, siendo así, un aporte para la educación.

show abstract

On-device Real-time Hand Gesture Recognition

Cited by 7 publications

References 5 publications

FAIME: A Framework for AI-Assisted Musical Devices

FAIME: A Framework for AI-Assisted Musical Devices

Quantification of motion during microvascular anastomosis simulation using machine learning hand detection

Inteligencia artificial y visión por computadora aplicada a la educación

Contact Info

Product

Resources

About