Real-time automated detection of older adults' hand gestures in home and clinical settings

Abstract

There is an urgent need, accelerated by the COVID-19 pandemic, for methods that allow clinicians and neuroscientists to remotely evaluate hand movements. This would help detect and monitor degenerative brain disorders that are particularly prevalent in older adults. With the wide accessibility of computer cameras, a vision-based real-time hand gesture detection method would facilitate online assessments in home and clinical settings. However, motion blur is one of the most challenging problems in the fast-moving hands data collection. The objective of this study was to develop a computer vision-based method that accurately detects older adults’ hand gestures using video data collected in real-life settings. We invited adults over 50 years old to complete validated hand movement tests (fast finger tapping and hand opening–closing) at home or in clinic. Data were collected without researcher supervision via a website programme using standard laptop and desktop cameras. We processed and labelled images, split the data into training, validation and testing, respectively, and then analysed how well different network structures detected hand gestures. We recruited 1,900 adults (age range 50–90 years) as part of the TAS Test project and developed UTAS7k—a new dataset of 7071 hand gesture images, split 4:1 into clear: motion-blurred images. Our new network, RGRNet, achieved 0.782 mean average precision (mAP) on clear images, outperforming the state-of-the-art network structure (YOLOV5-P6, mAP 0.776), and mAP 0.771 on blurred images. A new robust real-time automated network that detects static gestures from a single camera, RGRNet, and a new database comprising the largest range of individual hands, UTAS7k, both show strong potential for medical and research applications. Supplementary Information The online version contains supplementary material available at 10.1007/s00521-022-08090-8.

Brief webcam test of hand movements predicts episodic memory, executive function, and working memory in a community sample of cognitively asymptomatic older adults

“…The use of computer vision methods to extract finger‐tapping features to predict cognitive performance is novel 3 , 5 , 16 and efficient 12 but similar methods are increasingly being used to quantify various movement disorders as they are objective and granular, unlike clinical rating scales, 44 and provide evaluation of movements in 3D space through simple smartphones or laptops. 14 , 15 , 45 Computer vision approaches facilitate wide reach and accessibility, including to those in rural and remote regions, as webcams are so ubiquitous in phones and computers around the world.…”

“…Computer vision techniques can be applied to digital videos recorded through webcams in household computers or mobile phones to precisely measure movements 11–15 . In contrast to using keyboard‐tapping tasks, video‐based technologies enable analysis of hand movements in 3D space (so additional features such as amplitude and decrement can be extracted), and the non‐touch technique minimizes infection risks.…”

“…Computer vision techniques can be applied to digital videos recorded through webcams in household computers or mobile phones to precisely measure movements. 11 , 12 , 13 , 14 , 15 In contrast to using keyboard‐tapping tasks, video‐based technologies enable analysis of hand movements in 3D space (so additional features such as amplitude and decrement can be extracted), and the non‐touch technique minimizes infection risks. Computer vision analysis of finger‐tapping has not yet been employed via an online self‐test in home settings to aid prediction of cognitive performance in asymptomatic older adults.…”

Research on the System Design and Target Recognition Method of the Rebar-Tying Robot

“…In sequential order, the number of Residual modules in the network is increasing, the feature extraction and fusion ability is increasing, and the detection accuracy is improving; however, the corresponding time spent is also increasing. In addition, several studies have proposed improved algorithms with increased accuracy and speed based on the YOLOv5 architecture, including YOLOv5-P6 [25], LSK-YOLOv5 [26], and Ghost-YOLOv5 [27].…”

Development of a smartphone screening test for preclinical Alzheimer's disease and validation across the dementia continuum

“…We will use cutting-edge Arti cial Intelligence (AI) -based technologies, building on our previous research, to automatically analyse hand and speech-like movement from digital video and audio recordings respectively, and to 'learn' abnormal patterns of data that are associated with biomarker-de ned AD pathology (39)(40)(41)(42)(43)(44). These advanced computer science techniques provide a precise, automated and e cient method of analysis.…”