The Leap Motion Controller in Clinical Music Therapy - A Computer-based Approach to Intellectual and Motor Disabilities

Baratè, Adriano; Elia, Antonio; Ludovico, Luca A.; Oriolo, Eleonora

doi:10.5220/0006771204610469

Cited by 5 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The controller is often used in virtual reality (VR) and augmented reality (AR) applications, but it can also be used with other types of software, especially with disabled people. Although different studies and projects have been developed to use hand-tracking technology with disabled people, such as a hand rehabilitation system to assist in developing muscle tone and increase precision in gestures [33,34] for sign languages recognition [35], and in music therapy sessions [36], it is not a common or a widely used technology. Some limitations of the device include:…”

Section: Leap Motion Emg and Eegmentioning

confidence: 99%

Multimodal Interaction for Cobot Using MQTT

Rouillard

Vannobel

2023

MTI

View full text Add to dashboard Cite

For greater efficiency, human–machine and human–robot interactions must be designed with the idea of multimodality in mind. To allow the use of several interaction modalities, such as the use of voice, touch, gaze tracking, on several different devices (computer, smartphone, tablets, etc.) and to integrate possible connected objects, it is necessary to have an effective and secure means of communication between the different parts of the system. This is even more important with the use of a collaborative robot (cobot) sharing the same space and very close to the human during their tasks. This study present research work in the field of multimodal interaction for a cobot using the MQTT protocol, in virtual (Webots) and real worlds (ESP microcontrollers, Arduino, IOT2040). We show how MQTT can be used efficiently, with a common publish/subscribe mechanism for several entities of the system, in order to interact with connected objects (like LEDs and conveyor belts), robotic arms (like the Ned Niryo), or mobile robots. We compare the use of MQTT with that of the Firebase Realtime Database used in several of our previous research works. We show how a “pick–wait–choose–and place” task can be carried out jointly by a cobot and a human, and what this implies in terms of communication and ergonomic rules, via health or industrial concerns (people with disabilities, and teleoperation).

show abstract

Section: Leap Motion Emg and Eegmentioning

confidence: 99%

Multimodal Interaction for Cobot Using MQTT

Rouillard

Vannobel

2023

MTI

View full text Add to dashboard Cite

show abstract

“…Although different studies and projects have been developed to use hand tracking technology with disabled people, such as a hand rehabilitation system to assist in developing muscle tonus and increase precision in gestures (Alimanova et al 2017, Cortés-Pérez et al 2021, for sign languages recognition (Galván-Ruiz et al 2020), in music therapy sessions (Baratè et al 2018), it is not a common or a widely used technology.…”

Section: Case Studies A) Pick and Place (Unity 3d Application)mentioning

confidence: 99%

BIOFEE: Biomedical Framework for Enhanced Experimentation

Rouillard¹,

Vannobel²,

Bekaert³

2023

Health Informatics and Biomedical Engineering Applications

View full text Add to dashboard Cite

Biomedical Framework for Enhanced Experimentation, based on Firebase Real-Time Database, helps in designing and testing various multimodal solutions for patients suffering from progressive diseases. It allows interacting with objects and robots to test various multimodal solutions, such as touch, voice, gesture... to perform tasks according to the patient's pathology. We tested BIOFEE with Unity3D and Webots virtual worlds to compare different ways to interact for particular tasks (switch lamps, pick and place...).

show abstract

“…Colombini et al [27] review 19 studies objectively assessing the contribution of LMC in four specific psychological domains (autism spectrum disorder, attentiondeficit or hyperactivity disorder, dementia, mild cognitive impairment). The LMC is also successfully used in music therapy sessions [28]. The proposed system captures the user's gesture via the LMC and the generated signals are sent to a software tool that converts the movements into musical notes.…”

Section: The Lmc As An Assistive Technologymentioning

confidence: 99%

Towards a GUI Gesture Control Using the Leap Motion Controller

Vannobel¹,

Bekaert²,

Baumann³

2022

MMC_C

View full text Add to dashboard Cite

The Leap Motion Controller is a small hand motion capture device that has been the subject of numerous studies since it was launched in 2013. This sensor is particularly used in the field of disability, diagnosis or functional rehabilitation. This article presents two preliminary studies on the use of this device as a gesture mouse and then as a gesture controller. If the Leap Motion Controller is simple to implement, its lack of precision in some gesture configurations and of reliability (outliers) is a barrier which could certainly be lifted in the context of a multimodal interaction.

show abstract

The Leap Motion Controller in Clinical Music Therapy - A Computer-based Approach to Intellectual and Motor Disabilities

Cited by 5 publications

References 0 publications

Multimodal Interaction for Cobot Using MQTT

Multimodal Interaction for Cobot Using MQTT

BIOFEE: Biomedical Framework for Enhanced Experimentation

Towards a GUI Gesture Control Using the Leap Motion Controller

Contact Info

Product

Resources

About