A multimodal framework for sensor based sign language recognition

Kumar, Pradeep; Gauba, Himaanshu; Roy, Partha Pratim; Dogra, Debi Prosad

doi:10.1016/j.neucom.2016.08.132

Cited by 169 publications

(81 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It can generate various hand and finger motion tracking information, such as the palm position, velocity, normal, direction, and grab strength, as well as the fingertip position, velocity, direction, length, width, and finger extended state. Due to its small observation area and high resolution, Leap Motion Controller has been widely used as a human-machine interface in the context of robot control [17], sign-language recognition [32], and text segmentation [33].…”

Section: Leap Motion Controllermentioning

confidence: 99%

A Bimanual Robotic Teleoperation Architecture with Anthropomorphic Hybrid Grippers for Unstructured Manipulation Tasks

Zhu

Xiao

et al. 2020

Applied Sciences

View full text Add to dashboard Cite

Bimanual telemanipulation is vital for facilitating robots to complete complex and dexterous tasks that involve two handheld objects under teleoperation scenarios. However, the bimanual configuration introduces higher complexity, dynamics, and uncertainty, especially in those uncontrolled and unstructured environments, which require more advanced system integration. This paper presents a bimanual robotic teleoperation architecture with modular anthropomorphic hybrid grippers for the purpose of improving the telemanipulation capability under unstructured environments. Generally, there are two teleoperated subsystems within this architecture. The first one is the Leap Motion Controller and the anthropomorphic hybrid robotic grippers. Two 3D printed anthropomorphic hybrid robotic grippers with modular joints and soft layer augmentations are designed, fabricated, and equipped for telemanipulation tasks. A Leap Motion Controller is used to track the motion of two human hands, while each hand is utilized to teleoperate one robotic gripper. The second one is the haptic devices and the robotic arms. Two haptic devices are adopted as the master devices while each of them takes responsibility for one arm control. Based on such a framework, an average RMSE (root-mean-square-error) value of 0.0204 rad is obtained in joint tracking. Nine sign-language demonstrations and twelve object grasping tasks were conducted with the robotic gripper teleoperation. A challenging bimanual manipulation task for an object with 5.2 kg was well addressed using the integrated teleoperation system. Experimental results show that the proposed bimanual teleoperation system can effectively handle typical manipulation tasks, with excellent adaptabilities for a wide range of shapes, sizes, and weights, as well as grasping modes.

show abstract

Section: Leap Motion Controllermentioning

confidence: 99%

A Bimanual Robotic Teleoperation Architecture with Anthropomorphic Hybrid Grippers for Unstructured Manipulation Tasks

Zhu

Xiao

et al. 2020

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Diversos estudos demonstram a usabilidade e robustez do Leap Motion Controller. Alguns estudos discorrem sobre a precisão de rastreamento [21], [22]; outros sobre os seus benefícios em relação a outros sistemas de rastreamento [23], programas para o reconhecimento de linguagens de sinais [24], aplicações em jogos [25], medicina [26], robótica [27], educação [20], [28], entre outros.…”

Section: Leap Motion Controllerunclassified

Montando o Project North Star: Um Dispositivo de Visualização de Baixo Custo Baseado em Visão Óptica Direta

Fracaro¹,

Peres²,

Maurício³

2019

Anais Estendidos Do Simpósio De Realidade Virtual E Aumentada (SVR)

View full text Add to dashboard Cite

Uma das maiores dificuldades para o estudo da realidade aumentada utilizando a abordagem visão direta é o alto custo dos dispositivos disponíveis comercialmente como o HoloLens da Microsoft ou o Magic Leap One da Magic Leap. Através da união do sensor Leap Motion, impressão 3D, hardware e software de código aberto, a startup desenvolvedora do Leap Motion disponibilizou para o público uma soluçãode um visualizador montado na cabeça que utiliza a visão óptica direta chamado Project North Star. Este artigo tem como objetivo demonstrar o processo de montagem de uma versão deste dispositivo de baixo custo e como ele pode tornar-se uma ferramenta poderosa para o estudo da realidade aumentada.

show abstract

“…The use of constrained grammars and coloured gloves produced low error rates on both training and test data [13]. Using sensor devices, a multimodal framework is applied for isolated sign language translation [14]. The sensors are used to capture finger, palm positions and then the Hidden Markov Model (HMM) and Bidirectional Long Short-Term Memory Neural Network (BLSTM-NN) are used for classification purpose.…”

Section: Introductionmentioning

confidence: 99%

Sign Language Translation Using Deep Convolutional Neural Networks

2020

KSII TIIS

View full text Add to dashboard Cite

Sign language is a natural, visually oriented and non-verbal communication channel between people that facilitates communication through facial/bodily expressions, postures and a set of gestures. It is basically used for communication with people who are deaf or hard of hearing. In order to understand such communication quickly and accurately, the design of a successful sign language translation system is considered in this paper. The proposed system includes object detection and classification stages. Firstly, Single Shot Multi Box Detection (SSD) architecture is utilized for hand detection, then a deep learning structure based on the Inception v3 plus Support Vector Machine (SVM) that combines feature extraction and classification stages is proposed to constructively translate the detected hand gestures. A sign language fingerspelling dataset is used for the design of the proposed model. The obtained results and comparative analysis demonstrate the efficiency of using the proposed hybrid structure in sign language translation.

show abstract

A multimodal framework for sensor based sign language recognition

Cited by 169 publications

References 26 publications

A Bimanual Robotic Teleoperation Architecture with Anthropomorphic Hybrid Grippers for Unstructured Manipulation Tasks

A Bimanual Robotic Teleoperation Architecture with Anthropomorphic Hybrid Grippers for Unstructured Manipulation Tasks

Montando o Project North Star: Um Dispositivo de Visualização de Baixo Custo Baseado em Visão Óptica Direta

Sign Language Translation Using Deep Convolutional Neural Networks

Contact Info

Product

Resources

About