Integrated vision-based robotic arm interface for operators with upper limb mobility impairments

Abstract

An integrated, computer vision-based system was developed to operate a commercial wheelchair-mounted robotic manipulator (WMRM). In this paper, a gesture recognition interface system developed specifically for individuals with upper-level spinal cord injuries (SCIs) was combined with object tracking and face recognition systems to be an efficient, hands-free WMRM controller. In this test system, two Kinect cameras were used synergistically to perform a variety of simple object retrieval tasks. One camera was used to interpret the hand gestures to send as commands to control the WMRM and locate the operator's face for object positioning. The other sensor was used to automatically recognize different daily living objects for test subjects to select. The gesture recognition interface incorporated hand detection, tracking and recognition algorithms to obtain a high recognition accuracy of 97.5% for an eight-gesture lexicon. An object recognition module employing Speeded Up Robust Features (SURF) algorithm was performed and recognition results were sent as a command for "coarse positioning" of the robotic arm near the selected daily living object. Automatic face detection was also provided as a shortcut for the subjects to position the objects to the face by using a WMRM. Completion time tasks were conducted to compare manual (gestures only) and semi-manual (gestures, automatic face detection and object recognition) WMRM control modes. The use of automatic face and object detection significantly increased the completion times for retrieving a variety of daily living objects.

Hybrid Control of a Vision-Guided Robot Arm by EOG, EMG, EEG Biosignals and Head Movement Acquired via a Consumer-Grade Wearable Device

“…We aimed at delivering a compromise between ease of control and flexibility for assistive applications: according to the proposed hybrid approach, the user retains unconstrained control in steering the robot toward the target object, and engaging autonomous guidance afterwards relieves the user from the burden of fine adjustment of the joints to attain intended postures, maintaining the goal focus. We demonstrated this approach using a desktop robotic arm whose 5+1 degree-of-freedom kinematics are analogous to existing assistive robotic manipulators, aiding clinical translation of the results [3], [5], [7], [38], [50], [65]. This experiment relied on elementary object detection via hue and geometric features, but the approach is viable with arbitrary vision systems, e.g., capable of recognizing objects belonging to specific classes through deep learning techniques; furthermore, it is, in principle, applicable to both image-based and positionbased vision servoing [5], [44], [46], [66]- [68].…”

“…A substantial challenge in this area is to develop effective human-machine interfaces and paradigms for robot control, with the available technologies differing substantially in their residual motor function requirements, command throughput, ease of use, technical complexity and cost. At the bottom end of the spectrum, joystick (or micro-switch) control is suitable mainly for patients with at least partially preserved hand function (e.g., as after hemispheric stroke), being inexpensive and highly effective for driving, e.g., motorized wheelchairs and assistive arms [3], [4]; gesture-based control via low-cost camera systems is also emerging as a suitable alternative, posing less stringent requirements on upper limb and head movement capability [5], [6]. Highly accurate control of assistive devices has repeatedly been demonstrated based on superficially-recorded face-muscle electromyographic (EMG) and/or electrooculographic (EOG) signals; although harvesting information from these signals is generally more expensive and technically-demanding compared to micro-switch and camera-based interfaces, at a minimum, it only requires integrity of the cranial nerves function, which is generally preserved in patients with spinal lesions [7]- [12].…”

“…This need has driven recourse to either degreesof-freedom reduction approaches, namely, linking axes via pre-determined relationships or postures, or ''sparse'' control schemes, wherein the user selects and activates preestablished motor sequences via ''high-level'' commands with or without the support of a graphical user interface [25], [33], [38]- [42]. Such approach has proven particularly effective when combined with computer vision systems implementing object recognition and partially-autonomous guidance, which can drastically simplify and accelerate the performance of object manipulation tasks using robotic arms and even humanoid robots [5], [42]- [46].…”

Object Affordance-Based Implicit Interaction for Wheelchair-Mounted Robotic Arm Using a Laser Pointer

“…To facilitate the operation of robots for users with different levels of physical ability, many human-robot interactions (HRI) interfaces utilizing residual limb abilities have been studied, such as using chin [6], shoulder [7], gesture [8], and eye movement [9]. In these studies, the interaction operations mapped the residual limb movement to robot instructions, such as forward, back, left, right, rotation, or other cartesian motions for the robotic arm, and some preset simple household tasks, which could help users perform some structured tasks, but still needing frequent limb movement.…”

Assisted Grasping in Individuals with Tetraplegia: Improving Control through Residual Muscle Contraction and Movement

“…EMG signals have also been used to control an upper limb exoskeleton in [15]. Finally, body movement was used in [16] and [17] with camera-based systems and inertial measurement units (IMUs), respectively. In [18], contralateral shoulder motions were related to hand muscle stimulation by an external shoulder position transducer.…”