The Prosthetic Hand Can Grasp Occluded Objects with the Help of 3rd-Eye Based on Artificial Vision and AR Technology

Abstract

Grasping occluded objects is a large challenge for hand amputees because of the hand’s perception loss. This paper proposed an alternative approach based on artificial vision and augmented reality technology, termed 3rd-Eye, to solve the problem. In detail, a live stream of an RGB camera attached to the prosthetic hand was presented to the user who wore an augmented reality helmet, HoloLens 2. The 3rd-Eye was integrated into a hybrid control prosthetic system (i-MYO proposed in our previous work). It was tested in a special reach-and-grasp experiment where the subject needed to grasp an occluded object in cluttered environments (one to six objects in total) using the prosthetic hand. The new approach has achieved satisfactory results (success rate: 100% in all).