Hands Holding Clues for Object Recognition in Teachable Machines

Abstract

Camera manipulation confounds the use of object recognition applications by blind people. This is exacerbated when photos from this population are also used to train models, as with teachable machines, where out-of-frame or partially included objects against cluttered backgrounds degrade performance. Leveraging prior evidence on the ability of blind people to coordinate hand movements using proprioception, we propose a deep learning system that jointly models hand segmentation and object localization for object classification. We investigate the utility of hands as a natural interface for including and indicating the object of interest in the camera frame. We confirm the potential of this approach by analyzing existing datasets from people with visual impairments for object recognition. With a new publicly available egocentric dataset and an extensive error analysis, we provide insights into this approach in the context of teachable recognizers.

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

“…Similarly, Feiz et al [ 87 ] used AI technologies to develop an application enabling blind people to write on printed forms. Lee et al [ 88 ] attempts to improve object recognition based applications for blind people by leveraging the hand as a point to consider when focusing on an object in a frame. Zhao et al [ 89 ] developed a face recognition tool for visually impaired people to identify their friends.…”

“…Considering the works that focused on the user side, some researchers catered to general end-users or consumers [83,101,200,210], while others on specific end-users. Examples for these include people who need assistance [2,80,[86][87][88][89]96,117,147], medical professionals [57,67,110,192,193], international travelers [50], Amazon Mechanical Turk [60,99], drivers [161,162], musicians [102], teachers [124], students [128], children [72,125], UX designers [65,115,206,209], UI designers [103,111,173], data analysts [97], video creators [84], and game designers [70,165,174,211]. Apart from focusing on a specific user group, some have tried to understand multiple user-perspectives from ML engineers to the end-user [48].…”

GO-Finder: A Registration-Free Wearable System for Assisting Users in Finding Lost Objects via Hand-Held Object Discovery

“…Since objects appearing in first-person videos are typically handled by hands, hand information is used to improve object detection. Lee et al [19,20] proposed using hands as a guide to identify an object of interest from a photo taken by people with visual impairment. Shan et al [31] collected a large-scale dataset of hand-object interaction along with annotated bounding boxes of hands and objects in contact with each other.…”

Hand with Sensing Sphere: Body-Centered Spatial Interactions with a Hand-Worn Spherical Camera

“…With our prototype, we can recognize what the index finger is pointing at, as we discussed in Section 5.3. For example, users can hold and indicate an object to be recognized, an interaction which was recently introduced to leverage proprioception by Lee et al, [30]. When the hand without the device holds an object and the other index finger points to it, the fingertip and the object appear to line up in the image, as shown in Figure 7.…”