Recognizing Gestures on Projected Button Widgets with an RGB-D Camera Using a CNN

Chiu, Patrick; Kim, Chelhwon; Oda, Hideto

doi:10.1145/3279778.3279907

Cited by 5 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Chung et al [6] improve the system by using three sidemounted cameras that allow for more accurate finger capture. Some systems, e.g., Chiu et al [5], use neural networks to classify touch gestures on the display surface, achieving 3.5% gesture error. Hand posture sensing is also used to develop touch screen surfaces for interaction.…”

Section: Sensing Hands In Human-computer Interactionmentioning

confidence: 99%

ShadowSense

Bejarano

Hoffman

2020

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

This paper proposes and evaluates the use of image classification for detailed, full-body human-robot tactile interaction. A camera positioned below a translucent robot skin captures shadows generated from human touch and infers social gestures from the captured images. This approach enables rich tactile interaction with robots without the need for the sensor arrays used in traditional social robot tactile skins. It also supports the use of touch interaction with non-rigid robots, achieves high-resolution sensing for robots with different sizes and shape of surfaces, and removes the requirement of direct contact with the robot. We demonstrate the idea with an inflatable robot and a standing-alone testing device, an algorithm for recognizing touch gestures from shadows that uses Densely Connected Convolutional Networks, and an algorithm for tracking positions of touch and hovering shadows. Our experiments show that the system can distinguish between six touch gestures under three lighting conditions with 87.5 - 96.0% accuracy, depending on the lighting, and can accurately track touch positions as well as infer motion activities in realistic interaction conditions. Additional applications for this method include interactive screens on inflatable robots and privacy-maintaining robots for the home.

show abstract