TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing

Meier, Manuel; Streli, Paul; Fender, Andreas; Holz, Christian

doi:10.1109/vr50410.2021.00076

Cited by 48 publications

(9 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The method first fits a 3D plane to the wall or tabletop surface and thresholds the distance between plane and the fingertip. TapID [40] uses subtle accelerations from a pair of IMU sensors on the wrist to detect contact between fingers and surfaces. TouchAnywhere [43] detects touch via heuristics by estimating the intersection between finger and its shadow, but only detects binary contact and does not include pressure ground truth.…”

Section: Related Workmentioning

confidence: 99%

PressureVision: Estimating Hand Pressure from a Single RGB Image

Grady¹,

Tang²,

Brahmbhatt³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

PressureVision: Estimating Hand Pressure from a Single RGB Image

Grady¹,

Tang²,

Brahmbhatt³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The interaction between robots and controllers is a rather complex process that includes highly sensitive sensing, signal and data processing and the execution of control [43,49] . In particular, this process places high demands on response speed and execution accuracy.…”

Section: P-hmis For Robotic Controlmentioning

confidence: 99%

“…This review covers the recent progress made regarding smart flexible piezoelectric devices in the field of HMIs. Research concerning flexible piezoelectric electronics is comprehensively summarized and highlighted along with real-life P-HMIs, including robotic control, the Internet of Things (IoT), sports coaching, acoustic therapeutics and machine learning-enhanced P-HMIs [Figure 1] [27,[42][43][44][45][46] . Finally, the current challenges and prospects of P-HMIs are provided from our perspective, ranging from materials science and structural design to diverse potential applications in the near future.…”

Section: Introductionmentioning

confidence: 99%

Recent progress in flexible piezoelectric devices toward human-machine interactions

2022

Soft Sci

View full text Add to dashboard Cite

Human-machine interactions are becoming increasingly required for intelligent sensing and effective manipulation. Recent developments in flexible piezoelectric sensors with short response time and high force-electric interconversion efficiency present a tendency toward facilitating diverse human-machine interactive applications. Here, we review the development of flexible piezoelectric human-machine interactions in the context of robotic control, the Internet of Things, sports coaching and acoustic therapeutics. The synthesis of unique materials, the distinct design of device structures, the typical applications of piezoelectric human-machine interactions and the integration of cutting-edge technologies are elaborated in detail based on recent research. Finally, we highlight the current challenges and directions for the development of piezoelectric human-machine interactions for more advanced application scenarios.

show abstract

“…Given a video sequence or an RGB image captured from a camera or mobile device, the task of markerless pose estimation is to predict the positions of the body keypoints (including joints and vertices) relative to a certain coordinate system [ 1 ]. As one most frequently used parts that human beings interact with the environment, hand pose estimation (HPE) is of great research interest and has numerous applications in areas such as robotics, virtual reality (VR) and augmented reality (AR), AI-aided diagnosis and smart human-computer interaction (HCI) systems [ 2 , 3 ]. Apart from those downstream applications, HPE also plays an important role in many basic upstream tasks, including gesture recognition [ 4 , 5 , 6 ] and sign language recognition (SLR) [ 7 , 8 ].…”

Section: Introductionmentioning

confidence: 99%

“…In recent years, with the rapid development of hardware (e.g., Microsoft Kinect [ 9 ], Oak-D camera [ 10 ], wearable sensors) and advances in deep learning algorithms, HPE research has achieved considerable progress. The state-of-the-art approaches have achieved promising performance in a controlled environment with different data modalities, such as 2D images, 2D images with depth map [ 11 , 12 , 13 ], wearable gloves and sensors [ 3 , 14 ]. Among these modalities, since single-view 2D RGB images are much more available than sensors and depth images, HPE from a single RGB image can be widely used and easily deployed to various end devices.…”

Section: Introductionmentioning

confidence: 99%

PA-Tran: Learning to Estimate 3D Hand Pose with Partial Annotation

Bidulka

McKeown

et al. 2023

Sensors

View full text Add to dashboard Cite

This paper tackles a novel and challenging problem—3D hand pose estimation (HPE) from a single RGB image using partial annotation. Most HPE methods ignore the fact that the keypoints could be partially visible (e.g., under occlusions). In contrast, we propose a deep-learning framework, PA-Tran, that jointly estimates the keypoints status and 3D hand pose from a single RGB image with two dependent branches. The regression branch consists of a Transformer encoder which is trained to predict a set of target keypoints, given an input set of status, position, and visual features embedding from a convolutional neural network (CNN); the classification branch adopts a CNN for estimating the keypoints status. One key idea of PA-Tran is a selective mask training (SMT) objective that uses a binary encoding scheme to represent the status of the keypoints as observed or unobserved during training. In addition, by explicitly encoding the label status (observed/unobserved), the proposed PA-Tran can efficiently handle the condition when only partial annotation is available. Investigating the annotation percentage ranging from 50–100%, we show that training with partial annotation is more efficient (e.g., achieving the best 6.0 PA-MPJPE when using about 85% annotations). Moreover, we provide two new datasets. APDM-Hand, is for synthetic hands with APDM sensor accessories, which is designed for a specific hand task. PD-APDM-Hand, is a real hand dataset collected from Parkinson’s Disease (PD) patients with partial annotation. The proposed PA-Tran can achieve higher estimation accuracy when evaluated on both proposed datasets and a more general hand dataset.

show abstract

TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing

Cited by 48 publications

References 44 publications

PressureVision: Estimating Hand Pressure from a Single RGB Image

PressureVision: Estimating Hand Pressure from a Single RGB Image

Recent progress in flexible piezoelectric devices toward human-machine interactions

PA-Tran: Learning to Estimate 3D Hand Pose with Partial Annotation

Contact Info

Product

Resources

About