HIGS: Hand Interaction Guidance System

Lu, Yao; Mayol-Cuevas, Walterio

doi:10.1109/ismar-adjunct.2019.000-7

Cited by 8 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditional task guidance systems Pritchett, 1998, 2000) focused on providing the user with pre-loaded task-specific information without tracking the current state of the environment, or being able to generalize to new tasks (Leelasawassuk et al, 2017;Reyes et al, 2020;Lu and Mayol-Cuevas, 2019;Wang et al, 2016). The complexity of the problem comes from various aspects, such as environment understanding, object and action recognition, user's preference and mental state detection, real-time inference, etc (Manuvinakurike et al, 2018;Kim et al, 2022).…”

Section: Task Guidance Systemsmentioning

confidence: 99%

Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?

Bao,

Yu,

Zhang

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. These systems need to have a sophisticated understanding of the user as well as the environment, and make timely accurate decisions on when and what to say. To address this issue, we created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We further proposed two tasks: User and Environment Understanding, and Instructor Decision Making. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance. Our quantitative, qualitative, and human evaluation results show that these models can demonstrate fair performances in some cases with no task-specific training, but a fast and reliable adaptation remains a significant challenge. Our benchmark and baselines will provide a stepping stone for future work on situated task guidance.

show abstract

Section: Task Guidance Systemsmentioning

confidence: 99%

Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?

Bao,

Yu,

Zhang

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…In addition to this, in the work from Chen et al [7], with the depth sensor, they compare the depth within the palm area and the depth around the palm area to determine weather the hand is touching an object. Similarly, Lu et al [8] use the same strategy and they plot an AUC (area under curve) curve to determine the best threshold for the criterion of 'touch' detection. Likitlersuang et al [9] put attention on higher level features, they build a hand-object interaction classifier with the optical flow and hand shape as input.…”

Section: Related Work a Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection

Lu¹,

Liu²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a method to jointly determine the status of hand-object interaction. This is crucial for egocentric human activity understanding and interaction. From a computer vision perspective, we believe that determining whether a hand is interacting with an object depends on whether there is an interactive hand pose and whether the hand is touching the object. Thus, we extract the hand pose, handobject masks to jointly determine the interaction status. In order to solve the problem of hand pose estimation due to inhand object occlusion, we use a multi-cam system to capture hand pose data from multiple perspectives. We evaluate and compare our method with the most recent work from Shan et al.[1] on selected images from EPIC-KITCHENS [2] dataset and achieve 89% accuracy on HOI (hand-object interaction) detection which is comparative to Shan's (92%). However, for real-time performance, our method can run over 30 FPS which is much more efficient than Shan's (1 ∼ 2 FPS). A demo can be found from https://www.youtube.com/watch?v=XVj3zBuynmQ.

show abstract

“…Besides the object detector based method like [24,8]. In work from Chen et al [3], and Lu et al [15], in order to find a 'touch' with the machine, the depth sensor is used to measure the distance between the hand and machine, which is effective but inflexible. Likitlersuang et al [12] build a hand-object interaction classifier with the optical flow and hand shape as input.…”

Section: Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection and Application

Lu¹,

Mayol-Cuevas²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we present a method to detect the handobject interaction from egocentric perspective. In contrast to massive data driven discriminator based method like [24], we propose a novel workflow that utilise the cues of hand and object. Specifically, we train networks predicting hand pose, hand mask and in-hand object mask to jointly predict the hand-object interaction status. We compare our method with the most recent work from Shan et al. [24] on selected images from EPIC-KITCHENS [4] dataset and achieve 89% accuracy on HOI (hand-object interaction) detection which is comparative to Shan's (92%). However, for real-time performance, with the same machine, our method can run over 30 FPS which is much efficient than Shan's (1 ∼ 2 FPS). Furthermore, with our approach, we are able to segment script-less activities from where we extract the frames with the HOI status detection. We achieve 68.2% and 82.8% F1 score on GTEA [7] and the UTGrasp [1] dataset respectively which are all comparative to the SOTA methods.

show abstract

HIGS: Hand Interaction Guidance System

Cited by 8 publications

References 14 publications

Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?

Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?

Egocentric Hand-object Interaction Detection

Egocentric Hand-object Interaction Detection and Application

Contact Info

Product

Resources

About