Hotspots detection for machine operation in egocentric vision

Chen, Longfei; Kondo, Kazuaki; Nakamura, Yūichi; Damen, Dima; Mayol-Cuevas, Walterio

doi:10.23919/mva.2017.7986841

Cited by 7 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Shan et al [1], [6], the task of hand-object interaction detection is treated as object detection, different hand related cues are treated as different object classes, and whether an object is being interacted with or not is also treated as different object classes in training. In addition to this, in the work from Chen et al [7], with the depth sensor, they compare the depth within the palm area and the depth around the palm area to determine weather the hand is touching an object. Similarly, Lu et al [8] use the same strategy and they plot an AUC (area under curve) curve to determine the best threshold for the criterion of 'touch' detection.…”

Section: Related Work a Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection

Lu¹,

Liu²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a method to jointly determine the status of hand-object interaction. This is crucial for egocentric human activity understanding and interaction. From a computer vision perspective, we believe that determining whether a hand is interacting with an object depends on whether there is an interactive hand pose and whether the hand is touching the object. Thus, we extract the hand pose, handobject masks to jointly determine the interaction status. In order to solve the problem of hand pose estimation due to inhand object occlusion, we use a multi-cam system to capture hand pose data from multiple perspectives. We evaluate and compare our method with the most recent work from Shan et al.[1] on selected images from EPIC-KITCHENS [2] dataset and achieve 89% accuracy on HOI (hand-object interaction) detection which is comparative to Shan's (92%). However, for real-time performance, our method can run over 30 FPS which is much more efficient than Shan's (1 ∼ 2 FPS). A demo can be found from https://www.youtube.com/watch?v=XVj3zBuynmQ.

show abstract

Section: Related Work a Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection

Lu¹,

Liu²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides the object detector based method like [24,8]. In work from Chen et al [3], and Lu et al [15], in order to find a 'touch' with the machine, the depth sensor is used to measure the distance between the hand and machine, which is effective but inflexible. Likitlersuang et al [12] build a hand-object interaction classifier with the optical flow and hand shape as input.…”

Section: Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection and Application

Lu¹,

Mayol-Cuevas²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we present a method to detect the handobject interaction from egocentric perspective. In contrast to massive data driven discriminator based method like [24], we propose a novel workflow that utilise the cues of hand and object. Specifically, we train networks predicting hand pose, hand mask and in-hand object mask to jointly predict the hand-object interaction status. We compare our method with the most recent work from Shan et al. [24] on selected images from EPIC-KITCHENS [4] dataset and achieve 89% accuracy on HOI (hand-object interaction) detection which is comparative to Shan's (92%). However, for real-time performance, with the same machine, our method can run over 30 FPS which is much efficient than Shan's (1 ∼ 2 FPS). Furthermore, with our approach, we are able to segment script-less activities from where we extract the frames with the HOI status detection. We achieve 68.2% and 82.8% F1 score on GTEA [7] and the UTGrasp [1] dataset respectively which are all comparative to the SOTA methods.

show abstract

“…The hand is another important spot, which indicate the operators' intentions and procedures to perform operations. The third important spot is "hotspot", which represents the frequent-interacting location between a hand and a machine [1].…”

Section: Characteristics Of Operationsmentioning

confidence: 99%

“…A hotspot is detected by clustering the touches of between the hand and machine surface utilizing both spatial and temporal locations. The details are given by [1]. Temporal duration: The operation experience is divided into a sequence of operation units (OUs).…”

Section: Feature Extractionmentioning

confidence: 99%

Detecting Clues for Skill Levels and Machine Operation Difficulty from Egocentric Vision

Chen¹,

Nakamura²,

Kondo³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

With respect to machine operation tasks, the experiences from different skill level operators, especially novices, can provide worthy understanding about the manner in which they perceive the operational environment and formulate knowledge to deal with various operation situations. In this study, we describe the operator's behaviors by utilizing the relations among their head, hand, and operation location (hotspot) during the operation. A total of 40 experiences associated with a sewing machine operation task performed by amateur operators was recorded via a headmounted RGB-D camera. We examined important features of operational behaviors in different skill level operators and confirmed their correlation to the difficulties of the operation steps. The result shows that the pure-gazing behavior is significantly reduced when the operator's skill improved. Moreover, the hand-approaching duration and the frequency of attention movement before operation are strongly correlated to the operational difficulty in such machine operating environments.

show abstract

Hotspots detection for machine operation in egocentric vision

Cited by 7 publications

References 8 publications

Egocentric Hand-object Interaction Detection

Egocentric Hand-object Interaction Detection

Egocentric Hand-object Interaction Detection and Application

Detecting Clues for Skill Levels and Machine Operation Difficulty from Egocentric Vision

Contact Info

Product

Resources

About