Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video

Liu, Miao; Tang, Siyu; Li, Yin; Rehg, James M.

doi:10.1007/978-3-030-58452-8_41

Cited by 93 publications

(81 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…2D human-object interaction. Perceiving human-object interactions in 2D images has been studied extensively [5,9,10,15,16,22,23,24,35,39,40,43]. Gkioxari et al [10] detect (human, verb, object) triplets using human appearance as cues to localize interacted objects.…”

Section: Related Workmentioning

confidence: 99%

PLACE: Proximity Learning of Articulation and Contact in 3D Environments

Zhang

et al. 2020

2020 International Conference on 3D Vision (3DV)

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

PLACE: Proximity Learning of Articulation and Contact in 3D Environments

Zhang

et al. 2020

2020 International Conference on 3D Vision (3DV)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Egocentric Computer Vision Egocentric vision has been studied in various applications. To name a few, understanding human actions from egocentric cameras, including action/activity recognition [9,35,28,49,36,51,15,52], action anticipation [43,16], and human object interaction [26] have been widely studied. Egocentric hand detection/segmentation [23,22,1,42], and pose estimation [54,66,31] are among other applications.…”

Section: Related Workmentioning

confidence: 99%

Revisiting 3D Object Detection From an Egocentric Perspective

Deng¹,

Qi²,

Najibi³

et al. 2021

Preprint

View full text Add to dashboard Cite

3D object detection is a key module in safety-critical robotics applications such as autonomous driving. For such applications, we care the most about how the detections impact the ego-agent's behavior and safety (the egocentric perspective). Intuitively, we seek more accurate descriptions of object geometry when it's more likely to interfere with the ego-agent's motion trajectory. However, current detection metrics, based on box Intersection-over-Union (IoU), are object-centric and are not designed to capture the spatio-temporal relationship between objects and the ego-agent. To address this issue, we propose a new egocentric measure to evaluate 3D object detection: Support Distance Error (SDE). Our analysis based on SDE reveals that the egocentric detection quality is bounded by the coarse geometry of the bounding boxes. Given the insight that SDE can be improved by more accurate geometry descriptions, we propose to represent objects as amodal contours, specifically amodal star-shaped polygons, and devise a simple model, StarPoly, to predict such contours. Our experiments on the large-scale Waymo Open Dataset show that SDE better reflects the impact of detection quality on the ego-agent's safety compared to IoU; and the estimated contours from StarPoly consistently improve the egocentric detection quality over recent 3D object detectors.

show abstract

“…The main focus of these works is to extract relevant information from the observations to predict the label of the action starting in τ seconds, varying between zero [32] to 10s of seconds [33]. Other models leverage external cues such as hand movements to help with the anticipation task [34,35].…”

Section: Related Workmentioning

confidence: 99%

Weakly-Supervised Dense Action Anticipation

Zhang

Chen

Yao

2021

Preprint

View full text Add to dashboard Cite

Dense anticipation aims to forecast future actions and their durations for long horizons. Existing approaches rely on fully-labelled data, i.e. sequences labelled with all future actions and their durations. We present a (semi-) weakly supervised method using only a small number of fullylabelled sequences and predominantly sequences in which only the (one) upcoming action is labelled. To this end, we propose a framework that generates pseudo-labels for future actions and their durations and adaptively refines them through a refinement module. Given only the upcoming action label as input, these pseudo-labels guide action/duration prediction for the future. We further design an attention mechanism to predict context-aware durations. Experiments on the Breakfast and 50Salads benchmarks verify our method's effectiveness; we are competitive even when compared to fully supervised state-of-the-art models. We will make our code available at: https://github.com/zhanghaotong1/WSLVideoDenseAnticipation.

show abstract

Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video

Cited by 93 publications

References 56 publications

PLACE: Proximity Learning of Articulation and Contact in 3D Environments

PLACE: Proximity Learning of Articulation and Contact in 3D Environments

Revisiting 3D Object Detection From an Egocentric Perspective

Weakly-Supervised Dense Action Anticipation

Contact Info

Product

Resources

About