Learning human activities and object affordances from RGB-D           videos

Koppula, Hema Swetha; Gupta, Rudhir; Saxena, Ashutosh

doi:10.1177/0278364913478446

Cited by 617 publications

(548 citation statements)

References 64 publications

Supporting

Mentioning

523

Contrasting

Order By: Relevance

“…However, they manually define affordances in terms of kinematic and dynamic constraints. Recently, Koppula et al [23,21] show that human-actor based affordances are essential for robots working in human spaces in order for them to interact with objects in a human-desirable way. They applied it to look-ahead reactive planning for robots.…”

Section: Related Workmentioning

confidence: 99%

“…We use the OpenNI skeleton tracker, to obtain the skeleton poses in these RGBD videos. We obtained the ground-truth object bounding box labels via labeling and SIFT-based tracking using the code given in [23]. The dataset is publicly available (along with open-source code): http://pr.cs.cornell.edu/humanactivities/data.php.…”

Section: Generating Physically-grounded Affordancesmentioning

confidence: 99%

“…The dataset is publicly available (along with open-source code): http://pr.cs.cornell.edu/humanactivities/data.php. We combined our affordance RGB-D videos with those from the CAD-120 dataset [23] and obtain a total of 815 instances for our experiments.…”

Section: Generating Physically-grounded Affordancesmentioning

confidence: 99%

“…Previous activity labeling approaches [23,22] heavily rely on human poses for temporal segmentation and labeling, which sometimes miss boundaries between sub-activities and result in labeling errors. Koppula et al [23] show that good temporal segmentation is very important for the labeling task.…”

Section: Activity and Semantic Affordance Labelingmentioning

confidence: 99%

“…We show that our generative model can reconstruct these trajectories well. We also show that our approach can improve the affordance and activity detection performance on the CAD-120 dataset [23]. The contributions of this paper are as follows:…”

Section: Introductionmentioning

confidence: 97%

See 4 more Smart Citations

Physically Grounded Spatio-temporal Object Affordances

Koppula

Saxena

2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Objects in human environments support various functionalities which govern how people interact with their environments in order to perform tasks. In this work, we discuss how to represent and learn a functional understanding of an environment in terms of object affordances. Such an understanding is useful for many applications such as activity detection and assistive robotics. Starting with a semantic notion of affordances, we present a generative model that takes a given environment and human intention into account, and grounds the affordances in the form of spatial locations on the object and temporal trajectories in the 3D environment. The probabilistic model also allows uncertainties and variations in the grounded affordances. We apply our approach on RGB-D videos from Cornell Activity Dataset, where we first show that we can successfully ground the affordances, and we then show that learning such affordances improves performance in the labeling tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Generating Physically-grounded Affordancesmentioning

confidence: 99%

Section: Generating Physically-grounded Affordancesmentioning

confidence: 99%

Section: Activity and Semantic Affordance Labelingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 97%

See 3 more Smart Citations

Physically Grounded Spatio-temporal Object Affordances

Koppula

Saxena

2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

Densely connected GCN model for motion prediction

Qiu

Wang

et al. 2020

Computer Animation & Virtual

View full text Add to dashboard Cite

Human motion prediction is a fundamental problem in understanding human natural movements. This task is very challenging due to the complex human body constraints and diversity of action types. Due to the human body being a natural graph, graph convolutional network (GCN)-based models perform better than the traditional recurrent neural network (RNN)-based models on modeling the natural spatial and temporal dependencies lying in the motion data. In this paper, we develop the GCN-based models further by adding densely connected links to increase their feature utilizations and address oversmoothing problem. More specifically, the GCN block is used to learn the spatial relationships between the nodes and each feature map of the GCN block propagates directly to every following block as input rather than residual linked. In this way, the spatial dependency of human motion data is exploited more sufficiently and the features of different level of scale are fused more efficiently. Extensive experiments demonstrate our model achieving the state-of-the-art results on CMU dataset.

show abstract

Distillation of human–object interaction contexts for action recognition

Almushyti

2022

Computer Animation & Virtual

View full text Add to dashboard Cite

Modeling spatial-temporal relations is imperative for recognizing human actions, especially when a human is interacting with objects, while multiple objects appear around the human differently over time. Most existing action recognition models focus on learning overall visual cues of a scene but disregard a holistic view of human-object relationships and interactions, that is, how a human interacts with respect to short-term task for completion and long-term goal. We therefore argue to improve human action recognition by exploiting both the local and global contexts of human-object interactions (HOIs). In this paper, we propose the Global-Local Interaction Distillation Network (GLIDN), learning human and object interactions through space and time via knowledge distillation for holistic HOI understanding. GLIDN encodes humans and objects into graph nodes and learns local and global relations via graph attention network. The local context graphs learn the relation between humans and objects at a frame level by capturing their co-occurrence at a specific time step.The global relation graph is constructed based on the video-level of human and object interactions, identifying their long-term relations throughout a video sequence. We also investigate how knowledge from these graphs can be distilled to their counterparts for improving HOI recognition. Finally, we evaluate our model by conducting comprehensive experiments on two datasets including Charades and CAD-120. Our method outperforms the baselines and counterpart approaches.

show abstract

Learning human activities and object affordances from RGB-D videos

Cited by 617 publications

References 64 publications

Physically Grounded Spatio-temporal Object Affordances

Physically Grounded Spatio-temporal Object Affordances

Densely connected GCN model for motion prediction

Distillation of human–object interaction contexts for action recognition

Contact Info

Product

Resources

About