Object Level Visual Reasoning in Videos

Abstract

Human activity recognition is typically addressed by detecting key concepts like global and local motion, features related to object classes present in the scene, as well as features related to the global context. The next open challenges in activity recognition require a level of understanding that pushes beyond this and call for models with capabilities for fine distinction and detailed comprehension of interactions between actors and objects in a scene. We propose a model capable of learning to reason about semantically meaningful spatio-temporal interactions in videos. The key to our approach is a choice of performing this reasoning at the object level through the integration of state of the art object detection networks. This allows the model to learn detailed spatial interactions that exist at a semantic, object-interaction relevant level. We evaluate our method on three standard datasets (Twenty-BN Something-Something, VLOG and EPIC Kitchens) and achieve state of the art results on all of them. Finally, we show visualizations of the interactions learned by the model, which illustrate object classes and their interactions corresponding to different activity classes.

Multitask Learning to Improve Egocentric Action Recognition

“…In [57], the relationship between tasks is modelled in a latent space to transfer knowledge between them and reduce the number of required training samples. MTL in egocentric vision appears in [1,28,25,18,29,47].…”

Grouped Spatial-Temporal Aggregation for Efficient Action Recognition

“…For the Epic Kitchen Dataset [4], there are a total of 125 verb classes and each verb can be acted on different objects. We report the results on the validation set using the same split as [1]. We only evaluate on the verb class prediction following [1] since the main purpose of this paper is on temporal action recognition instead of objects.…”

“…We report the results on the validation set using the same split as [1]. We only evaluate on the verb class prediction following [1] since the main purpose of this paper is on temporal action recognition instead of objects. For the EGTEA Gaze++ dataset, it contains 106 classes with 19 different verbs.…”

“…[1] LFB [31] GST(Ours) Top1 (Top 5) 40.89 (-) 54.4 (81.8) 56.50 (82.72) Table 6. Results on validation set of Epic-Kitchen verb classification tasks using the same split as in [1]…”

“…We use two recently collected egocentric dataset, Epic Kitchen [4] and EGTEA Gaze+ [14]. For Epic Kitchen, we report the verb classification results using the same split as [1]. EGTEA Gaze++ is a recently collected dataset with approximately 10K samples of 106 activity classes.…”